Household appliance and boiling detection method

Abstract

The embodiment of the invention discloses a household appliance and a boiling detection method, relates to the technical field of household appliances, and is used for avoiding pot overflowing and pot drying in the cooking process, the household appliance comprises an image acquisition device used for acquiring an image of a target area, and a cooker and a pot are placed in the target area; the controller is configured to respond to a boiling detection instruction initiated by a user, and acquire a first image of the pot in a cooking state through the image acquisition equipment; inputting the first image into a boiling detection model to obtain a detection result of the cookware; wherein the detection result comprises that the pot is provided with a pot cover or the pot is not provided with a pot cover, and soup in the pot is boiled or the soup in the pot is not boiled; and when the detection result is that the pot has no pot cover, if the soup in the pot is boiled, turning down the fire gear of the stove.

Description

Technical Field

[0001] The present application relates to the technical field of household electrical appliances, and in particular to a household electrical appliance and a boiling detection method. Background Art

[0002] With the popularization of smart devices and the maturity of artificial intelligence (AI) technology, people are increasingly keen on using smart home appliances to facilitate their daily lives.

[0003] Nowadays, when users are cooking in the kitchen, they need to always pay attention to the soup or ingredients in the pot, and manually adjust the fire level or add ingredients in time to avoid overflowing or burning the pot.

[0004] However, this cooking method not only requires users to spend a lot of time in the kitchen, limiting the flexibility of life, but also the burning substances and smoke produced by the gas stove will have an impact on human health. Therefore, it is necessary to find a more convenient and healthy alternative. Summary of the invention

[0005] The present application provides a household appliance and a boiling detection method for avoiding overflowing or drying out of a pot during cooking.

[0006] In order to achieve the above objectives, this application adopts the following technical solutions.

[0007] In a first aspect, an embodiment of the present application provides a household appliance, comprising: an image acquisition device, for acquiring an image of a target area, where a stove and a pot are placed; a controller, configured to: in response to a boiling detection instruction initiated by a user, obtain a first image of the pot in a cooking state through the image acquisition device; input the first image into a boiling detection model to obtain a detection result of the pot; wherein the detection result includes whether the pot has a lid or the pot has no lid and whether the soup in the pot has boiled or the soup in the pot has not boiled; when the detection result is that the pot has no lid, if the soup in the pot has boiled, lower the fire level of the stove.

[0008] The technical solution provided by the embodiment of the present application brings at least the following beneficial effects: the technical solution obtains the image of the pot during the cooking process through the image acquisition device, and in the case that the pot in the image does not have a pot cover, determines whether the soup has boiled according to the image of the soup in the pot, and automatically lowers the fire level of the stove when the soup in the pot has boiled, so as to avoid the phenomenon of overflowing the pot. The method can effectively monitor the state of the soup and the entire cooking process of the soup cooked using the stove, and autonomously adjust the fire level of the gas stove, so that the workload of the user is reduced, and there is no need to frequently monitor the cooking process. The entire cooking process is healthier and safer, and the user can enjoy a more convenient and high-quality cooking experience.

[0009] In some embodiments, the detection result also includes whether side dishes have been added to the pot or no side dishes have been added to the pot; the controller is further configured to: when the detection result is that the pot has no lid, if the soup in the pot is boiling and no side dishes have been added to the pot, lower the fire level of the stove and send a reminder to the user to add side dishes; after the user adds side dishes to the pot, increase the fire level of the stove.

[0010] In some embodiments, the boiling detection model is trained by the following method: obtaining a training sample set; the training sample set includes multiple labeled samples, the multiple labeled samples include boiling image data of sample pots, and the labels include boiling data sets of sample pots; according to the training sample set, the boiling detection model is trained to obtain the boiling detection model after training.

[0011] In some embodiments, the controller is configured to input the first image into a boiling detection model to obtain a detection result of the soup in the cooker, and is specifically configured to: obtain at least one frame image in the first image; input at least one frame image into a boiling detection model to obtain a prediction score of at least one frame image; wherein the prediction score includes a soup boiling prediction score, a pot lid presence prediction score, and a side dish addition prediction score; and determine the detection result of the cooker based on the prediction score.

[0012] In some embodiments, the household appliance further includes: a temperature sensor for detecting the gas temperature in the cooker; the controller is further configured to: when the cooker has a lid, periodically obtain the gas temperature in the cooker at preset time intervals through the temperature sensor; determine whether the soup in the cooker has boiled based on the gas temperature in the cooker in the current cycle and the gas temperature in the cooker in the first cycle; wherein the first cycle is the previous cycle of the current cycle.

[0013] In some embodiments, the controller is configured to determine whether the soup in the cooker has boiled based on the gas temperature in the cooker in a current cycle and the gas temperature in the cooker in a first cycle. Specifically, the controller is configured to: determine that the soup in the cooker has boiled if the gas temperature in the cooker in the current cycle is greater than or equal to a preset temperature threshold and the gas temperature in the cooker in the first cycle is less than the preset temperature threshold; determine that the soup in the cooker has not boiled if the gas temperature in the cooker in the current cycle is less than the preset temperature threshold.

[0014] In a second aspect, an embodiment of the present application provides a boiling detection method, the method comprising: in response to a boiling detection command initiated by a user, obtaining a first image of the cooker in a cooking state; inputting the first image into a boiling detection model to obtain a detection result of the cooker; wherein the detection result includes whether the cooker has a lid or has no lid and whether the soup in the cooker has boiled or has not boiled; when the detection result is that the cooker has no lid, if the soup in the cooker has boiled, lowering the firepower level of the cooker.

[0015] In the third aspect, an embodiment of the present application provides a controller comprising: one or more processors; one or more memories; wherein the one or more memories are used to store computer program codes, the computer program codes include computer instructions, and when the one or more processors execute the computer instructions, the controller executes any one of the boiling detection methods provided in the second aspect.

[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are executed on a computer, the computer executes the method provided in the second aspect and possible implementation methods.

[0017] In a fifth aspect, an embodiment of the present invention provides a computer program product, which can be directly loaded into a memory and contains software code. After being loaded and executed by a computer, the computer program product can implement the method provided in the second aspect and possible implementation methods.

[0018] It should be noted that the above computer instructions may be stored in whole or in part on a computer-readable storage medium, wherein the computer-readable storage medium may be packaged together with the processor of the controller, or may be packaged separately from the processor of the controller, which is not limited in this application.

[0019] The beneficial effects described in the second to fifth aspects of the present application can be referred to the beneficial effect analysis of the first aspect and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings are used to provide a further understanding of the technical solution of the present invention and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present invention and do not constitute a limitation on the technical solution of the present invention.

[0021] Figure 1 A hardware configuration diagram of a household appliance system provided in an embodiment of the present application;

[0022] Figure 2 A schematic diagram of the structure of a cookware provided in an embodiment of the present application;

[0023] Figure 3A hardware configuration block diagram of a household appliance provided in an embodiment of the present application;

[0024] Figure 4 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0025] Figure 5 A schematic diagram of a boiling detection method provided in an embodiment of the present application;

[0026] Figure 6 A schematic flow chart of another boiling detection method provided in an embodiment of the present application;

[0027] Figure 7 A schematic diagram of a model training method flow provided in an embodiment of the present application;

[0028] Figure 8 A schematic flow chart of another boiling detection method provided in an embodiment of the present application;

[0029] Fig. 9 A schematic flow chart of another boiling detection method provided in an embodiment of the present application;

[0030] Fig.10 A schematic flow chart of another boiling detection method provided in an embodiment of the present application;

[0031] Fig.11 A schematic flow chart of another boiling detection method provided in an embodiment of the present application;

[0032] Fig.12 A schematic flow chart of another boiling detection method provided in an embodiment of the present application;

[0033] Fig.13 A schematic flow chart of another boiling detection method provided in an embodiment of the present application;

[0034] Fig.14 A schematic diagram of the structure of a boiling detection device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0036] It should be noted that all directional indications in the embodiments of the present invention (such as up, down, left, right, front, back, etc.) are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0037] The terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, unless otherwise specified, "plurality" means two or more.

[0038] In the description of this application, it should be noted that, unless otherwise clearly specified and limited, the terms "connected" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances. In addition, when describing a pipeline, the "connected" and "connection" used in this application have the meaning of conduction. The specific meaning needs to be understood in conjunction with the context.

[0039] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0040] To facilitate understanding, some basic concepts of terms or technologies involved in the embodiments of the present invention are first briefly introduced and explained.

[0041] As mentioned above, traditional cooking methods waste users' time and are not convenient or healthy enough.

[0042] Based on this, an embodiment of the present application provides a household appliance, including: an image acquisition device, used to acquire an image of a target area, where a stove and a pot are placed; a controller, configured to: respond to a boiling detection instruction initiated by a user, obtain a first image of the pot in a cooking state through the image acquisition device; input the first image into a boiling detection model to obtain a detection result of the pot; wherein the detection result includes whether the pot has a lid or the pot has no lid and whether the soup in the pot has boiled or the soup in the pot has not boiled; when the detection result is that the pot has no lid, if the soup in the pot has boiled, lower the firepower level of the stove.

[0043] This reduces the user's workload, makes the cooking process healthier and safer, and allows users to enjoy a more convenient and high-quality cooking experience.

[0044] The embodiments provided in this application are described in detail below in conjunction with the accompanying drawings.

[0045] Figure 1 A hardware structure diagram of a household appliance system provided in an embodiment of the present application, such as Figure 1 As shown, the household appliance system 1 includes a household appliance 2, a stove 3 and a cooker 4, wherein the household appliance 2, the stove 3 and the cooker 4 are communicatively connected with each other.

[0046] In some embodiments, the household appliance 2 may be a robot. The present application uses the robot to detect the state of the soup in the pot.

[0047] It should be noted that the embodiments of the present application use a home appliance as a robot for illustration. In a specific implementation, the home appliance may also be any other possible product, and the present application does not limit this.

[0048] In some embodiments, the household appliance 2 includes an image acquisition device 21 and a controller 1000 ( Figure 1 not shown).

[0049] In some embodiments, the image acquisition device 21 is used to acquire images of a target area, where the target area includes stoves and pots.

[0050] Optionally, the image acquisition device 21 may be a camera, and the image acquisition device 21 may be set at any position of the robot.

[0051] It should be noted that the image acquisition device 21 may also be other devices that can be used to acquire images, and this application does not limit this.

[0052] In some embodiments, the robot has a mapping and navigation module, which is responsible for building a point cloud map of the user's residence and navigating to the specified coordinates.

[0053] It should be noted that mapping is based on simultaneous localization and mapping (SLAM) technology. To turn on the mapping function mode, the user needs to push the robot to move around the house to cover the entire house as much as possible. Residence space identification points can also be set during the mapping process.

[0054] For example, after being pushed to the kitchen, manually or by voice, set that this is the kitchen, and the robot will record the current coordinate information name as the kitchen.

[0055] Furthermore, the navigation function automatically plans the motion path based on the established spatial map information and controls the robot to move to the designated location.

[0056] In some embodiments, the robot also has a voice announcement module for issuing voice reminders to the user.

[0057] In some embodiments, the cooker 3 is used to perform cooking tasks.

[0058] In some embodiments, the pot 4 is used to hold cooked soup and cooked side dishes.

[0059] In some embodiments, the pot 4 includes a pot cover 41 .

[0060] In some embodiments, Figure 2 As shown, the cooker 4 is also provided with a temperature sensor 42 for detecting the gas temperature in the cooker.

[0061] In the embodiment shown in the present application, the controller 1000 refers to a device that can generate an operation control signal according to the instruction operation code and the timing signal to instruct the household appliance 2 to execute the control instruction. Exemplarily, the controller 1000 can be a central processing unit (CPU), a general-purpose processor network processor (NP), a digital signal processor (DSP), a microprocessor, a microcontroller, a programmable logic device (PLD) or any combination thereof. The controller 1000 can also be other devices with processing functions, such as circuits, devices or software modules, and the embodiments of the present application do not impose any restrictions on this.

[0062] In addition, the controller 1000 can be used to control various components inside the household appliance 2 so that each component operates to realize various predetermined functions of the household appliance 2.

[0063] In some embodiments, the controller 1000 is configured to: in response to a boiling detection instruction initiated by a user, obtain a first image of the cooker in a cooking state through an image acquisition device; input the first image into a boiling detection model to obtain a detection result of the cooker; wherein the detection result includes whether the cooker has a lid or has no lid and whether the soup in the cooker has boiled or has not boiled; when the detection result is that the cooker has no lid, if the soup in the cooker has boiled, lower the firepower level of the cooker.

[0064] In some embodiments, the controller 1000 is further configured to: when the detection result shows that the pot has no lid, if the soup in the pot is boiling and no side dishes have been added to the pot, lower the fire level of the stove and send a reminder to the user to add side dishes; after the user adds side dishes to the pot, increase the fire level of the stove.

[0065] In some embodiments, the controller 1000 is further configured to: acquire at least one frame image in the first image; input at least one frame image into a boiling detection model to obtain a prediction score for at least one frame image; wherein the prediction score includes a soup boiling prediction score, a pot lid presence prediction score, and a side dish addition prediction score; and determine the detection result of the cookware based on the prediction score.

[0066] In some embodiments, the controller 1000 is further configured to: when the cooker has a lid, periodically obtain the gas temperature in the cooker at preset time intervals through a temperature sensor; determine whether the soup in the cooker has boiled based on the gas temperature in the cooker in a current cycle and the gas temperature in the cooker in a first cycle; wherein the first cycle is the previous cycle of the current cycle.

[0067] In some embodiments, the controller 1000 is further configured to: determine that the soup in the pot has boiled when the gas temperature in the pot in the current cycle is greater than or equal to a preset temperature threshold and the gas temperature in the pot in the first cycle is less than the preset temperature threshold; and determine that the soup in the pot has not boiled when the gas temperature in the pot in the current cycle is less than the preset temperature threshold.

[0068] Figure 3 FIG. 2 is a hardware configuration block diagram of a household appliance 2 provided by the present application according to an exemplary embodiment. Figure 3 As shown, the household appliance 2 may further include the following two items: a memory 1002 and a communication interface 1003 .

[0069] In some embodiments, the memory 1002 can be used to store software programs and data. The controller 1000 executes various functions and data processing of the household appliance 2 by running the software programs or data stored in the memory 1002. The memory 1002 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. The memory 1002 stores an operating system that enables the household appliance 2 to run. In the present application, the memory 1002 can store an operating system and various application programs, and can also store code for executing the boiling detection method provided in the embodiment of the present application.

[0070] In some embodiments, the communication interface 1003 is used to establish a communication connection with other network entities, for example, to establish a communication connection with a terminal device. The communication interface 1003 may include a radio frequency (RF) module, a cellular module, a wireless fidelity (WIFI) module, and a GPS module. Taking the RF module as an example, the RF module can be used to receive and send signals, in particular, to send the received information to the controller 1000 for processing; in addition, the signal generated by the controller 1000 is sent out. Typically, the RF circuit may include but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, and the like.

[0071] Those skilled in the art will understand that Figure 3 The hardware structure shown in the figure does not constitute a limitation on the home appliance. The home appliance may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0072] In some embodiments, the electronic device for executing the boiling detection method provided in the embodiments of the present application can be a mobile phone, a tablet computer, a desktop, a laptop, a handheld computer, a notebook computer, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook, and a cellular phone, a personal digital assistant (personal digital assistant, PDA), an augmented reality (augmented reality, AR)\virtual reality (virtual reality, VR) device, etc. The present invention does not impose any special restrictions on the specific form of the electronic device. It can interact with the user through one or more methods such as a keyboard, a touchpad, a touch screen, a remote control, a voice interaction or a handwriting device.

[0073] For example, Figure 4 The hardware structure diagram of an electronic device provided by an embodiment of the present invention is shown in FIG. Figure 4 As shown, the electronic device 10 includes a processor 101, a memory 102, a network interface 103, and the like.

[0074] Among them, the processor 101 is the core component of the electronic device 10, and the processor 101 is used to run the operating system of the electronic device 10 and the applications on the electronic device 10 (including system applications and third-party applications) to implement the defect detection method of the electronic device 10.

[0075] In the embodiment of the present invention, the processor 101 may be a central processing unit (CPU).

[0076] processing unit, CPU), microprocessor, digital signal processor (digital signal processor)

[0077] A processor may be a computer programmable logic device (CPU), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a transistor logic device, a hardware component or any combination thereof, which can implement or execute the various exemplary logic blocks, modules and circuits described in conjunction with the contents disclosed in the embodiments of the present invention; a processor may also be a combination that implements a computing function, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0078] Optionally, the processor 101 of the electronic device 10 includes one or more CPUs, which are single-core CPUs (single-CPU) or multi-core CPUs (multi-CPU).

[0079] In some embodiments, the memory 102 includes, but is not limited to, a random access memory (RAM), a read only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, or an optical memory, etc. The memory 102 stores operating system code.

[0080] Optionally, the processor 101 implements the boiling detection method in the embodiment of the present invention by reading instructions stored in the memory 102, or the processor 101 implements the boiling detection method provided by the embodiment of the present invention through internally stored instructions. In the case where the processor 101 implements the boiling detection method provided by the embodiment of the present invention by reading instructions stored in the memory, the memory stores instructions for implementing the boiling detection method provided by the embodiment of the present invention.

[0081] The network interface 103 is a wired interface, such as a fiber distributed data interface (FDDI) or a gigabit Ethernet (GE) interface. Alternatively, the network interface 103 is a wireless interface. The network interface 103 is used for the electronic device 10 to communicate with other devices.

[0082] In some embodiments, the memory 102 is used to store multiple images to be identified included in each of the at least one scanned image. The at least one processor 101 further executes the method described in the embodiment of the present invention according to the multiple images to be identified included in each of the at least one scanned image stored in the memory 102. For more details on how the processor 101 implements the above functions, please refer to the description of each method embodiment below.

[0083] Optionally, the electronic device 10 further includes a bus 104, and the processor 101 and the memory 102 are connected to each other via the bus 104, or are connected to each other in other ways.

[0084] Optionally, the electronic device 10 further includes an input / output interface 105, which is used to connect to an input device and receive a control request input by a user through the input device. The input device includes but is not limited to a keyboard, a touch screen, a microphone, etc. The input / output interface 105 is also used to connect to an output device and output the control result of the processor 101. The output device includes but is not limited to a display, a printer, etc.

[0085] Figure 5 A schematic diagram of a boiling detection method provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, the method comprises the following steps:

[0086] S101. In response to a boiling detection instruction initiated by a user, a controller obtains a first image of a cooker in a cooking state.

[0087] Optionally, the controller may acquire a first image of the cookware in a cooking state through an image acquisition device.

[0088] In some embodiments, after the user initiates a boiling detection instruction to the robot, the robot navigates to the kitchen through the mapping and navigation module, and turns on the image acquisition device to acquire a first image of the target area.

[0089] Among them, the target areas include stoves and pots.

[0090] It should be noted that the image acquisition technology of the image acquisition device is developed using the universal serial bus video class (UVC) protocol. The UVC protocol is a protocol standard specifically defined for USB video capture devices. The image acquisition device that supports the UVC protocol supports infrared fill light when the light is insufficient, so that barrier-free monitoring can be achieved in home rooms when the light is insufficient.

[0091] Figure 6 A schematic flow chart of another boiling detection method provided in an embodiment of the present application is used to drive the opening of an image acquisition device, such as Figure 6 As shown, when the USB device is initialized (unit), the relevant information of the USB device is obtained, and after obtaining the permission to drive the image acquisition device in the permission algorithm (onAttach) in the USB callback event, the image acquisition device is opened through the opening algorithm (onConnect).

[0092] Furthermore, the frame preview subfunction (onFrame) in the image acquisition device callback class (IFrameCallback) obtains the real-time captured image information in the ByteBuffer format, and draws the image information in this format to the preview function (SurfaceView) to realize the screen preview of the image acquisition device.

[0093] In some embodiments, when boiling detection needs to obtain a first image, the image information in ByteBuffer format is first converted into image information in NV21 format, and then the image information in NV21 format is converted into image information in Bitmap format, and then the image information in Bitmap format is converted into image information in Mat format during inference in the neural network forward computing framework.

[0094] S102: The controller inputs the first image into a boiling detection model to obtain a detection result of the cookware.

[0095] The test results include whether the pot has a lid or does not have a lid and whether the soup in the pot has boiled or has not boiled.

[0096] In some embodiments, the boiling detection model is obtained by training. Figure 7 A flow chart of a model training method provided in an embodiment of the present application is used to train a boiling detection model, such as Figure 7 As shown, the method comprises the following steps:

[0097] S201: Obtain a training sample set.

[0098] The training sample set includes a plurality of samples carrying labels, the plurality of samples carrying labels include boiling image data of sample pots, and the labels include boiling data sets of sample pots.

[0099] In some embodiments, the boil detection model employs object detection methods within the field of deep learning.

[0100] It should be understood that the characteristics of boiling when the user is cooking soup are that the soup in the pot rolls and hot air bubbles come out, so boiling can be detected based on the changing characteristics of the soup.

[0101] It should be noted that since there is no international open source boiling data set for reference, we collected pictures of the boiling soup in the pot and the lid of the pot to form the training sample set and the validation set.

[0102] Optionally, the training sample set can be 1,000.

[0103] S202: Train the boiling detection model according to the training sample set to obtain a trained boiling detection model.

[0104] Optionally, the model framework of the boiling training model can adopt the target detection framework YoloV7.

[0105] Furthermore, the obtained training sample set is used as a label, and the collected picture of the boiling soup in the pot is used as input data.

[0106] Optionally, the collected pictures of the boiling soup in the pot can be saved in a preset folder (such as a Train folder), and the label data format of each picture can be set to a text file (.txt).

[0107] Among them, the .txt file contains the input integer type (int) and floating point type (float) values ​​of the specific area and point of the label (label), and the folder path of the picture is specified in the file of file type yaml in the data (data) in the YoloV7 detection framework.

[0108] In some embodiments, when the boiling detection model starts training, when executing the Train.py part of the code in the Train folder, the corresponding image data and label data can be loaded into the project through the dataset loading interface (create_dataloader) of the YoloV7 detection framework.

[0109] Furthermore, the image features are extracted through the feature extraction module (backbone). When the loss function (loss) is calculated, the categories of the boiling soup in the pot and the pot cover are classified using the classification idea, and the position positioning is trained and learned using the regression idea.

[0110] In some embodiments, taking a 3060ti graphics card with a batch_size of 32 images loaded at a time, an epoch (epochs) of 500, and a learning rate (lr) of an initial value as an example, the parameters of the YoloV7s model are used as pre-training parameters, fine-tinning training is performed, and training is performed using an open source deep learning framework (Pytorch). When the epochs training is completed, a boiling detection model after training is finally obtained.

[0111] In some embodiments, after the boiling detection model is trained, in order to protect user privacy, an artificial intelligence (AI) end-side deployment method is adopted. However, the model trained by the Pytorch framework cannot be directly inferred on the Android side of the robot system, so a neural network forward computing framework is adopted.

[0112] In some embodiments, pictures of pots with lids and pictures of side dishes added to the pot can also be collected to train the boiling detection model, and then the boiling area features, pot lid features, and side dish features are calibrated through a label marking tool, and boiling, pot lid, and side dishes are marked with different labels, so that the boiling detection model can be used to detect whether the pot has a lid and whether side dishes are added to the pot. The specific training method is the same as the above training steps and will not be repeated here.

[0113] Figure 8 A schematic flow chart of another boiling detection method provided in an embodiment of the present application is used to deploy a boiling detection model in a robot, such as Figure 8 As shown, first, the boiling detection model needs to be converted from the original format (.pt) to the TorchScript format by setting the end2end, simplify, iou, and threshhold parameters through export.py in YoloV7, and then the boiling detection model in TorchScript format is converted into a neural network forward computing framework model through the pytorch model deployment method (pytorch neuralnetwork exchange, PNNX).

[0114] It should be noted that when used in the terminal, the fundamental purpose of model loading is to load information in the .param format and the .bin file format into the target neural network (an ncnn::Net structure).

[0115] Among them, the .param format file describes the structure of the neural network, including the layer name, layer input and output information, and layer parameter information (such as the kernel size of the convolutional layer, etc.).

[0116] Furthermore, the .bin format file records the data information required for neural network operations (such as the weights and bias information of the convolutional layer). When deployed, the boiling detection model is deployed in the robot by using the include and .a / .so in the neural network forward computing framework, and then the relevant APIs opened by the neural network forward computing framework are called to load the model and perform inference and prediction.

[0117] In some embodiments, when the boiling detection model is successfully deployed in the robot, the controller inputs the acquired first image into the boiling detection model to obtain a detection result of the cookware. Fig. 9 A schematic flow chart of another boiling detection method provided in an embodiment of the present application, which is used to determine the detection result of a cookware, such as Fig. 9 As shown, the method comprises the following steps:

[0118] S301. A controller obtains at least one frame image in a first image.

[0119] It should be noted that the first image includes at least one frame image. After acquiring the first image, the controller needs to separate the frame images in the first image and input the separated frame images into the boiling detection model.

[0120] S302: The controller inputs at least one frame image into a boiling detection model to obtain a prediction score of at least one frame image.

[0121] Among them, the prediction scores include the prediction score for soup boiling, the prediction score for the presence of a pot lid, and the prediction score for the addition of side dishes.

[0122] Specifically, the soup boiling prediction score is used to indicate whether the soup in the pot has boiled, the pot lid presence prediction score is used to indicate whether the pot has a pot lid, and the side dish addition prediction score is used to indicate whether the side dish has been added to the pot.

[0123] In some embodiments, after the controller inputs at least one frame image into the boiling detection model, the boiling detection model outputs a result.

[0124] Optionally, the specific form of the output result can be [x, y, w, h, label, score].

[0125] Among them, x and y are the coordinates of the upper left corner of the detection area rectangle, w and h are the width and height of the detection area rectangle, label is the type number of the detection object, and score is the model's predicted score for the detection object.

[0126] Optionally, the types of detected objects include soup, pot cover and side dishes.

[0127] For example, the label of the soup is 0, the label of the pot lid is 1, and the label of the side dish is 2.

[0128] Optionally, the prediction score of the detection object has a value interval of [0, 1].

[0129] S303: The controller determines the detection result of the cookware according to the predicted score.

[0130] In some embodiments, when the predicted score of the soup 0 is greater than or equal to a preset score, it is determined that the detection result of the cooker is that the soup in the cooker has boiled.

[0131] It should be noted that the preset score is pre-set by the home appliance manufacturer and stored in the memory. The preset scores set by different manufacturers may be different, and this application does not limit this.

[0132] Optionally, the preset score may be 0.65.

[0133] Specifically, when the predicted score of soup body 0 is greater than or equal to 0.65, it is determined that the soup in the pot has boiled; if the predicted score of soup body 0 is less than 0.65, it is determined that the detection result of the pot is that the soup in the pot has not boiled.

[0134] In some embodiments, when the predicted score of the pot cover 1 is greater than or equal to the preset score, the detection result of the pot is determined to be that the pot has a pot cover.

[0135] Specifically, when the predicted score of pot cover 1 is greater than or equal to 0.65, the detection result of the pot is determined to be that the pot has a pot cover; if the predicted score of pot cover 1 is less than 0.65, the detection result of the pot is determined to be that the pot has no pot cover.

[0136] In some embodiments, when the predicted score of the side dish 2 is greater than or equal to the preset score, it is determined that the detection result of the cookware is that the side dish has been added into the cookware.

[0137] Specifically, when the predicted score of side dish 2 is greater than or equal to 0.65, the detection result of the pot is determined to be that the pot has a lid; if the predicted score of side dish 2 is less than 0.65, the detection result of the pot is determined to be that the side dish has been added into the pot.

[0138] S103: When the detection result shows that the pot has no lid, if the soup in the pot has boiled, the controller lowers the fire level of the cooker.

[0139] In some embodiments, when the detection result is that the pot has no lid, the robot can directly determine through the first image whether the soup in the pot is boiling. If the soup in the pot is boiling, the controller lowers the fire level of the stove to avoid overflowing or drying out the pot.

[0140] In some embodiments, when the detection result is that the pot has no lid, if the soup in the pot is boiling and no side dishes are added to the pot, the fire level of the stove is lowered, the robot will move in the user's residence, and send a voice reminder to the user to add side dishes through the text to speech interface (TTS) in the robot's voice module (for example: "The soup is boiling, please add side dishes in time!").

[0141] Furthermore, after the user adds side dishes into the pot, the fire level of the stove is increased until the boiling of the soup in the pot is detected again. The controller then lowers the fire level of the stove, stews the ingredients in the pot for a preset time, and then turns off the stove.

[0142] It should be noted that the preset duration is pre-set by the home appliance manufacturer and stored in the memory. The preset durations set by different manufacturers may be different, and this application does not limit this.

[0143] Optionally, the preset duration may be 15 minutes.

[0144] Specifically, after the controller lowers the fire level of the stove, the ingredients in the pot are stewed for 15 minutes, the stove is turned off, the cooking task is ended, and the cooking of the ingredients is completed.

[0145] In some embodiments, when the detection result shows that the pot has no lid, if the soup in the pot is boiling and side dishes have been added to the pot, lower the fire level of the stove, stew the ingredients in the pot for 15 minutes, turn off the stove, end the cooking task, and complete the cooking of the ingredients.

[0146] In some embodiments, when the cooking task is completed, the image acquisition device needs to be turned off. Fig.10 A schematic flow chart of another boiling detection method provided in an embodiment of the present application, which is used to shut down an image acquisition device, such as Fig.10 As shown, when the USB device is initialized (unit), the relevant information of the USB device is obtained, and after obtaining the permission to close the image acquisition device in the permission algorithm (onAttach) in the USB callback event, the image acquisition device is closed through the closing algorithm (onDetach) or the disconnection algorithm (onDisconnect).

[0147] The technical solution provided by the embodiment of the present application brings at least the following beneficial effects: the technical solution obtains the image of the pot during the cooking process through the image acquisition device, and in the case that the pot in the image does not have a pot cover, determines whether the soup has boiled according to the image of the soup in the pot, and automatically lowers the fire level of the stove when the soup in the pot has boiled, so as to avoid the phenomenon of overflowing the pot. The method can effectively monitor the state of the soup and the entire cooking process of the soup cooked using the stove, and autonomously adjust the fire level of the gas stove, so that the workload of the user is reduced, and there is no need to frequently monitor the cooking process. The entire cooking process is healthier and safer, and the user can enjoy a more convenient and high-quality cooking experience.

[0148] In some embodiments, when the detection result shows that the pot has a lid, the state of the soup in the pot cannot be obtained through the first image. At this time, a temperature sensor in the pot can be used to determine whether the soup in the pot has boiled. Fig.11 A flow chart of another boiling detection method provided in an embodiment of the present application is used to determine the state of the soup in the pot according to the temperature sensor, such as Fig.11 As shown, the method comprises the following steps:

[0149] S401. The controller obtains the gas temperature in the cooker.

[0150] Optionally, the controller may periodically obtain the gas temperature in the cooker at preset time intervals via a temperature sensor.

[0151] It should be noted that the preset time interval is pre-set by the home appliance manufacturer and stored in the memory. The preset time intervals set by different manufacturers may be different, and this application does not limit this.

[0152] Optionally, the preset time interval may be 1 second.

[0153] Specifically, the controller may periodically obtain the gas temperature in the cooker at intervals of 1 second through the temperature sensor.

[0154] Optionally, the temperature sensor may be a K-type thermocouple, and the present application does not limit the type of the temperature sensor.

[0155] In some embodiments, the temperature sensor is a K-type thermocouple, for example, which is disposed inside the pot cover to collect the gas temperature inside the pot. The K-type thermocouple is connected to a data conversion circuit board, which has a Bluetooth module, so the gas temperature inside the pot collected by the K-type thermocouple is transmitted to the robot end using the Bluetooth low energy (BLE) protocol.

[0156] S402: The controller determines whether the soup in the pot has boiled according to the gas temperature in the pot in the current cycle and the gas temperature in the pot in the first cycle.

[0157] The first cycle is the cycle before the current cycle.

[0158] Fig.12 A flow chart of another boiling detection method provided in an embodiment of the present application, which is used to determine whether the soup in the pot has boiled. Fig.12 As shown, the method comprises the following steps:

[0159] S4021. When the gas temperature in the cooker in the current cycle is greater than or equal to a preset temperature threshold, and the gas temperature in the cooker in the first cycle is less than the preset temperature threshold, the controller determines that the soup in the cooker has boiled.

[0160] It should be noted that the preset temperature threshold is pre-set by the home appliance manufacturer and stored in the memory. The preset temperature threshold set by different manufacturers may be different, and this application does not limit this.

[0161] Optionally, the preset temperature threshold may be 97°C.

[0162] For example, when the gas temperature T in the cooker of the current cycle 1 is greater than or equal to the preset temperature threshold, and the gas temperature T in the cooker in the first cycle 2 When the temperature is lower than the preset temperature threshold, it is determined that the soup in the pot has boiled.

[0163] Specifically, when T 1 ≥97℃ and T 2 When the temperature is less than 97℃, make sure the soup in the pot is boiling.

[0164] In some embodiments, when it is determined that the soup in the pot has boiled, the controller lowers the fire level of the cooker.

[0165] S4022: When the gas temperature in the pot in the current cycle is lower than a preset temperature threshold, the controller determines that the soup in the pot is not boiling.

[0166] Specifically, when T 1 When the temperature is less than 97℃, make sure the soup in the pot is not boiling.

[0167] In some embodiments, the method for identifying the state of the soup in the pot by using a temperature sensor can also be performed as follows: Fig.13 The process shown is implemented as Fig.13 As shown, the K-type thermocouple transmits the acquired gas temperature data to the robot's application (Robot App) through the OnStart, Connect, and Notify methods in the BLE protocol of the Bluetooth module on the conversion board. The robot's application analyzes the gas temperature change pattern (PreframeTemp<100CurrFrameTemp>100) and takes action based on the change pattern (cookStateMachineFIRE_LEVEL_WATER_BOILED).

[0168] The embodiment of the present invention can divide the functional modules of the electronic device etc. according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present invention is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0169] In the case of dividing each functional module into corresponding functional modules, Fig.14 A possible structural diagram of the boiling detection device involved in the above embodiment is shown. Fig.14 As shown, the boiling detection device 40 may include: an acquisition module 401 and a processing module 402 .

[0170] In some embodiments, the acquisition module 401 is used to acquire a first image of the cookware in a cooking state through an image acquisition device in response to a boiling detection instruction initiated by a user.

[0171] In some embodiments, the processing module 402 is used to input the first image into a boiling detection model to obtain a detection result of the cookware.

[0172] In some embodiments, the processing module 402 is further configured to, when the detection result shows that the pot has no lid, lower the fire level of the stove if the soup in the pot has boiled.

[0173] In some embodiments, the processing module 402 is also used to, when the detection result shows that the pot has no lid, if the soup in the pot is boiling and no side dishes are added to the pot, lower the fire level of the stove and send a reminder to the user to add side dishes.

[0174] In some embodiments, the processing module 402 is further used to increase the fire level of the stove after the user adds side dishes into the pot.

[0175] In some embodiments, the acquisition module 401 is further used to acquire a training sample set.

[0176] In some embodiments, the processing module 402 is further used to train the boiling detection model according to the training sample set to obtain the boiling detection model after training.

[0177] In some embodiments, the acquisition module 401 is further used to acquire at least one frame image in the first image.

[0178] In some embodiments, the processing module 402 is further used to input at least one frame image into the boiling detection model to obtain a prediction score of the at least one frame image.

[0179] In some embodiments, the processing module 402 is further configured to determine the detection result of the cookware according to the predicted score.

[0180] In some embodiments, the acquisition module 401 is further used to periodically acquire the gas temperature in the cooker at preset time intervals through a temperature sensor when the cooker has a lid.

[0181] In some embodiments, the processing module 402 is further used to determine whether the soup in the pot has boiled according to the gas temperature in the pot in the current cycle and the gas temperature in the pot in the first cycle.

[0182] In some embodiments, the processing module 402 is also used to determine that the soup in the pot has boiled when the gas temperature in the pot in the current cycle is greater than or equal to a preset temperature threshold and the gas temperature in the pot in the first cycle is less than the preset temperature threshold.

[0183] In some embodiments, the processing module 402 is further used to determine that the soup in the pot is not boiling when the gas temperature in the pot in the current cycle is less than a preset temperature threshold.

[0184] An embodiment of the present invention further provides a computer-readable storage medium, which includes computer-executable instructions. When the computer-executable instructions are executed on a computer, the computer executes the method provided in the above embodiment.

[0185] An embodiment of the present invention further provides a computer program product, which can be directly loaded into a memory and contains software code. After being loaded and executed by a computer, the computer program product can implement the method provided in the above embodiment.

[0186] Those skilled in the art will appreciate that in one or more of the above examples, the functions described in the present invention may be implemented in hardware, software, firmware, or any combination thereof. When implemented using software, the functions may be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any media that facilitates the transmission of a computer program from one place to another. The storage medium may be any available medium that a general or special-purpose computer can access.

[0187] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0188] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely exemplary, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The unit described as a separate component may or may not be physically separated, and the component displayed as a unit may be a physical unit or multiple physical units, that is, it may be located in one place, or it may be distributed in multiple different places. Some or all of the units can be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0189] In addition, each functional unit in each embodiment of the present invention can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a ROM, a RAM, a disk, or an optical disk.

[0190] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A household appliance, characterized in that: include: An image acquisition device, used to acquire an image of a target area, where stoves and pots are placed; The controller is configured as: In response to a boiling detection instruction initiated by a user, acquiring a first image of the cookware in a cooking state through the image acquisition device; Inputting the first image into a boiling detection model to obtain a detection result of the pot; wherein the detection result includes whether the pot has a lid or the pot has no lid and whether the soup in the pot has boiled or the soup in the pot has not boiled; When the detection result shows that the pot has no lid, if the soup in the pot has boiled, lower the fire level of the stove.

2. The household appliance according to claim 1, characterized in that: The detection result also includes whether a side dish has been added into the pot or whether a side dish has not been added into the pot; The controller is further configured to: In the case where the detection result is that the pot has no lid, if the soup in the pot has boiled and no side dishes have been added to the pot, lower the fire level of the stove and send a reminder to the user to add side dishes; After the user adds side dishes into the pot, the fire level of the stove is increased.

3. The household appliance according to claim 1, characterized in that: The boiling detection model is trained by the following method: Acquire a training sample set; the training sample set includes a plurality of samples carrying labels, the plurality of samples carrying labels include boiling image data of sample pots, and the labels include boiling data sets of the sample pots; The boiling detection model is trained according to the training sample set to obtain the boiling detection model after training.

4. The household appliance according to claim 2, characterized in that: The controller is configured to input the first image into a boiling detection model to obtain a detection result of the soup in the pot, and is specifically configured as follows: Acquire at least one frame image of the first image; Inputting the at least one frame image into the boiling detection model to obtain a prediction score of the at least one frame image; wherein the prediction score includes a soup boiling prediction score, a pot cover presence prediction score, and a side dish addition prediction score; The detection result of the cookware is determined according to the predicted score.

5. The household appliance according to claim 1, characterized in that: The household appliance further comprises: A temperature sensor, used to detect the gas temperature in the cooker; The controller is further configured to: In the case where the cooker has a cover, the temperature of the gas in the cooker is periodically acquired at preset time intervals by the temperature sensor; Whether the soup in the pot has boiled is determined based on the gas temperature in the pot in the current cycle and the gas temperature in the pot in the first cycle; wherein the first cycle is a cycle before the current cycle.

6. The household appliance according to claim 5, characterized in that: The controller is configured to determine whether the soup in the pot has boiled according to the gas temperature in the pot in the current cycle and the gas temperature in the pot in the first cycle, and is specifically configured as follows: When the gas temperature in the pot in the current cycle is greater than or equal to a preset temperature threshold, and the gas temperature in the pot in the first cycle is less than the preset temperature threshold, determining that the soup in the pot has boiled; When the gas temperature in the pot in the current cycle is less than the preset temperature threshold, it is determined that the soup in the pot is not boiling.

7. A boiling detection method, characterized in that: The method comprises: In response to a boiling detection instruction initiated by a user, acquiring a first image of the cookware in a cooking state; Inputting the first image into a boiling detection model to obtain a detection result of the pot; wherein the detection result includes whether the pot has a lid or the pot has no lid and whether the soup in the pot has boiled or the soup in the pot has not boiled; When the detection result shows that the pot has no lid, if the soup in the pot has boiled, lower the fire level of the stove.

8. The method according to claim 7, characterized in that The detection result also includes whether a side dish has been added into the pot or whether a side dish has not been added into the pot; The method further comprises: In the case where the detection result is that the pot has no lid, if the soup in the pot has boiled and no side dishes have been added to the pot, lower the fire level of the stove and send a reminder to the user to add side dishes; After the user adds side dishes into the pot, the fire level of the stove is increased.

9. The method according to claim 8, characterized in that The step of inputting the first image into a boiling detection model to obtain a detection result of the soup in the pot includes: Acquire at least one frame image of the first image; Inputting the at least one frame image into the boiling detection model to obtain a prediction score of the at least one frame image; wherein the prediction score includes a soup boiling prediction score, a pot cover presence prediction score, and a side dish addition prediction score; The detection result of the cookware is determined according to the predicted score.

10. The method according to claim 7, characterized in that The method further comprises: When the cooker has a cover, obtaining the gas temperature in the cooker; Whether the soup in the pot has boiled is determined based on the gas temperature in the pot in the current cycle and the gas temperature in the pot in the first cycle; wherein the first cycle is a cycle before the current cycle.

Images