Method and device for detecting cooking state, electronic equipment and medium

Abstract

This application discloses a method, apparatus, electronic device, and medium for detecting cooking status. Specifically, upon determining that a target cooking device is being started, cooking voice data and cooking image data are acquired during the device's operation. This cooking data is then input into a cooking detection model to obtain a cooking status recognition result. When the cooking status recognition result corresponds to an abnormal state, a corresponding processing event is executed. By applying the technical solution of this application, when the cooking device is detected to be started, image and sound data of the cooking device can be acquired in real time. A pre-trained neural network model is then used to automatically determine the semantic information and correlations of the cooking image and voice data, thereby achieving the purpose of automatically determining the cooking device's status. This avoids phenomena such as boiling over and burning that easily occur when users use cooking devices in related technologies.

Description

Technical Field

[0001] This application relates to data processing technology, and in particular to a method, apparatus, electronic device, and medium for detecting cooking status. Background Technology

[0002] With the rise of the communication era and society, smart devices have continued to evolve as more and more users adopt them.

[0003] With the increasing prevalence of internet-connected and information-enabled home appliances in people's lives, and the development of society and the economy, people are pursuing higher quality homes, leading to a stronger demand for comfortable and intelligent living. The smart kitchen is a crucial component of the smart home ecosystem, and intelligent cooking is paramount within it. However, during the cooking process, users frequently encounter issues such as food spilling or burning. This poses safety hazards when using a smart kitchen, thus reducing the user experience. Summary of the Invention

[0004] This application provides a method, device, electronic device, and medium for detecting cooking status. This application aims to solve the problems of boiling over and burning that easily occur when users use cooking equipment to cook food in related technologies.

[0005] According to one aspect of the embodiments of this application, a method for detecting cooking state is provided, characterized in that it includes:

[0006] The target cooking device is activated, and cooking data to be identified during the operation of the target cooking device is collected. The cooking data to be identified includes corresponding cooking voice data and cooking image data.

[0007] The cooking data to be identified is input into the cooking detection model to obtain the cooking state identification result;

[0008] When the cooking state identification result is determined to correspond to an abnormal state, the corresponding processing event is executed.

[0009] Optionally, in another embodiment based on the method described above in this application, the step of collecting the cooking data to be identified during the operation of the target cooking device includes:

[0010] Acquire an image of the cooking equipment to characterize the state of the target cooking equipment, wherein the cooking equipment image data corresponds to at least one color feature and a contour feature;

[0011] Based on the image of the cooking device, it is determined that the target cooking device is in the start-up state, and the cooking data to be identified of the target cooking device is obtained.

[0012] Optionally, in another embodiment based on the method described above in this application, the step of inputting the cooking data to be identified into the cooking detection model to obtain the cooking state identification result includes:

[0013] Acquire at least one frame of cooking image data and at least one segment of cooking audio data corresponding to the cooking image data;

[0014] The cooking images are input into the cooking detection model to obtain the first cooking feature; and the cooking voice data is input into the cooking detection model to obtain the second cooking feature.

[0015] The first cooking feature and the second cooking feature are spliced ​​and merged to obtain the first merged cooking feature;

[0016] The first combined cooking feature is subjected to feature classification and location regression to obtain the second combined cooking feature.

[0017] The cooking state recognition result is obtained by performing a non-maximum suppression operation on the second merged cooking feature.

[0018] Optionally, in another embodiment based on the method described above in this application, the step of inputting the cooking data to be identified into the cooking detection model to obtain the cooking state identification result includes:

[0019] Obtain the material information of the target cooking equipment;

[0020] Based on the cooking image data of at least one frame, the cooking voice data of at least one segment corresponding to the cooking image data, and the material information of the target cooking device, a cooking state recognition result is obtained.

[0021] Optionally, in another embodiment based on the method described above in this application, before inputting the cooking data to be identified into the cooking detection model, the method further includes:

[0022] Acquire at least one sample data generated for a sample cooking device, wherein the sample data includes sample cooking audio and sample cooking images;

[0023] The sample cooking image is annotated with a first content, which includes the coordinates of the cooking equipment location box and the cooking status result;

[0024] The sample cooking audio is annotated with a second content, which includes the cooking status result.

[0025] The cooking detection model is generated based on sample cooking images carrying the first content annotation and sample cooking audio carrying the second content annotation.

[0026] Optionally, in another embodiment based on the method described above in this application, generating the cooking detection model based on the sample cooking image carrying the first content annotation and the sample cooking speech carrying the second content annotation includes:

[0027] The sample cooking image carrying the first content annotation is input into the image training module of the preset image semantic segmentation model;

[0028] In addition, the sample cooking speech carrying the second content annotation is input into the speech training module of the image semantic segmentation model;

[0029] When the training meets the preset conditions, the image semantic segmentation model that meets the preset conditions is used as the cooking detection model.

[0030] Optionally, in another embodiment based on the method described above in this application, after using the image semantic segmentation model that meets the preset conditions as the cooking detection model, the following steps are included:

[0031] Deploy the cooking detection model onto the target cooking device; or...

[0032] The cooking detection model was deployed to a cloud server.

[0033] Optionally, in another embodiment based on the method described above, when determining that the cooking state identification result corresponds to an abnormal state, executing the corresponding processing event includes:

[0034] If the cooking status identification result corresponds to an abnormal state, play an alarm audio, or generate a shutdown command, the shutdown command being used to turn off the start-up state of the target cooking device.

[0035] According to another aspect of the embodiments of this application, a cooking state detection device is provided, comprising:

[0036] The determination module is configured to determine the start of the target cooking device and collect the cooking data to be identified during the operation of the target cooking device. The cooking data to be identified includes corresponding cooking voice data and cooking image data.

[0037] The generation module is configured to input the cooking data to be identified into the cooking detection model to obtain the cooking state identification result;

[0038] The execution module is configured to execute the corresponding processing event when it is determined that the cooking state identification result corresponds to an abnormal state.

[0039] According to another aspect of the embodiments of this application, an electronic device is provided, comprising:

[0040] Memory, used to store executable instructions;

[0041] A display is provided for displaying the executable instructions to the memory to perform the operation of any of the above-described cooking state detection methods.

[0042] According to another aspect of the embodiments of this application, a computer-readable storage medium is provided for storing computer-readable instructions, which, when executed, perform the operation of any of the above-described cooking state detection methods.

[0043] In this application, upon determining that a target cooking device is to be started, cooking data to be identified during the operation of the target cooking device is acquired. This cooking data includes corresponding cooking voice data and cooking image data. The cooking data to be identified is input into a cooking detection model to obtain a cooking state identification result. When it is determined that the cooking state identification result corresponds to an abnormal state, a corresponding processing event is executed. By applying the technical solution of this application, when the start of the cooking device is detected, image and sound data of the cooking device can be collected in real time, and a pre-trained neural network model can be used to automatically determine the semantic information and correlation of the cooking image and voice data. This achieves the purpose of automatically determining the state of the cooking device, thereby avoiding phenomena such as boiling over and burning that easily occur when users use cooking devices in related technologies.

[0044] The technical solution of this application will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0045] The accompanying drawings, which form part of this specification, illustrate embodiments of this application and, together with the description, serve to explain the principles of this application.

[0046] This application can be more clearly understood with reference to the accompanying drawings and the following detailed description, wherein:

[0047] Figure 1 This is a schematic diagram of the cooking status detection system proposed in this application;

[0048] Figure 2 This is a schematic diagram of a cooking state detection method proposed in this application;

[0049] Figure 3 A display diagram showing images captured by the cooking equipment proposed in this application;

[0050] Figure 4 This is an architecture diagram of the cooking detection model for acquiring images using cooking equipment proposed in this application;

[0051] Figure 5This is a flowchart illustrating the overall operation of the cooking equipment proposed in this application;

[0052] Figure 6 This is a schematic diagram of the cooking state detection device of this application;

[0053] Figure 7 This is a schematic diagram of the electronic device structure shown in this application. Detailed Implementation

[0054] Various exemplary embodiments of the present application will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present application.

[0055] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0056] The following description of at least one exemplary embodiment is merely illustrative and is not intended to limit the scope of this application or its application or use.

[0057] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0058] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0059] Furthermore, the technical solutions of the various embodiments of this application can be combined with each other, but only if they are based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by this application.

[0060] It should be noted that all directional indicators (such as up, down, left, right, front, back, etc.) in the embodiments of this application are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicator will also change accordingly.

[0061] The following is combined with Figures 1-5 This application describes a method for detecting cooking state according to exemplary embodiments thereof. It should be noted that the following application scenarios are shown only to facilitate understanding of the spirit and principles of this application, and the embodiments of this application are not limited in any way. Rather, the embodiments of this application can be applied to any applicable scenario.

[0062] Figure 1 A schematic diagram of an exemplary system architecture 100 for detecting cooking state, which can be applied to embodiments of this application, is shown.

[0063] like Figure 1 As shown, system architecture 100 may include one or more of cooking devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between cooking devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0064] It should be understood that Figure 1 The number of cooking devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of cooking devices, networks, and servers can be included. For example, server 105 could be a server cluster consisting of multiple servers.

[0065] Users can use cooking devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, capture images, etc. Cooking devices 101, 102, and 103 may include various electronic devices with displays and camera capture devices.

[0066] The cooking devices 101, 102, and 103 in this application can be cooking devices that provide various services. For example, a user can use cooking device 103 (or cooking device 101 or 102) to: determine and start a target cooking device; collect cooking data to be identified during the operation of the target cooking device, including corresponding cooking voice data and cooking image data; input the cooking data to be identified into a cooking detection model to obtain a cooking state recognition result; and when the cooking state recognition result corresponds to an abnormal state, execute the corresponding processing event.

[0067] It should be noted that the cooking state detection method provided in this application embodiment can be executed by one or more of the cooking devices 101, 102, 103, and / or the server 105. Accordingly, the cooking state detection device provided in this application embodiment is generally set in the corresponding cooking device and / or the server 105, but this application is not limited thereto.

[0068] Furthermore, this application also proposes a method, apparatus, target terminal, and medium for detecting cooking status.

[0069] Figure 2 A schematic flowchart illustrating a method for detecting cooking state according to an embodiment of this application is shown. Figure 2 As shown, the method includes:

[0070] S101, determine to start the target cooking device, and obtain the cooking data to be identified during the operation of the target cooking device. The cooking data to be identified includes the corresponding cooking voice data and cooking image data.

[0071] Furthermore, with the development of the information society, more and more network and information appliances are appearing in people's lives. The development of social and economic levels has led people to pursue higher and higher quality of home life, and the demand for home comfort and intelligence is becoming stronger and stronger.

[0072] The smart kitchen is a crucial component of the entire smart home ecosystem, and smart cooking is the core of the smart kitchen. A key approach to achieving smart cooking is to deploy cameras on the range hood to capture images of the cooktop, and then analyze these images using detection methods based on deep convolutional neural networks. Deep learning, relying on massive amounts of data, identifies and judges various information at each stage of the cooking process, including different ingredients, human postures, and movements. Combined with big data statistics, it provides users with accurate cooking identification, various warning prompts, and safer and healthier cooking suggestions. However, the complexity of cooking, the diversity of cookware, cooktops, ingredients, individual habits, and environmental complexities all contribute to the challenges of smart cooking.

[0073] Furthermore, this application proposes a deep learning method that combines sound and image to address anomaly detection in cooking equipment, such as pot boiling over or burning. Understandably, pot boiling over, burning, or overflowing frequently occurs during cooking, posing significant risks to food, the kitchen, and even personal safety. Therefore, this application captures both images and sound signals simultaneously, using deep learning to allow the network to automatically combine the features of both, thus improving the accuracy and precision of anomaly detection in cooking equipment.

[0074] Furthermore, this application can collect cooking voice data and cooking image data in real time or periodically during the operation of the cooking device after determining that the target cooking device has been started.

[0075] It should be noted that this application does not specifically limit the cooking equipment; for example, it can be a cooking equipment that uses fire as a medium or a cooking equipment that uses electricity as a medium.

[0076] Furthermore, this application does not specifically limit the quantity of cooking voice data and cooking image data. For example, cooking image data can be one image or multiple images. As for cooking voice data, it can be one audio segment or multiple audio segments.

[0077] Furthermore, this application can acquire cooking voice data and cooking image data through a data acquisition device installed in the cooking equipment. For example, cooking image data can be acquired through a camera installed in the cooking equipment, and cooking voice data can be acquired through a microphone device installed in the cooking equipment.

[0078] S102, input the cooking data to be identified into the cooking detection model to obtain the cooking state identification result.

[0079] Furthermore, when cooking equipment experiences phenomena such as boiling over, burning, or overflowing, the image quality is significantly affected by the steam and the adhesion of oil fumes to the camera. Additionally, when cooking with a lid on, the lid obstructs the view inside the pot, making it impossible for the camera to detect and identify phenomena like boiling over.

[0080] Understandably, when cooking equipment malfunctions, such as boiling over, burning, or overflowing, it often results in issues like steam constantly impacting the lid, causing it to vibrate, or liquid overflowing from the pot while accompanied by a hissing sound from the steam. Figure 3 As shown, this application can improve the accuracy and precision of detecting abnormalities in cooking equipment by inputting cooking voice data and cooking image data into a pre-trained cooking detection model and using deep learning to allow the network to automatically combine the features of both.

[0081] This application does not specifically limit the cooking detection model. For example, it can be a Convolutional Neural Network (CNN). A CNN is a type of feedforward neural network that includes convolutional computation and has a deep structure; it is one of the representative algorithms of deep learning. CNNs possess representation learning capabilities, enabling them to perform translation-invariant classification of input information according to their hierarchical structure. Thanks to the powerful feature representation capabilities of CNNs for images, they have achieved remarkable results in image classification, object detection, and semantic segmentation. It should be noted that after obtaining the cooking detection model, this application can deploy it to the target cooking device or to a cloud server.

[0082] Furthermore, this application can use a cooking detection model to detect feature information in multiple cooking images captured by the camera device of a cooking equipment, and then perform feature recognition on the feature information to determine whether the image to be recognized contains target features (e.g., features corresponding to flames, features corresponding to smoke, features of water droplets, etc.). Specifically, the cooking image to be recognized needs to be input into a preset cooking detection model, and the output of the last fully connected layer (FC) of the cooking detection model is used as the recognition result of the feature data corresponding to the image to be recognized.

[0083] S103, when it is determined that the cooking status recognition result corresponds to an abnormal status, the corresponding processing event is executed.

[0084] In one approach, for example, when the cooking status identification result of the cooking device corresponds to an abnormal state such as burning or boiling over, this application can play an alarm audio or generate a shutdown command to turn off the target cooking device. Furthermore, for example, when the cooking device is a fire-based cooking device, the shutdown command can be used to instruct the device to turn off the flame; and when the cooking device is an electric cooking device, the shutdown command can be used to instruct the device to disconnect the power.

[0085] In this application, upon determining that a target cooking device is to be started, cooking data to be identified during the operation of the target cooking device is acquired. This cooking data includes corresponding cooking voice data and cooking image data. The cooking data to be identified is input into a cooking detection model to obtain a cooking state identification result. When it is determined that the cooking state identification result corresponds to an abnormal state, a corresponding processing event is executed. By applying the technical solution of this application, when the start of the cooking device is detected, image and sound data of the cooking device can be collected in real time, and a pre-trained neural network model can be used to automatically determine the semantic information and correlation of the cooking image and voice data. This achieves the purpose of automatically determining the state of the cooking device, thereby avoiding phenomena such as boiling over and burning that easily occur when users use cooking devices in related technologies.

[0086] Optionally, in one possible implementation of this application, S102 (acquiring the cooking data to be identified during the operation of the target cooking device) includes:

[0087] Acquire an image of the cooking equipment to characterize the state of the target cooking equipment, wherein the cooking equipment image data corresponds to at least one color feature and contour feature;

[0088] Based on the image of the cooking equipment, it is determined that the target cooking equipment is in the start-up state, and the cooking data to be identified of the target cooking equipment is obtained.

[0089] Furthermore, before collecting cooking voice data and cooking image data during the operation of the target cooking device, this application needs to first determine that the cooking device has been activated by the user for cooking. For example, a cooking detection model can be deployed to the cooking device first, then the image acquisition device of the cooking device can be turned on, and the acquired cooking device image, which represents the state of the target cooking device, can be transmitted to the device backend. After the device backend determines that the target cooking device is in the activated state based on the cooking device image, it can then begin collecting cooking voice data and cooking image data.

[0090] It should be noted that this application does not specifically limit the images of cooking equipment; for example, it may be one image or multiple images.

[0091] Optionally, in one possible implementation of this application, step S102 (inputting the cooking data to be identified into the cooking detection model to obtain the cooking state identification result) includes:

[0092] Acquire at least one frame of cooking image data, and at least one segment of cooking audio data corresponding to the cooking image data;

[0093] The cooking images are input into the cooking detection model to obtain the first cooking feature; and the cooking voice data is input into the cooking detection model to obtain the second cooking feature.

[0094] The first cooking feature and the second cooking feature are combined and merged to obtain the first merged cooking feature;

[0095] The first combined cooking feature is subjected to feature classification and location regression to obtain the second combined cooking feature.

[0096] The second merged cooking feature is subjected to nonmaximum suppression to obtain the cooking state recognition result.

[0097] Furthermore, the overall design of the cooking detection model established in this application is as follows: Figure 4 As shown, the cooking detection model can be divided into two parts: an image detection network based on images and a speech feature extraction network based on sound. The features of the two networks are fused together before detection.

[0098] Furthermore, this application can input the synchronization signals of cooking audio data and cooking image data into the cooking detection model. It should be noted, however, that in this application, the cooking image corresponds to at least one frame of the image detection network file at a certain moment, while the cooking audio data can be a signal over a continuous time period; therefore, the temporal dimensions of the two are not consistent.

[0099] In one approach, this application first extracts the image detection network corresponding to a certain time t. Then, this application predefines a time span, such as the sound signal corresponding to a time length one second before the current time. The image detection network is then input into the image detection network, and the extracted high-order semantic information is saved first, with the corresponding feature dimension being W1xH1xC1 (i.e., the first cooking feature).

[0100] It should be noted that C1 refers to the output channel, W1 refers to the feature width, and H1 refers to the feature height. Depending on the deep learning framework, the arrangement of the channels may vary. In one approach, the channel can be placed in the last dimension. The corresponding audio signal is then synchronously input into the speech feature extraction network. Since the audio signal is one-dimensional, the corresponding feature dimension is W2x1xC2 (i.e., the second cooking feature).

[0101] Furthermore, this application can perform a global average pooling (GAP) operation on the feature (W2x1xC2, i.e., the second cooking feature) to become 1x1xC2. Then, a broadcasting operation is performed on the audio feature of the 1x1xC2 feature to finally become W1xH1xC2, which is the final fused audio feature.

[0102] Furthermore, this application can also perform a splicing and fusion operation on the feature information of the image detection network (i.e., the first cooking feature, W1xH1xC1) and the feature information of the sound (i.e., the second cooking feature, W1xH1xC2), and finally become the dimension W1xH1x(C1+C2) (i.e. the first merged cooking feature).

[0103] Furthermore, this application can also classify and regress the features in the first merged cooking feature (W1xH1x(C1+C2)) dimension to obtain the second merged cooking feature, and finally perform non-maximum suppression (NMS) operation on the second merged cooking feature to obtain the cooking state recognition result.

[0104] The cooking detection model proposed in this application mainly consists of two parts: a speech module and a graphics module. These modules can be arbitrarily replaced. For example, the graphics module can use anchor-based algorithms such as YOLO, SSD, RetinaNet, Faster R-CNN, and other single-stage or two-stage detection algorithms; or it can use anchor-free algorithms such as CornerNet, CenterNet, and FCOS. Similarly, for the speech module, any speech feature extraction model can be selected. In short, the entire model architecture is highly flexible, allowing for the reasonable allocation or selection of suitable modules based on the requirements of actual application deployment, and flexible replacement to meet the needs of the scenario.

[0105] Optionally, this application inputs the cooking data to be identified into a cooking detection model to obtain cooking state identification results, including:

[0106] Obtain the material information of the target cooking equipment;

[0107] The cooking state recognition result is obtained based on at least one frame of cooking image data, at least one segment of cooking voice data corresponding to the cooking image data, and the material information of the target cooking equipment.

[0108] This application can obtain cooking status recognition results not only based on cooking image data and at least one segment of cooking voice data corresponding to the cooking image data, but also based on the material parameters of the cooking equipment. It is understandable that when the cooking equipment is determined to be made of iron, it is less likely to experience abnormal cooking status conditions compared to aluminum equipment.

[0109] Optionally, in one possible implementation of this application, before S102 (inputting the cooking data to be identified into the cooking detection model), the following step is further included:

[0110] Acquire at least one sample data generated for a sample cooking device, wherein the sample data includes sample cooking voice and sample cooking images;

[0111] The sample cooking images are annotated with the first content, which includes the coordinates of the cooking equipment location box and the cooking status result.

[0112] The sample cooking audio is annotated with a second content, which includes the cooking status result.

[0113] A cooking detection model is generated based on sample cooking images with first content annotations and sample cooking audio with second content annotations.

[0114] Furthermore, this application can select cooking devices of various materials for cooking, while simultaneously recording related video and audio data. This results in at least one sample data point generated by the corresponding cooking device.

[0115] Furthermore, this application can sample cooking images by extracting images at given time intervals. The sample cooking images are then manually annotated, with the first annotation including the coordinates of the pot's location and whether a cooking state such as boiling over has occurred. Next, for the sample cooking audio, sounds from the corresponding time point and a given time span prior are selected as audio data. The selected audio data is then annotated, with the second annotation indicating whether a cooking state such as boiling over has occurred.

[0116] Finally, this application pairs cooking image data with corresponding audio data. Once one data point is labeled as an abnormal state such as a pot boiling over, the entire set of sample data is defined as abnormal. If neither data point is an abnormal state like a pot boiling over, the entire set of sample data is labeled as normal, thus forming a sample dataset of version images and audio. A cooking detection model is then generated for the sample cooking image carrying the first content annotation and the sample cooking audio carrying the second content annotation.

[0117] Optionally, this application generates a cooking detection model based on sample cooking images carrying first content annotations and sample cooking audio carrying second content annotations, including:

[0118] The sample cooking images carrying the first content annotation are input into the image training module of the preset image semantic segmentation model;

[0119] In addition, the sample cooking speech carrying the second content annotation is input into the speech training module of the image semantic segmentation model;

[0120] When the training meets the preset conditions, the image semantic segmentation model that meets the preset conditions will be used as the cooking detection model.

[0121] Specifically, a batch of training sample data can be randomly selected, and the sound and image can be input into the corresponding speech module and image detection module respectively to complete feature extraction and fusion. Training is then performed based on the corresponding "cooking pot" label. Once the training meets preset conditions, the image semantic segmentation model that satisfies these conditions is used as the cooking detection model.

[0122] This application can identify sample features (such as size features, contour features, color features, etc.) of at least one region of an object in a sample image using a pre-defined image semantic segmentation model. Furthermore, the image semantic segmentation model can also classify various image features and speech features in the sample image, and group sample features belonging to the same category into objects of the same type. Thus, the multiple sample features obtained after semantic segmentation of the sample image can be sample features composed of multiple different types.

[0123] It should be noted that when a neural network image classification model performs semantic segmentation on sample images, the more accurately it classifies the pixels in the sample image, the higher the accuracy of identifying the labeled objects in the sample image. It should also be noted that the preset conditions can be custom-defined.

[0124] For example, a preset condition can be set such that the classification accuracy of pixels reaches 70% or more. Then, multiple sample images are used to repeatedly train the image detection and classification model. When the classification accuracy of the neural network image classification model reaches 70% or more, the image detection and classification model can be applied to the embodiments of this application to perform feature recognition on multiple cooking images captured by the camera device in the cooking equipment.

[0125] Further, optionally, after using the image semantic segmentation model that meets the preset conditions as the cooking detection model in this application, the following is included:

[0126] The cooking detection model can be deployed to the target cooking device; or, the cooking detection model can be deployed to a cloud server. Understandably, the cooking device can choose whether to perform the cooking status detection itself or let the server perform it, based on its own load status.

[0127] Furthermore, such as Figure 5 As shown, the abnormal state of the cooking equipment is described as a boil-over state.

[0128] 1: Select ingredients that are prone to boiling over when cooking, and record relevant video and audio data at the same time.

[0129] 2: Sample the video file and extract images at given time intervals.

[0130] 3: Manually annotate the sampled images, including the coordinates of the pot's location and whether it has boiled over.

[0131] 4: For audio files, select the sound at the corresponding time point and the sound over a given time span before it as the sound data.

[0132] 5: Label the selected sound data, indicating whether it is a boil or not.

[0133] 6: Pair the image data with the corresponding sound data. Once one is labeled as "pooping," the data set is defined as "pooping." If neither is "pooping," the data set is labeled as "normal." In this way, this application constitutes a pooping sample dataset integrating images and sounds.

[0134] 7. Randomly select a batch of training sample data, and input the sound and image into the corresponding speech module and image detection module respectively to complete feature extraction and fusion. (See the algorithm flow of the overall architecture in the previous section for details.)

[0135] 8: Train according to the corresponding "pooping" label.

[0136] 9. If the predefined loss error or the pre-specified number of iterations is not reached, return to step 7 for the next batch of data training; otherwise, continue.

[0137] 10: If the termination condition is met at this point, the training ends.

[0138] 11: Save the network model.

[0139] 12: Deploy the network model to the range hood.

[0140] 13: Enable image and voice sensors

[0141] 14: Pass the image to the backend

[0142] 15: Detect whether cooking is occurring based on the image.

[0143] 16: If not, skip to 14; otherwise, continue.

[0144] 17: This is the cooking mode, where images and voice are simultaneously transmitted to the model deployed in this application.

[0145] 18: Check for overcooking. If overcooking occurs, immediately sound an alarm or automatically shut off the heat. Otherwise, continue.

[0146] 19: If no boiling over is detected, check if cooking is still in progress. If cooking is still in progress, skip to 17; otherwise, continue.

[0147] 20: Cooking is now complete. Skip to 15.

[0148] By applying the technical solution of this application, when the cooking equipment is detected to be started, the image and sound data of the cooking equipment can be collected in real time, and a pre-trained neural network model can be used to automatically determine the semantic information and correlation of the cooking image and sound data, thereby achieving the purpose of automatically determining the status of the cooking equipment. This avoids the problems of boiling over and burning that users may encounter when using the cooking equipment to cook food, which are common in related technologies.

[0149] In another embodiment of this application, such as Figure 6 As shown, this application also provides a cooking state detection device. The device includes a determination module 201, a generation module 202, and an execution module 203, wherein...

[0150] The determination module 201 is configured to determine the start of the target cooking device and collect the cooking data to be identified during the operation of the target cooking device. The cooking data to be identified includes corresponding cooking voice data and cooking image data.

[0151] The generation module 202 is configured to input the cooking data to be identified into the cooking detection model to obtain the cooking state identification result;

[0152] The execution module 203 is configured to execute the corresponding processing event when it is determined that the cooking state identification result corresponds to an abnormal state.

[0153] In this application, upon determining that a target cooking device is to be started, cooking data to be identified during the operation of the target cooking device is acquired. This cooking data includes corresponding cooking voice data and cooking image data. The cooking data to be identified is input into a cooking detection model to obtain a cooking state identification result. When it is determined that the cooking state identification result corresponds to an abnormal state, a corresponding processing event is executed. By applying the technical solution of this application, when the start of the cooking device is detected, image and sound data of the cooking device can be collected in real time, and a pre-trained neural network model can be used to automatically determine the semantic information and correlation of the cooking image and voice data. This achieves the purpose of automatically determining the state of the cooking device, thereby avoiding phenomena such as boiling over and burning that easily occur when users use cooking devices in related technologies.

[0154] In another embodiment of this application, the determining module 201 further includes:

[0155] The determining module 201 is configured to acquire a cooking device image for characterizing the state of the target cooking device, wherein the cooking device image data corresponds to at least one color feature and a contour feature;

[0156] The determination module 201 is configured to determine, based on the image of the cooking device, that the target cooking device is in the start state, and to acquire the cooking data to be identified of the target cooking device.

[0157] In another embodiment of this application, the determining module 201 further includes:

[0158] The determining module 201 is configured to acquire the cooking image data of at least one frame and the cooking voice data of at least one segment corresponding to the cooking image data;

[0159] The determining module 201 is configured to input the number of cooking images into the cooking detection model to obtain a first cooking feature; and to input the cooking voice data into the cooking detection model to obtain a second cooking feature;

[0160] The determining module 201 is configured to splice and merge the first cooking feature and the second cooking feature to obtain a first merged cooking feature;

[0161] The determination module 201 is configured to perform feature classification and location regression operations on the first combined cooking features to obtain the second combined cooking features;

[0162] The determination module 201 is configured to perform a non-maximum suppression operation on the second merged cooking features to obtain the cooking state recognition result.

[0163] In another embodiment of this application, the determining module 201 further includes:

[0164] The determination module 201 is configured to acquire the material information of the target cooking device;

[0165] The determining module 201 is configured to obtain a cooking state recognition result based on the at least one frame of cooking image data, at least one segment of cooking voice data corresponding to the cooking image data, and the material information of the target cooking device.

[0166] In another embodiment of this application, the determining module 201 further includes:

[0167] The determining module 201 is configured to acquire at least one sample data generated for the sample cooking device, wherein the sample data includes sample cooking voice and sample cooking images;

[0168] The determination module 201 is configured to perform a first content annotation on the sample cooking image, the first content annotation including the coordinates of the cooking device location box and the cooking status result;

[0169] The determination module 201 is configured to perform a second content annotation on the sample cooking voice, the second content annotation including the cooking status result;

[0170] The determination module 201 is configured to generate the cooking detection model based on sample cooking images carrying the first content annotation and sample cooking speech carrying the second content annotation.

[0171] In another embodiment of this application, the determining module 201 further includes:

[0172] The determination module 201 is configured to input the sample cooking image carrying the first content annotation into the image training module of the preset image semantic segmentation model;

[0173] The determination module 201 is configured to input the sample cooking speech carrying the second content annotation into the speech training module in the image semantic segmentation model.

[0174] The determination module 201 is configured to determine that when the training reaches the preset conditions, the image semantic segmentation model that meets the preset conditions is used as the cooking detection model.

[0175] In another embodiment of this application, the determining module 201 further includes:

[0176] The determination module 201 is configured to deploy the cooking detection model into the target cooking device; or,

[0177] The determination module 201 is configured to deploy the cooking detection model to a cloud server.

[0178] In another embodiment of this application, the determining module 201 further includes:

[0179] The determination module 201 is configured to determine that the cooking state identification result corresponds to an abnormal state, play an alarm audio, or generate a shutdown command, the shutdown command being used to shut down the start-up state of the target cooking device.

[0180] Figure 7 This is a logical structure block diagram of an electronic device according to an exemplary embodiment. For example, the electronic device 300 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0181] Reference Figure 7 The electronic device 300 may include one or more of the following components: processor 301 and memory 302.

[0182] Processor 301 may include one or more processing cores, such as a quad-core processor or an octa-core processor. Processor 301 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 301 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 301 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 301 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0183] The memory 302 may include one or more computer-readable storage media, which may be non-transitory. The memory 302 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 302 are used to store at least one instruction, which is executed by the processor 301 to implement the interactive effects calibration method provided in the method embodiments of this application.

[0184] In some embodiments, the electronic device 300 may optionally include a peripheral device interface 303 and at least one peripheral device. The processor 301, memory 302, and peripheral device interface 303 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 303 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 304, a touch display screen 305, a camera 306, an audio circuit 307, a positioning component 308, and a power supply 309.

[0185] The peripheral device interface 303 can be used to connect at least one I / O (Input / Output) related peripheral device to the processor 301 and the memory 302. In some embodiments, the processor 301, memory 302, and peripheral device interface 303 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 301, memory 302, and peripheral device interface 303 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0186] The radio frequency (RF) circuit 304 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 304 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 304 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 304 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 304 can communicate with other terminals through at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: metropolitan area networks (MANs), various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks (WLANs), and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 304 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0187] Display screen 305 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 305 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 301 for processing. In this case, display screen 305 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 305, which serves as the front panel of the electronic device 300; in other embodiments, there may be at least two display screens, respectively disposed on different surfaces of the electronic device 300 or in a folded design; in still other embodiments, display screen 305 may be a flexible display screen, disposed on a curved or folded surface of the electronic device 300. Furthermore, display screen 305 may be configured as a non-rectangular, irregular shape, i.e., a non-rectangular screen. Display screen 305 may be made of materials such as LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode).

[0188] The camera assembly 306 is used to acquire images or videos. Optionally, the camera assembly 306 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 306 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm-light flash and a cool-light flash, which can be used for light compensation at different color temperatures.

[0189] The audio circuit 307 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 301 for processing, or input to the radio frequency circuit 304 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each located in a different part of the electronic device 300. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from the processor 301 or the radio frequency circuit 304 into sound waves. The speaker may be a conventional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 307 may also include a headphone jack.

[0190] Positioning component 308 is used to locate the current geographic location of electronic device 300 for navigation or LBS (Location Based Service). Positioning component 308 can be a positioning component based on the US GPS (Global Positioning System), China's BeiDou system, Russia's Granas system, or the European Union's Galileo system.

[0191] The power supply 309 is used to supply power to the various components in the electronic device 300. The power supply 309 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When the power supply 309 includes a rechargeable battery, the rechargeable battery can support wired or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0192] In some embodiments, the electronic device 300 further includes one or more sensors 410. The one or more sensors 410 include, but are not limited to: an accelerometer 411, a gyroscope 412, a pressure sensor 413, a fingerprint sensor 414, an optical sensor 415, and a proximity sensor 416.

[0193] Accelerometer 411 can detect the magnitude of acceleration on the three coordinate axes of a coordinate system established by electronic device 300. For example, accelerometer 411 can be used to detect the components of gravitational acceleration on the three coordinate axes. Processor 301 can control touch screen 305 to display the user interface in landscape or portrait view based on the gravitational acceleration signal acquired by accelerometer 411. Accelerometer 411 can also be used for games or for acquiring user motion data.

[0194] The gyroscope sensor 412 can detect the orientation and rotation angle of the electronic device 300. The gyroscope sensor 412, in conjunction with the accelerometer sensor 411, can collect 3D motion data from the user on the electronic device 300. Based on the data collected by the gyroscope sensor 412, the processor 301 can perform the following functions: motion sensing (e.g., changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0195] The pressure sensor 413 can be disposed on the side bezel of the electronic device 300 and / or on the lower layer of the touch display screen 305. When the pressure sensor 413 is disposed on the side bezel of the electronic device 300, it can detect the user's grip signal on the electronic device 300, and the processor 301 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 413. When the pressure sensor 413 is disposed on the lower layer of the touch display screen 305, the processor 301 can control the operable controls on the UI interface based on the user's pressure operation on the touch display screen 305. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0196] The fingerprint sensor 414 is used to collect a user's fingerprint. The processor 301 identifies the user's identity based on the fingerprint collected by the fingerprint sensor 414, or the fingerprint sensor 414 identifies the user's identity based on the collected fingerprint. When the user's identity is identified as trusted, the processor 301 authorizes the user to perform relevant sensitive operations, including unlocking the screen, viewing encrypted information, downloading software, making payments, and changing settings. The fingerprint sensor 414 can be located on the front, back, or side of the electronic device 300. When the electronic device 300 has physical buttons or a manufacturer's logo, the fingerprint sensor 414 can be integrated with the physical buttons or manufacturer's logo.

[0197] An optical sensor 415 is used to collect ambient light intensity. In one embodiment, the processor 301 can control the display brightness of the touch screen 305 based on the ambient light intensity collected by the optical sensor 415. Specifically, when the ambient light intensity is high, the display brightness of the touch screen 305 is increased; when the ambient light intensity is low, the display brightness of the touch screen 305 is decreased. In another embodiment, the processor 301 can also dynamically adjust the shooting parameters of the camera assembly 306 based on the ambient light intensity collected by the optical sensor 415.

[0198] The proximity sensor 416, also known as a distance sensor, is typically located on the front panel of the electronic device 300. The proximity sensor 416 is used to detect the distance between the user and the front of the electronic device 300. In one embodiment, when the proximity sensor 416 detects that the distance between the user and the front of the electronic device 300 is gradually decreasing, the processor 301 controls the touch display screen 305 to switch from a screen-on state to a screen-off state; when the proximity sensor 416 detects that the distance between the user and the front of the electronic device 300 is gradually increasing, the processor 301 controls the touch display screen 305 to switch from a screen-off state to a screen-on state.

[0199] Those skilled in the art will understand that Figure 7 The structure shown does not constitute a limitation on the electronic device 300, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0200] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 302 including instructions. These instructions can be executed by a processor 420 of an electronic device 300 to complete the cooking state detection method described above. The method includes: determining to start a target cooking device; collecting cooking data to be identified during the operation of the target cooking device, the cooking data to be identified including corresponding cooking voice data and cooking image data; inputting the cooking data to be identified into a cooking detection model to obtain a cooking state identification result; and executing a corresponding processing event when the cooking state identification result corresponds to an abnormal state. Optionally, the instructions can also be executed by the processor 420 of the electronic device 300 to complete other steps involved in the exemplary embodiment described above. For example, the non-transitory computer-readable storage medium can be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0201] In an exemplary embodiment, an application / computer program product is also provided, including one or more instructions that can be executed by the processor 420 of the electronic device 300 to complete the cooking state detection method described above. The method includes: determining to start a target cooking device; collecting cooking data to be identified during the operation of the target cooking device, the cooking data to be identified including corresponding cooking voice data and cooking image data; inputting the cooking data to be identified into a cooking detection model to obtain a cooking state identification result; and executing a corresponding processing event when the cooking state identification result corresponds to an abnormal state. Optionally, the above instructions can also be executed by the processor 420 of the electronic device 300 to complete other steps involved in the exemplary embodiment described above.

[0202] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0203] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for detecting cooking state, characterized in that, include: The target cooking device is activated, and cooking data to be identified during the operation of the target cooking device is collected. The cooking data to be identified includes corresponding cooking voice data and cooking image data. The cooking voice data includes the sound of the pot lid vibrating and / or the sound of steam beeping when the target cooking device is in an abnormal operating state. The cooking data to be identified is input into the cooking detection model to obtain the cooking state identification result; When the cooking state identification result is determined to correspond to an abnormal state, the corresponding processing event is executed; The step of inputting the cooking data to be identified into the cooking detection model to obtain the cooking state identification result includes: Acquire at least one frame of cooking image data, and at least one segment of cooking audio data corresponding to the cooking image data; The cooking images are input into the cooking detection model to obtain the first cooking feature; and the cooking voice data is input into the cooking detection model to obtain the second cooking feature. The first cooking feature and the second cooking feature are combined and merged to obtain the first merged cooking feature; The first combined cooking feature is subjected to feature classification and location regression to obtain the second combined cooking feature; The cooking state recognition result is obtained by performing a non-maximum suppression operation on the second merged cooking feature.

2. The method as described in claim 1, characterized in that, The process of collecting the cooking data to be identified during the operation of the target cooking device includes: Acquire an image of the cooking equipment to characterize the state of the target cooking equipment, wherein the cooking equipment image data corresponds to at least one color feature and a contour feature; Based on the image of the cooking device, it is determined that the target cooking device is in the start-up state, and the cooking data to be identified of the target cooking device is obtained.

3. The method as described in claim 2, characterized in that, The step of inputting the cooking data to be identified into the cooking detection model to obtain the cooking state identification result includes: Obtain the material information of the target cooking equipment; Based on the cooking image data of at least one frame, the cooking voice data of at least one segment corresponding to the cooking image data, and the material information of the target cooking device, a cooking state recognition result is obtained.

4. The method as described in claim 1, characterized in that, Before inputting the cooking data to be identified into the cooking detection model, the method further includes: Acquire at least one sample data generated for a sample cooking device, wherein the sample data includes sample cooking audio and sample cooking images; The sample cooking image is annotated with a first content, which includes the coordinates of the cooking equipment location box and the cooking status result; The sample cooking audio is annotated with a second content, which includes the cooking status result. The cooking detection model is generated based on sample cooking images carrying the first content annotation and sample cooking audio carrying the second content annotation.

5. The method as described in claim 4, characterized in that, The step of generating the cooking detection model based on sample cooking images carrying the first content annotation and sample cooking audio carrying the second content annotation includes: The sample cooking image carrying the first content annotation is input into the image training module of the preset image semantic segmentation model; In addition, the sample cooking speech carrying the second content annotation is input into the speech training module of the image semantic segmentation model; When the training meets the preset conditions, the image semantic segmentation model that meets the preset conditions is used as the cooking detection model.

6. The method as described in claim 5, characterized in that, After using the image semantic segmentation model that meets the preset conditions as the cooking detection model, the following steps are included: Deploy the cooking detection model onto the target cooking device; or... The cooking detection model was deployed to a cloud server.

7. The method as described in claim 1 or 6, characterized in that, When it is determined that the cooking state identification result corresponds to an abnormal state, the corresponding processing event is executed, including: If the cooking status identification result corresponds to an abnormal state, play an alarm audio, or generate a shutdown command, the shutdown command being used to turn off the start-up state of the target cooking device.

8. A device for detecting cooking state, characterized in that, include: The determination module is set to determine the start of the target cooking device and collect the cooking data to be identified during the operation of the target cooking device. The cooking data to be identified includes corresponding cooking voice data and cooking image data. The cooking voice data is used to collect the sound of the pot lid vibrating and / or the sound of steam beeping when the target cooking device is in an abnormal operating state. The generation module is configured to input the cooking data to be identified into the cooking detection model to obtain the cooking state identification result; The execution module is configured to execute the corresponding processing event when it is determined that the cooking state identification result corresponds to an abnormal state; The generation module is also configured to perform: Acquire at least one frame of cooking image data, and at least one segment of cooking audio data corresponding to the cooking image data; The number of cooking images is input into the cooking detection model to obtain the first cooking feature; The cooking voice data is then input into the cooking detection model to obtain a second cooking feature. The first cooking feature and the second cooking feature are spliced ​​and merged to obtain the first merged cooking feature; The first combined cooking feature is subjected to feature classification and location regression to obtain the second combined cooking feature. The cooking state recognition result is obtained by performing a non-maximum suppression operation on the second merged cooking feature.

9. An electronic device, characterized in that, include: Memory, used to store executable instructions; A processor, configured to communicate with the memory to execute the executable instructions to perform the operation of the cooking state detection method according to any one of claims 1-7.

10. A computer-readable storage medium for storing computer-readable instructions, characterized in that, When the instruction is executed, it performs the operation of the cooking state detection method according to any one of claims 1-7.

Images