Cooking prompting method and device, equipment and storage medium

Abstract

The invention provides a cooking prompting method and device, electronic equipment and a storage medium, relates to the technical field of artificial intelligence, and is applied to a cooking scenario. The method comprises the following steps: obtaining table top temperature field image data and table top temperature matrix data; carrying out feature extraction on the table top temperature field image data and the table top temperature matrix data to obtain food material feature information and temperature field feature information; fusing the cooking food material feature information and the temperature field feature information to obtain cooking fusion feature information; the cooking fusion feature information is input into a maturity prediction model, target maturity information is obtained, and the target maturity information is used for indicating the maturity of the detected food material; and generating first indication information based on the target maturity information, wherein the first indication information is used for indicating a cooking stage corresponding to the target maturity information. According to the invention, the cooking process can be automatically identified, the prompt information is output, and the intelligent level of cooking is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a cooking prompt method, device, equipment and storage medium. Background Art

[0002] Currently, smart cooking devices in kitchens often include automatic or assisted cooking functions. Users select a smart recipe on the terminal, and the smart cooking device assists them with cooking based on the selected recipe through voice prompts. However, smart recipes are typically static, and the smart cooking device cannot provide feedback or jump to the next step based on user operation. Users must identify the current step and adjust the cooking timing themselves, which complicates the cooking process and results in poor results. Summary of the Invention

[0003] The present application provides a cooking prompt method, device, equipment and storage medium, which can significantly improve the accuracy and reliability of each stage of the cooking process.

[0004] In one aspect, the present application provides a cooking prompt method, comprising:

[0005] Acquire table image data and table temperature matrix data;

[0006] Performing feature extraction on the table image data and the table temperature matrix data respectively to obtain food feature information and temperature field feature information;

[0007] Fusing the food feature information and the temperature field feature information to obtain cooking fusion feature information;

[0008] Inputting the cooking fusion feature information into a maturity prediction model to obtain target maturity information, wherein the target maturity information is used to indicate the maturity of the detected food;

[0009] First indication information is generated based on the target maturity information, where the first indication information is used to indicate a cooking stage corresponding to the target maturity information.

[0010] Another aspect provides a cooking prompt device, the device comprising:

[0011] A table data acquisition module is used to acquire table image data and table temperature matrix data;

[0012] A feature extraction module is used to extract features from the table image data and the table temperature matrix data respectively to obtain food feature information and temperature field feature information;

[0013] a feature fusion module, configured to fuse the food feature information and the temperature field feature information to obtain cooking fusion feature information;

[0014] a target maturity information prediction module, configured to input the cooking fusion feature information into a maturity prediction model to obtain target maturity information, wherein the target maturity information is used to indicate the maturity of the detected food;

[0015] The first indication module is configured to generate first indication information based on the target maturity information, where the first indication information is used to indicate a cooking stage corresponding to the target maturity information.

[0016] On the other hand, a cooking reminder device is provided, which includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the above-mentioned cooking reminder method.

[0017] On the other hand, a computer-readable storage medium is provided, in which at least one instruction or at least one program is stored. The at least one instruction or the at least one program is loaded and executed by a processor to implement the cooking prompt method as described above.

[0018] On the other hand, a server is provided, which includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the cooking prompt method as described above.

[0019] On the other hand, a terminal is provided, which includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the abnormal image detection method or the cooking prompt method as described above.

[0020] On the other hand, a computer program product is provided, including a computer program, characterized in that when the computer program is executed by a processor, it implements the cooking prompt method as described above.

[0021] The cooking reminder method, device, equipment, storage medium, server, terminal, computer program, and computer program product provided by this application have the following technical effects:

[0022] The technical solution of the present application first obtains countertop temperature field image data and countertop temperature matrix data; then performs feature extraction on the countertop temperature field image data and countertop temperature matrix data to obtain ingredient feature information and temperature field feature information; then fuses the cooking ingredient feature information and temperature field feature information to obtain cooking fusion feature information; then inputs the cooking fusion feature information into a maturity prediction model to obtain target maturity information, which is used to indicate the maturity of the detected ingredient; and then generates first indication information based on the target maturity information, which is used to indicate the cooking stage corresponding to the target maturity information. The present application can improve the automatic recognition of the cooking process and output prompt information, thereby enhancing the intelligent level of cooking. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present application or the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0024] Figure 1 is a schematic diagram of an application environment provided by an embodiment of the present application;

[0025] Figure 2 This is a flowchart of a cooking prompt method provided by an embodiment of the present application;

[0026] Figure 3 This is a flowchart of a cooking prompt method provided by an embodiment of the present application;

[0027] Figure 4 This is a schematic diagram of a cooking reminder device provided by an embodiment of the present application;

[0028] Figure 5 This is a hardware structure block diagram of an electronic device for a cooking reminder method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or server comprising a series of steps or submodules is not necessarily limited to those steps or submodules clearly listed, but may include other steps or submodules that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] Before further explaining the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0032] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also involves studying the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0033] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0034] Computer vision (CV) is the science of making machines "see." Specifically, it refers to machine vision, where cameras and computers replace the human eye in identifying and measuring objects, performing further image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0035] In recent years, with the research and progress of artificial intelligence technology, artificial intelligence technology has been widely used in many fields. The solutions provided in the embodiments of this application involve artificial intelligence machine learning / deep learning and natural language processing technologies, which are specifically illustrated by the following embodiments.

[0036] See also Figure 1 , Figure 1 This is a schematic diagram of an application environment provided by an embodiment of the present application, such as Figure 1 As shown, the application environment may include at least a monitoring terminal, a central control terminal and a user terminal. In actual applications, the monitoring terminal, the central control terminal and the user terminal may be directly or indirectly connected via wired or wireless communication, and this application does not limit this.

[0037] See also Figure 1 , Figure 1 This is a schematic diagram of an application environment provided by an embodiment of the present application. Figure 1 As shown, the application environment may include at least a terminal 01 and a server 02. In actual applications, the terminal 01 and the server 02 may be connected directly or indirectly via wired or wireless communication, which is not limited in this application.

[0038] The server 02 in the embodiment of the present application can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0039] Specifically, cloud technology refers to a hosting technology that unifies hardware, software, and network resources within a wide area network (WAN) or local area network (LAN) to enable data computing, storage, processing, and sharing. Cloud technology can be applied in a variety of fields, such as healthcare cloud, cloud IoT, cloud security, cloud education, cloud conferencing, artificial intelligence cloud services, cloud applications, cloud calling, and cloud social networking. Based on the cloud computing business model, cloud technology distributes computing tasks across a resource pool consisting of a large number of computers, enabling various application systems to access computing power, storage space, and information services as needed. The network that provides resources is called the "cloud." To users, the resources in the cloud appear infinitely scalable and can be accessed at any time, used on demand, and expanded at any time, with a pay-per-use policy. Providers of cloud computing infrastructure establish a cloud computing resource pool (referred to as a cloud platform, commonly referred to as IaaS (Infrastructure as a Service)) and deploy various types of virtual resources within the resource pool for external clients to choose from. The cloud computing resource pool primarily includes computing devices (virtualized machines, including operating systems), storage devices, and network devices.

[0040] Based on logical functional divisions, the PaaS (Platform as a Service) layer can be deployed on top of the IaaS layer, and the SaaS (Software as a Service) layer can be deployed on top of the PaaS layer. SaaS can also be deployed directly on top of IaaS. PaaS is a platform for software execution, such as databases and web containers. SaaS is a variety of business software, such as web portals and text messaging apps. Generally speaking, SaaS and PaaS are upper layers relative to IaaS.

[0041] Specifically, the server 02 mentioned above may include a physical device, which may specifically include a network communication submodule, a processor, a memory, etc., and may also include software running in the physical device, which may specifically include an application program, etc.

[0042] Specifically, terminal 01 may include physical devices such as smart phones, desktop computers, tablet computers, laptops, digital assistants, augmented reality (AR) / virtual reality (VR) devices, intelligent voice interaction devices, smart home appliances, smart wearable devices, and vehicle-mounted terminal devices, and may also include software running in physical devices, such as applications.

[0043] In this embodiment of the present application, Terminal 01 can be used to send countertop temperature field image data and countertop temperature matrix data to Server 02, so that Server 02 can perform corresponding operations such as feature extraction and classification. Server 02 can be used to provide feature extraction and classification services to obtain target maturity information and target cooking action information. Specifically, Server 02 can also be used to provide various model training services to obtain cooking instruction information, and can also be used to store source sample training sets and model training data.

[0044] Furthermore, it is understandable that Figure 1 What is shown is merely an application environment of the cooking prompt method. The application environment may include more or fewer nodes, and this application does not impose any limitation thereto.

[0045] The following describes a cooking prompt method of the present application based on the above application environment. The embodiment of the present application can be applied to various scenarios, especially to smart cooking scenarios. Figure 2 , Figure 2 It is a flowchart of a cooking reminder method provided by an embodiment of the present application. This specification provides method operation steps such as the embodiment or flowchart, but may include more or fewer operation steps based on conventional or non-creative labor. The order of steps listed in the embodiment is only one way of executing the order of many steps and does not represent the only execution order. When the actual system or server product is executed, it can be executed in the order shown in the embodiment or the accompanying drawings or in parallel (for example, in a parallel processor or multi-threaded processing environment). Specifically, as Figure 2 As shown, the method may include the following steps S201-S209.

[0046] S201: Acquire table surface temperature field image data and table surface temperature matrix data.

[0047] Specifically, the terminal of the embodiment of the present application includes a monitoring terminal and a central control terminal, and the table temperature field image data and the table temperature matrix data are obtained through the monitoring terminal, wherein the monitoring terminal includes an infrared thermal imaging module and transmits the table temperature field image data and the table temperature matrix data to the central control terminal.

[0048] Specifically, the central control terminal includes a control device of the cooking prompt method of the present application, and the central control terminal includes a display screen. In one possible embodiment, the display screen is a touch screen. The central control terminal has data storage and transmission capabilities, temperature monitoring and AI computing capabilities. The central control terminal receives data from the monitoring terminal and uses AI computing capabilities to process and analyze it, and outputs cooking prompt information, wherein the display screen is used to display intelligent assisted recipes, which include recipe preparation content, recipe steps and other matters. Users can select intelligent assisted recipes and then use the cooking assistance function.

[0049] In one possible embodiment, the central control terminal is located on a range hood; in another possible embodiment, the central control terminal is located on a refrigerator. The location of the central control terminal can be adjusted based on the user's cooking scenario.

[0050] S203: performing feature extraction on the table temperature field image data and the table temperature matrix data respectively to obtain food feature information and temperature field feature information.

[0051] Specifically, the countertop temperature field image data is input into the food feature extraction network, which is constructed using a self-supervised learning method. The self-supervised method includes a masked autoencoder (MAE) structure, in which the backbone network of the first encoder part is based on a visual self-attention neural network (ViT). In the training phase, the first encoder TableTempEncoder is trained based on a first preset data set. The first preset data set includes but is not limited to temperature field image data of a variety of ingredients involved in different cooking processes. The temperature field image data includes but is not limited to the temperature change distribution of different types of ingredients during the heating process. During the training process, a masking operation is applied to the countertop temperature field image to hide part of the image area. The encoder encodes the visible area, and the decoder restores the temperature field information of the masked area and constructs a loss function based on the reconstruction error between the restored image and the original image. The decoder optimizes the network parameters by minimizing the reconstruction loss, and finally obtains a countertop temperature field feature representation with discriminative ability. After the training is completed, the masked autoencoder can be used to extract structured temperature field features from any input temperature field image to obtain food feature information.

[0052] Specifically, the countertop temperature matrix data is input into the food feature extraction network. The food feature extraction network uses a self-supervised method and utilizes the second encoder in the masked autoencoder, wherein the main network structure of the second encoder is based on a visual self-attention neural network. The encoder CookFoodEncoder is trained using a second preset data set, which includes different cooking ingredient data sets. The second preset data set includes but is not limited to relevant data on the placement of cooking utensils in the stove core area. The data attributes of the cooking utensils include position, size, material, and whether they have a lid, etc. During the training process, a mask is applied to the input temperature matrix data, the encoder extracts the temperature features of the unmasked part, and the decoder is responsible for reconstructing the data of the masked area. By comparing the difference between the restored result and the original temperature matrix, a reconstruction loss function is constructed and minimized to optimize the network parameters. After training, the network can be used to extract structured feature information of cooking ingredients from any input temperature matrix data.

[0053] S205: Fusing the cooking ingredient feature information and the temperature field feature information to obtain cooking fusion feature information.

[0054] Specifically, the extracted countertop temperature field features and cooking ingredient features are fused in a random manner, where the random method includes learnable parameters. A fusion layer is first created, and then a weighted sum of the temperature field feature information and cooking ingredient feature information is performed, where the weights are learnable.

[0055] Specifically, the temperature field feature information and the cooking ingredient feature information are fused to obtain the cooking fusion feature information including:

[0056] S2051: Normalize the cooking ingredient feature information and the countertop temperature field feature information to obtain a cooking ingredient feature vector and a countertop temperature field feature vector.

[0057] Specifically, the countertop temperature field feature information and the cooking ingredient feature information are normalized to obtain a countertop temperature field feature vector and a cooking ingredient feature vector, which are vectors of the same dimension.

[0058] In an optional embodiment, mean-variance normalization, L2 regularization, or Robust scaling is performed on the table temperature field feature information and the cooking ingredient feature information to obtain the table temperature field feature vector and the cooking ingredient feature vector.

[0059] S2052: Input the cooking ingredient feature vector and the table temperature field feature vector into the target fusion model to obtain cooking fusion feature information.

[0060] Specifically, the target fusion model is a fusion neural network used to fuse features. The random weight fusion layer defined above is adopted and integrated into the feature extraction network and the classification network to construct an overall model. The model is compiled and trained using the first preset data set to optimize the weights of the fusion layer and the parameters of the classification network.

[0061] Specifically, by fusing different types of feature information, the model can learn richer feature combinations, improve the model's performance and generalization capabilities, and input the normalized feature vector into the fusion model to integrate information, so that the model can more accurately capture the maturity or other cooking status of the ingredients.

[0062] S207: Input the cooking fusion feature information into the maturity prediction model to obtain target maturity information, which is used to indicate the maturity of the detected food.

[0063] Specifically, the fused feature information is input into the maturity prediction model, forward propagation is performed, and the target maturity information is obtained and converted into a binary classification result, maturity probability, continuous maturity score or maturity level.

[0064] In a specific embodiment, the cooking fusion feature information is input into a maturity prediction model to obtain target maturity information. The target maturity information is used to indicate the maturity of the detected food and includes:

[0065] S2071: Input the cooking fusion feature information into the maturity classification network to obtain target label information; the target label information corresponds to the original label information of the cooking fusion feature information.

[0066] Specifically, the aforementioned feature extraction network is frozen. A multilayer perceptron (MLP) is used as the maturity classification network, and regularization techniques are used to prevent overfitting. The maturity classification network consists of an input layer, multiple hidden layers, and an output layer, where the number of neurons in the output layer is equal to the number of categories. The obtained cooking fusion feature information is input into the maturity classification network for prediction, and the most likely label is obtained from the predicted probability sequence.

[0067] S2072: Compare the loss function of the target label information and the original label information, and update the parameters of the maturity classification network until the loss function converges.

[0068] Specifically, the loss function between the target label information and the original label information is calculated, the network parameters are updated by calculating the gradient of the loss function and performing backpropagation, and the calculated gradient is applied to the network parameters using an optimizer to update the parameters of the maturity classification network.

[0069] Specifically, the network parameters are optimized through the back-propagation algorithm, so that the loss function converges, so that the model achieves a lower prediction error, reduces the risk of overfitting, and improves the model's generalization ability on new data. In practical applications, it improves the ability to judge the maturity of ingredients and avoids overcooking or undercooking.

[0070] In a specific embodiment:

[0071] Compare the food feature extraction loss and classification loss of the original label information and the original label information, and update the parameters of the maturity classification network until the food feature extraction loss and classification loss converge.

[0072] Specifically, the weighted sum of the food feature extraction loss and classification loss is combined into a total loss. The weights of these two losses are updated inversely in a randomized manner using self-supervised learning. This ensures that the model improves both feature extraction and classification. Dynamically adjusting the weights of these two losses helps find the optimal balance during training, enabling the model to effectively extract features and accurately classify.

[0073] S403: Compare the first cross entropy of the target cooking action classification information and the original cooking action classification information, and update the parameters of the cooking action classification network until the cross entropy converges. The original cooking action classification information represents the label corresponding to the sample action dataset.

[0074] Specifically, the original cooking action classification information represents the labels corresponding to the sample action dataset. The original cooking action classification information is used to supervise the real data labels for training. The model is trained using the first cross entropy as the loss function between the target cooking action classification information and the original cooking action classification information until the first cross entropy converges. The trained model is then used to predict and evaluate cooking action types. At this point, the feature extraction network can be frozen or partially frozen to obtain the cooking fusion feature information and the original cooking action classification information. The loss function is then defined and the cross entropy loss is calculated. The current cross entropy loss is calculated in each training iteration using the backpropagation algorithm. The parameters of the classification network are updated based on the gradient of the cross entropy loss. The model is continuously trained until the cross entropy loss converges on the validation set, resulting in a trained model.

[0075] S209: Generate first indication information based on the target maturity information, where the first indication information is used to indicate a cooking stage corresponding to the target maturity information.

[0076] Specifically, a preset mapping rule is defined to convert the maturity information into a specific cooking stage. In a specific embodiment, when the target maturity information is 0-0.2, the cooking stage is defined as "raw"; when the target maturity information is 0.2-0.4, the cooking stage is defined as "half-cooked"; when the target maturity information is 0.4-0.6, the cooking stage is defined as "cooked"; when the target maturity information is 0.6-0.8, the cooking stage is defined as "overcooked".

[0077] Specifically, the first indication information is used to issue a prompt signal to remind the user. The first indication information is displayed in the user's field of view and can be displayed in the form of a graphical display, text prompt, audio prompt, or visual prompt such as color change or a real-time progress bar. By integrating temperature field characteristic information and ingredient characteristic information and fully leveraging their complementary information, the performance and robustness of the prediction model can be improved, thereby helping users better control cooking time and methods, and can also help users adjust cooking strategies to achieve more ideal cooking results.

[0078] In a specific embodiment, the cooking prompt method further includes:

[0079] S301: Acquire table image data.

[0080] Specifically, the table image data is obtained through the visible light camera of the monitoring end. In one embodiment, the infrared thermal imaging module and the visible light camera of the monitoring end are integrated, that is, the monitoring end simultaneously obtains the table temperature field image data, the table temperature matrix data and the table image data.

[0081] S303: Extract features from the table image data to obtain feature information of the current cooking action.

[0082] Specifically, the obtained countertop image data is used in a self-supervised manner, and the encoder in the Masked Auto Encoders (MAE) is utilized, wherein the main network structure of the encoder is based on the Visual Self-Attention Neural Network (Vision Transformer, ViT), and the third preset data set is utilized, wherein the third preset data set includes different cooking action data sets, to train the encoder CookActEncoder, and the encoder CookActEncoder of the trained Masked Auto Encoder is used to extract cooking action features, and output the extracted current cooking action feature information.

[0083] Extracting features from the table image data to obtain the current cooking action feature information includes:

[0084] S401: Inputting current cooking action feature information into a cooking action classification network to determine target cooking action classification information.

[0085] Specifically, the aforementioned feature extraction network is frozen. A multi-layer perceptron (MLP) is used as the cooking action classification network, and regularization (such as dropout) is added. The cooking action classification network includes an input layer, a hidden layer, and an output layer. The input layer is used to receive the current cooking action feature information, the hidden layer contains several fully connected layers and activation functions, and the output layer is used to output the target cooking action classification information.

[0086] S403: Compare the first cross entropy of the target cooking action classification information and the original cooking action classification information, and update the parameters of the cooking action classification network until the cross entropy converges.

[0087] Specifically, the model is trained using the first cross-entropy loss function until the first cross-entropy converges. The trained model is then used to predict and evaluate cooking action types. At this point, the feature extraction network can be frozen, or partially frozen, to obtain the cooking fusion feature information and the original cooking action classification information. The loss function is then defined and the cross-entropy loss is calculated. The current cross-entropy loss is calculated in each training iteration. A backpropagation algorithm is used to update the parameters of the classification network based on the gradient of the cross-entropy loss. The model is continuously trained until the cross-entropy loss converges on the validation set, resulting in a trained model. The backpropagation algorithm is used to update the parameters of the cooking action classification network to minimize the cross-entropy loss and improve the accuracy of the classification network.

[0088] S305: Input the current cooking action feature information into the cooking action model to obtain target cooking action information.

[0089] Specifically, a previously trained model is used for prediction. The current cooking action feature information is input into the model to obtain the target cooking action information. The target cooking action information is then converted into a cooking action category label, such as stir-frying, adding ingredients, and covering the pot. Different action categories correspond to different probability value sequences.

[0090] S307: Generate second instruction information based on the target cooking action information, where the second instruction information is used to instruct to correct the user's cooking action.

[0091] Specifically, the target cooking action information is converted into second instruction information, which can be displayed in the form of text prompts, voice prompts, or graphical prompts on the user interface, guiding the user to perform the suggested actions through the interface or other means. This makes it easier for users to avoid common mistakes during the cooking process, thereby increasing the success rate of dishes. Accurate prompts can help users complete cooking operations more quickly, saving time and improving overall efficiency.

[0092] The embodiment of the present application also provides a cooking prompt device 400, such as Figure 4 As shown, Figure 4 The following is a schematic diagram showing the structure of a cooking reminder device provided by an embodiment of the present application. The device may include the following modules:

[0093] Table data acquisition module 410: used to acquire table temperature field image data and table temperature matrix data;

[0094] Feature extraction module 420: used to extract features from the table temperature field image data and the table temperature matrix data respectively to obtain food feature information and temperature field feature information;

[0095] Feature fusion module 430: used to fuse cooking ingredient feature information and temperature field feature information to obtain cooking fusion feature information;

[0096] Target maturity information prediction module 440: used to input the cooking fusion feature information into the maturity prediction model to obtain target maturity information, which is used to indicate the maturity of the detected food;

[0097] The first indication module 450 is configured to generate first indication information based on the target maturity information, where the first indication information is configured to indicate a cooking stage corresponding to the target maturity information.

[0098] Specifically, the device may further include the following modules:

[0099] Table image data acquisition module: used to acquire table image data;

[0100] Current cooking action feature information determination module: used to extract features from the countertop image data to obtain current cooking action feature information;

[0101] Target cooking action information determination module: used to input the current cooking action feature information into the cooking action model to obtain the target cooking action information;

[0102] The second instruction module is used to generate second instruction information based on the target cooking action information, and the second instruction information is used to instruct to correct the user's cooking action.

[0103] Specifically, the feature fusion module 430 includes:

[0104] Feature vector determination unit: used for normalizing the cooking ingredient feature information and the table temperature field feature information to obtain the cooking ingredient feature vector and the table temperature field feature vector;

[0105] Cooking fusion feature information determination unit: used to input the cooking ingredient feature vector and the table temperature field feature vector into the target fusion model to obtain cooking fusion feature information.

[0106] Specifically, the device may further include the following modules:

[0107] Target cooking action classification information determination module: used to input current cooking action feature information into the cooking action classification network to determine target cooking action classification information;

[0108] The first update module is used to compare the first cross entropy of the target cooking action classification information and the original cooking action classification information, and update the parameters of the cooking action classification network until the cross entropy converges. The original cooking action classification information represents the label corresponding to the sample action dataset.

[0109] Specifically, the target maturity information prediction module 440 may further include:

[0110] a target label information determining unit, configured to input the cooking fusion feature information into the maturity classification network to obtain target label information; the target label information corresponds to the original label information of the cooking fusion feature information;

[0111] The second updating unit is used to compare the original label information with the loss function of the original label information, and update the parameters of the maturity classification network until the loss function converges.

[0112] Specifically, the target tag information determining unit may further include:

[0113] Target label information determination unit: used to compare the target label information and the loss function of the original label information, update the parameters of the maturity classification network until the food feature extraction loss and the classification loss converge, and the loss function represents the weighted sum of the food feature extraction loss and the classification loss.

[0114] It should be noted that the above device embodiments and method embodiments are based on the same implementation method.

[0115] An embodiment of the present application provides a cooking reminder device, which can be a terminal or a server, including a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the cooking reminder method provided in the above-mentioned method embodiment.

[0116] The memory can be used to store software programs and modules. The processor executes various functional applications and image object recognition by running the software programs and modules stored in the memory. The memory can mainly include a program storage area and a data storage area. The program storage area can store the operating system, application programs required for functions, etc.; the data storage area can store data created based on the use of the device, etc. In addition, the memory can include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory can also include a memory controller to provide the processor with access to the memory.

[0117] The method embodiments provided in the embodiments of the present application can be executed in electronic devices such as mobile terminals, computer terminals, servers or similar computing devices. Figure 5 This is a hardware structure block diagram of an electronic device for a cooking prompt method provided by an embodiment of the present application. Figure 5 As shown, the electronic device 500 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPUs) 510 (the processor 510 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 530 for storing data, and one or more storage media 520 (such as one or more mass storage devices) for storing application programs 523 or data 522. Among them, the memory 530 and the storage medium 520 may be temporary storage or permanent storage. The program stored in the storage medium 520 may include one or more modules, each module may include a series of instruction operations on the electronic device. Furthermore, the central processing unit 510 may be configured to communicate with the storage medium 520 and execute a series of instruction operations in the storage medium 520 on the electronic device 500. The electronic device 500 may also include one or more power supplies 560, one or more wired or wireless network interfaces 550, one or more input and output interfaces 540, and / or one or more operating systems 521, such as Windows Server TM , Mac OS X TM , Unix TM , LinuxTM, FreeBSDTM, etc.

[0118] The input / output interface 540 can be used to receive or send data via a network. Specific examples of the aforementioned network may include a wireless network provided by a communications provider of the electronic device 500. In one embodiment, the input / output interface 540 includes a network interface controller (NIC), which can be connected to other network devices via a base station to communicate with the Internet. In one embodiment, the input / output interface 540 can be a radio frequency (RF) module for wirelessly communicating with the Internet.

[0119] It can be understood by those skilled in the art that Figure 5 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 5 More or fewer components than shown, or with Figure 5 Different configurations shown.

[0120] An embodiment of the present application also provides a computer-readable storage medium, which can be set in an electronic device to store at least one instruction or at least one program related to implementing a cooking prompt method in a method embodiment. The at least one instruction or the at least one program is loaded and executed by the processor to implement the cooking prompt method provided by the above method embodiment.

[0121] Optionally, in this embodiment, the storage medium may be located in at least one of a plurality of network servers in a computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0122] According to one aspect of the present application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various optional implementations described above.

[0123] The cooking prompt method, device, equipment, storage medium, server, terminal, computer program, and computer program product provided by the above-mentioned present application first obtains countertop temperature field image data and countertop temperature matrix data; and performs feature extraction on the countertop temperature field image data and countertop temperature matrix data respectively to obtain temperature field feature information and ingredient feature information; then fuses the temperature field feature information with the cooking ingredient feature information to obtain cooking fusion feature information; then inputs the cooking fusion feature information into a maturity prediction model to obtain target maturity information, which is used to indicate the maturity of the detected ingredient; and then generates first indication information based on the target maturity information, which is used to indicate the cooking stage corresponding to the target maturity information. The present application for countertop temperature field image data can improve the automatic recognition of the cooking process and output prompt information, thereby improving the intelligent level of cooking.

[0124] It should be noted that the order of the embodiments of the present application described above is for descriptive purposes only and does not represent the superiority or inferiority of the embodiments. The above description is of specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0125] The various embodiments in this application are described in a progressive manner. Similar portions between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the device, equipment, and storage medium embodiments are generally similar to the method embodiments, so their descriptions are relatively simple. For relevant portions, refer to the descriptions of the method embodiments.

[0126] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or may be accomplished by instructing the relevant hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk, or an optical disk, etc.

[0127] The above are only preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A cooking reminder method, characterized in that: The method comprises: Acquire table surface temperature field image data and table surface temperature matrix data; Performing feature extraction on the table temperature field image data and the table temperature matrix data respectively to obtain food feature information and temperature field feature information; Fusing the food feature information and the temperature field feature information to obtain cooking fusion feature information; Inputting the cooking fusion feature information into a maturity prediction model to obtain target maturity information, wherein the target maturity information is used to indicate the maturity of the detected food; First indication information is generated based on the target maturity information, where the first indication information is used to indicate a cooking stage corresponding to the target maturity information.

2. The method according to claim 1, characterized in that The method further comprises: Acquire table image data; Performing feature extraction on the table image data to obtain current cooking action feature information; Input the current cooking action feature information into the cooking action model to obtain the target cooking action information; Second instruction information is generated based on the target cooking action information, where the second instruction information is used to instruct correction of the user's cooking action.

3. The method according to claim 1, characterized in that The fusing of the food feature information and the temperature field feature information to obtain cooking fusion feature information includes: Normalizing the cooking ingredient feature information and the countertop temperature field feature information to obtain the cooking ingredient feature vector and the countertop temperature field feature vector; The cooking ingredient feature vector and the table temperature field feature vector are input into a target fusion model to obtain cooking fusion feature information.

4. The method according to claim 2, characterized in that The feature extraction of the table image data to obtain the current cooking action feature information includes: inputting the current cooking action feature information into a cooking action classification network to determine target cooking action classification information; Comparing first cross entropies of target cooking action classification information and original cooking action classification information, and updating parameters of the cooking action classification network until the cross entropy converges, wherein the original cooking action classification information represents a label corresponding to the sample action dataset.

5. The method according to claim 1, characterized in that Inputting the cooking fusion feature information into a maturity prediction model to obtain target maturity information, wherein the target maturity information is used to indicate the maturity of the detected food material, includes: Inputting the cooking fusion feature information into a maturity classification network to obtain target label information; the target label information corresponds to the original label information of the cooking fusion feature information; Compare the loss function of the target label information and the original label information, and update the parameters of the maturity classification network until the loss function converges.

6. The method according to claim 5, characterized in that The loss function comparing the target label information and the original label information, and updating the parameters of the maturity classification network until the loss function converges, includes: Compare the loss function of the target label information and the original label information, update the parameters of the maturity classification network until the food feature extraction loss and the classification loss converge, and the loss function represents the weighted sum of the food feature extraction loss and the classification loss.

7. A cooking reminder device, characterized in that: The device comprises: Table data acquisition module, used to obtain table temperature field image data and table temperature matrix data; A feature extraction module is used to extract features from the table temperature field image data and the table temperature matrix data respectively to obtain food feature information and temperature field feature information; A feature fusion module, configured to fuse the food feature information and the temperature field feature information to obtain cooking fusion feature information; a target maturity information prediction module, configured to input the cooking fusion feature information into a maturity prediction model to obtain target maturity information, wherein the target maturity information is used to indicate the maturity of the detected food; The first indication module is configured to generate first indication information based on the target maturity information, where the first indication information is used to indicate a cooking stage corresponding to the target maturity information.

8. A cooking reminder device, characterized in that: The device includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the cooking prompt method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The storage medium stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the cooking prompt method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the cooking prompt method according to any one of claims 1 to 6 is implemented.

Images