Cooking process monitoring method and device, storage medium and electronic device

Abstract

The invention discloses a cooking process monitoring method and device, a storage medium and an electronic device, and relates to the technical field of intelligent equipment control, and the cooking process monitoring method comprises the steps: obtaining a food material image sequence in a target cooking process; obtaining a state recognition result of the target cooking process according to the dynamic fusion features of the food material image sequence; and performing cooking state regulation and control on the target cooking process according to the state recognition result and the cooking regulation and control strategy. According to the technical scheme, the dynamic change trend of the food material state in the cooking process can be accurately recognized, the accuracy and reliability of the recognition result are improved, an accurate control decision is made according to the dynamic change trend of the food materials, and the cooking effect is optimized.

Description

Technical Field

[0001] This application relates to the field of intelligent device control, and more specifically, to a method, apparatus, storage medium, and electronic device for monitoring a cooking process. Background Technology

[0002] Methods for identifying the cooking status of ingredients based on image sensing technology have advantages such as not requiring contact with the ingredients and being able to efficiently obtain the overall status of the ingredients. There are two main implementation methods in the existing technology: the traditional static judgment method based on preset color thresholds, which pre-sets one or more color thresholds for a specific recipe and judges the maturity of the ingredients based on the threshold range of the real-time image; and the judgment method based on machine learning classifiers, which pre-trains a classifier based on manually extracted texture and morphological features of the ingredients, extracts the same features from the real-time image during the cooking process, and the classifier judges the maturity category of the ingredients.

[0003] Both of the above methods suffer from low portability across different scenarios, low reliability in complex cooking environments, and a lack of time-series dynamic analysis capabilities. Summary of the Invention

[0004] This application provides a cooking process monitoring method, device, storage medium, and electronic device, with the aim of accurately identifying the dynamic change trend of the food's state during the cooking process, improving the accuracy and reliability of the identification results, and making precise control decisions based on the dynamic change trend of the food to optimize the automatic cooking effect.

[0005] According to one aspect of the embodiments of this application, a cooking process monitoring method is provided, comprising:

[0006] Acquire food image sequences during the target cooking process;

[0007] Based on the dynamic fusion features of the food image sequence, the state recognition result of the target cooking process is obtained;

[0008] Based on the state recognition results and cooking control strategies, the cooking state of the target cooking process is controlled.

[0009] Optionally, acquiring the food image sequence during the target cooking process includes:

[0010] Detect the current cooking action during the target cooking process;

[0011] If it is determined that the current cooking action does not include image interference actions, an image frame acquisition action is triggered to obtain the food image sequence.

[0012] Optionally, obtaining the state recognition result of the target cooking process based on the dynamic fusion features of the food image sequence includes:

[0013] The current image frame and at least one historical image frame of the food image sequence are input into the state recognition model;

[0014] Obtain the state recognition result output by the state recognition model;

[0015] The state recognition model is used to obtain the dynamic fusion features of the food image sequence based on the input current image frame and the historical image frame, and output the state recognition result; the dynamic fusion features include local detail features, global channel features and temporal dynamic features.

[0016] Optionally, before obtaining the state recognition result of the target cooking process based on the dynamic fusion features of the food image sequence, the method further includes:

[0017] Functional parameters and dynamic weights are configured for the local feature branch, global feature branch, and temporal feature branch of the state recognition model, respectively.

[0018] The state recognition model is trained using preset training samples to update the functional parameter configuration and / or the dynamic weight configuration through the training of the target model.

[0019] Optionally, before training the state recognition model on the target model based on preset training samples, the method further includes:

[0020] Obtain a training image sequence with a preset number of samples;

[0021] The training image sequence is subjected to sample preprocessing; the sample preprocessing includes scene feature enhancement processing.

[0022] Based on the cooking start state frame and cooking completion state frame in the training image sequence, the food ingredient status is labeled on the training image data to obtain the preset training sample.

[0023] Optionally, the state recognition result includes: food ingredient state prediction curve;

[0024] The step of regulating the cooking state of the target cooking process based on the state recognition result and the cooking regulation strategy includes:

[0025] The predicted time distance between the current food state and the target food state is determined based on the food state prediction curve.

[0026] The cooking control strategy acquires a cooking control instruction that matches the predicted time distance; the cooking control instruction is used to control the cooking state of the target cooking process.

[0027] Optionally, after regulating the cooking state of the target cooking process based on the state recognition result and the cooking regulation strategy, the method further includes:

[0028] Obtain user feedback data on the cooking state control;

[0029] The cooking control strategy is updated based on the user feedback data.

[0030] According to another aspect of the embodiments of this application, a cooking process monitoring device is provided, comprising:

[0031] The image acquisition module is used to acquire image sequences of ingredients during the target cooking process;

[0032] The state recognition module is used to obtain the state recognition result of the target cooking process based on the dynamic fusion features of the food image sequence;

[0033] The state control module is used to control the cooking state of the target cooking process based on the state recognition results and the cooking control strategy.

[0034] According to another aspect of the embodiments of this application, a computer-readable storage medium is provided, the computer-readable storage medium including a stored program, wherein the program executes the cooking process monitoring method provided in any embodiment of the present invention when it is run.

[0035] According to another aspect of the present application, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the cooking process monitoring method provided in any embodiment of the present invention through the computer program.

[0036] This application provides a cooking process monitoring method, apparatus, storage medium, and electronic device. By acquiring food image sequences during the target cooking process and obtaining state recognition results of the target cooking process based on the dynamic fusion characteristics of the food image sequences, the cooking state of the target cooking process is controlled according to the state recognition results and cooking control strategies. This solves the problems of limited application scope, low reliability, and lack of time-series dynamic analysis capabilities of existing food cooking state recognition technologies. It achieves accurate recognition of the dynamic change trend of food state during the cooking process, improves the accuracy and reliability of the recognition results, and makes precise control decisions based on the dynamic change trend of food, thereby optimizing the automatic cooking effect. Attached Figure Description

[0037] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0038] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a schematic diagram of the hardware environment for an interaction method of a smart device according to an embodiment of this application;

[0040] Figure 2 This is a flowchart of a cooking process monitoring method provided in an embodiment of the present invention;

[0041] Figure 3 This is a model architecture block diagram of a state recognition model provided in an embodiment of the present invention;

[0042] Figure 4 This is a schematic diagram of the structure of a cooking process monitoring device provided in an embodiment of the present invention;

[0043] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0044] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0045] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0046] According to one aspect of the embodiments of this application, a cooking process monitoring method is provided. This cooking process monitoring method is widely applicable to whole-house intelligent digital control application scenarios such as smart homes, smart home ecosystems, and intelligence house ecosystems. Optionally, in this embodiment, the above-mentioned cooking process monitoring method can be applied to, for example... Figure 1 The hardware environment shown consists of terminal device 102 and server 104. For example... Figure 1 As shown, server 104 is connected to terminal device 102 via a network and can be used to provide services (such as application services) to the terminal or clients installed on the terminal. A database can be set up on the server or independently of the server to provide data storage services for server 104. Cloud computing and / or edge computing services can be configured on the server or independently of the server to provide data processing services for server 104.

[0047] The aforementioned network may include, but is not limited to, at least one of the following: wired network, wireless network. The aforementioned wired network may include, but is not limited to, at least one of the following: wide area network, metropolitan area network, local area network. The aforementioned wireless network may include, but is not limited to, at least one of the following: Wi-Fi (Wireless Fidelity), Bluetooth. The terminal device 102 may not be limited to PC, mobile phone, tablet computer, smart air conditioner, smart range hood, smart refrigerator, smart oven, smart stove, smart washing machine, smart water heater, smart washing equipment, smart dishwasher, smart projector, smart TV, smart clothes rack, smart curtains, smart audio-visual equipment, smart socket, smart speaker, smart speaker box, smart fresh air equipment, smart kitchen and bathroom equipment, smart bathroom equipment, smart robot vacuum cleaner, smart window cleaning robot, smart mopping robot, smart air purifier, smart steam oven, smart microwave oven, smart water heater, smart air purifier, smart water dispenser, smart door lock, etc.

[0048] This embodiment provides a cooking process monitoring method, which is used to automatically identify the cooking status of ingredients and intelligently control the cooking process accordingly. Figure 2 A flowchart of a cooking process monitoring method provided in an embodiment of the present invention includes the following steps:

[0049] Step 210: Obtain the image sequence of ingredients during the target cooking process.

[0050] The target cooking process can be any cooking process that needs to be monitored. The food image sequence can be a sequence of at least two frames of images of the same food during the target cooking process, acquired continuously at certain time intervals, arranged in the order of acquisition time.

[0051] Specifically, during the target cooking process, the surface morphology of the ingredients changes with the cooking state. Therefore, a sequence of ingredient images can be acquired in real time during the cooking process to record these surface morphology changes. This sequence can consist of consecutive image frames from a video segment captured at a certain frame rate, or it can be composed of multiple static image data captured consecutively at certain time intervals; no limitation is made here. Correspondingly, the ingredient image sequence can be acquired using a pre-set image acquisition device, preferably a fixed device, to ensure that each frame in the sequence captures the same ingredient.

[0052] In one optional implementation, acquiring the food image sequence in real time during the target cooking process may include: detecting the current cooking action during the target cooking process; and, if it is determined that the current cooking action does not include image interference actions, triggering an image frame acquisition action to acquire the food image sequence.

[0053] The current cooking action can be the action of cooking the ingredients at the current moment. Image interference actions can include cooking actions that may affect the image acquisition effect. Image frame acquisition actions can be the actions of acquiring image frames from the ingredient image sequence.

[0054] Specifically, during the automated cooking process of ingredients by cooking equipment, the current cooking action can be detected in real time. When it is necessary to acquire images to obtain an image sequence of ingredients, the current cooking action can be detected and it can be determined whether it includes image interference actions. Image interference actions can include pre-set cooking actions. For example, in monitoring the cooking process of an oven, the "heating tube operation" may cause overexposure interference to the image due to high-heat light sources. Therefore, this action can be pre-set as an image interference action. Thus, if it is determined that the current cooking action does not include image interference actions, it can be concluded that there are no interference factors in the image acquisition environment at this time, and an image frame acquisition action can be triggered to ensure that the acquisition environment of each frame in the ingredient image sequence is stable. Optionally, each triggering of the image frame acquisition action can trigger the acquisition of a video segment, one or more static images, etc., which can be set according to the data format of the ingredient image sequence. At the same time, it can be set according to the predicted time interval between two adjacent image interference actions, so that each triggering of the image frame acquisition action is carried out under the interference of no image interference actions.

[0055] Step 220: Based on the dynamic fusion features of the food image sequence, obtain the state recognition result of the target cooking process.

[0056] The dynamic fusion feature describes the visual features in a multi-frame image sequence that reflect the cooking state of the food, as well as the variation of these visual features between images. The state recognition result can be the cooking state of the food during the target cooking process reflected by the dynamic fusion feature, preferably the ripeness of the food.

[0057] Specifically, each frame of the food image sequence contains an image of the food surface during the target cooking process. Specific features can be extracted from these images, such as color, texture, local scorch marks, surface juices, and overall shape. Based on these features, the cooking state of the food at the time of image acquisition can be identified. Furthermore, the same feature can change over time across multiple frames of the food image sequence. Based on these changing features, the changes in the cooking state of the food during image acquisition can also be identified. In this embodiment, the features to be focused on and their temporal changes can be pre-defined, allowing for targeted feature extraction from the food image sequence to obtain dynamically fused features. Preferably, the food image sequence includes at least the image frame acquired at the current moment to reflect the real-time state of the food. Therefore, based on the dynamic fusion features constituted by the above features and their changing characteristics, the state recognition result of the target cooking process can be obtained.

[0058] In an optional implementation, obtaining the state recognition result of the target cooking process based on the dynamic fusion features of the food image sequence may include: inputting the current image frame and at least one historical image frame of the food image sequence into the state recognition model; and obtaining the state recognition result output by the state recognition model.

[0059] The state recognition model is used to obtain dynamic fusion features of the food image sequence based on the input current image frame and historical image frames, and outputs the state recognition result. The current image frame can be an image frame in the food image sequence whose acquisition time is the current moment, and the historical image frames can be image frames in the food image sequence whose acquisition time is a previous moment. The state recognition model can be a pre-built and trained neural network model.

[0060] Specifically, the dynamic fusion features that the state recognition model can identify can include local detail features, global channel features, and temporal dynamic features. Local detail features can include local details on the surface of the food within an image frame that reflect the cooking state, such as the "caramel spots" feature of an egg tart or the "muscle fiber separation" feature of a steak. Global channel features can include the channel features most relevant to the cooking state of the food among the global multi-channel features within an image frame. For example, during the baking process of a cake, global channel features can include the color channel features of the food, but not the shape channel features. Temporal dynamic features can include dynamic changes in features between multiple image frames in a sequence of food images, such as the vector of boiling juice or the speed of the cake surface rising. The state recognition model can be pre-built and trained to extract the above dynamic fusion features and output the state recognition result by combining these features.

[0061] In an optional implementation, before obtaining the state recognition result of the target cooking process based on the dynamic fusion features of the food image sequence, the method may further include: configuring functional parameters and dynamic weights for the local feature branches, global feature branches, and temporal feature branches of the state recognition model, respectively; training the state recognition model with a target model based on preset training samples, so as to update the functional parameter configuration and / or dynamic weight configuration through target model training.

[0062] The local feature branch can be a branch with local detail feature recognition function built in the state recognition model; the global feature branch can be a branch with global channel feature recognition function built in the state recognition model; and the temporal feature branch can be a branch with temporal dynamic feature recognition function built in the state recognition model. Functional parameter configuration involves configuring relevant parameters of the state recognition model to enable the operation of the above branches and their functions. Dynamic weight configuration involves configuring the weights of the influence of local detail features, global channel features, and temporal dynamic features on the state recognition result. Preferably, these weights can dynamically change with the cooking state; for example, at the end of cooking, the weights of local charred features and dynamic features of reduced juice will significantly increase, while the weight of global color features will decrease accordingly.

[0063] Specifically, there are options, Figure 3 This is a model architecture block diagram of a state recognition model provided in an embodiment of the present invention. Figure 3As shown, the local feature branch can employ a high-resolution convolutional network to avoid using a large downsampling rate, thus preserving the subtle but crucial local features of the food surface and compensating for the neglect of these features by general models. The global feature branch can use the SENet (Squeeze-and-Excitation Network) module, allowing the state recognition model to focus on the channel features most relevant to the state recognition result. The temporal feature branch can be used to first calculate the dense optical flow field between adjacent frames, and then input the optical flow sequence into a Bi-LSTM (Bidirectional Long Short-Term Memory) network. Here, the optical flow field explicitly describes pixel-level motion information, enabling Bi-LSTM to focus more on learning the relationship between physical motion and maturity, rather than static appearance. Further, optionally, a feature fusion module can be pre-built in the state recognition model, allowing for dynamic weight configuration of the features extracted from the above branches through the construction and configuration of the feature fusion module. Specifically, the feature fusion module can employ a multi-head attention mechanism to dynamically assign appropriate weights to the features of the three branches.

[0064] Furthermore, after the state recognition model is constructed, preset training samples can be input into the model to perform feature recognition learning and obtain the model's output. Based on the error between the output and the corresponding food cooking state of the preset training samples, the functional parameters and / or dynamic weight configuration of the state recognition model are updated and iterated to complete the target model training, making the model's output more accurate. Optionally, the target model can be trained by setting a training loss function. The preset training samples can be image sequences pre-collected in the cooking environment, and their food cooking state features can be pre-labeled.

[0065] In one optional implementation, before training the state recognition model on the target model based on the preset training samples, the method may further include: obtaining a training image sequence with a preset number of samples; performing sample preprocessing on the training image sequence; and labeling the food ingredients in the training image data according to the cooking start state frame and cooking completion state frame in the training image sequence to obtain the preset training samples.

[0066] The preset sample quantity can be the number of training sample data required to train the state recognition model. The training image sequence can be an image sequence pre-collected in a cooking environment. Sample preprocessing can be operations that process the image frames of the training image sequence in a preset manner to make their format and quality usable for training the target model; optionally, it can include color correction, ROI (Region of Interest) cropping, image enhancement, etc.

[0067] Specifically, sample preprocessing includes scene feature enhancement. Scene feature enhancement can be an operation that adds corresponding image features to the image frames of the training image sequence to address potential interference factors in the actual cooking environment. Optionally, scene feature enhancement may include adding highlights at random positions in the image frames of the training image sequence to simulate glare interference from high-temperature light-emitting devices such as heating elements; it may also include adding a hazy blur of random size and transparency to simulate water vapor obstruction; and it may include randomly adding positional offsets to simulate the slight movements of food during cooking due to heat convection or boiling of juices.

[0068] Furthermore, the food ingredient states can be labeled on the training image sequence, indicating the cooking state of the ingredients for each image frame to obtain preset training samples. Specifically, it is not necessary to label the precise cooking state of the ingredients for every frame. Instead, the cooking start state frame and the cooking completion state frame can be labeled with the cooking start state feature and cooking completion state feature, respectively, for example, the ingredient maturity of 0% and 100%. The cooking start state frame can be the image frame captured when cooking begins, and the cooking completion state frame can be the image frame captured when the ingredients are determined to be in the optimal state required for this cooking. Based on the labeling of these two frames, a temporal smoothing label can be used. In the T image frames captured from the start to the end of cooking, the t-th frame can be labeled as min(100%,(t / T)*100%*α); where α is a correction factor slightly greater than 1, used to simulate accelerated ripening in the later stages. Correspondingly, a temporal consistency constraint can be introduced into the training loss function to penalize cases where the predicted maturity value does not monotonically increase.

[0069] Step 230: Based on the state recognition results and cooking control strategies, control the cooking state of the target cooking process.

[0070] The cooking control strategy can be a pre-set rule that controls cooking actions based on the cooking state of the ingredients, so that the ingredients reach the preset optimal cooking state through the complete cooking process after control. Cooking state control can be the operation that controls the current and subsequent cooking actions of the target cooking process.

[0071] Specifically, based on the identified state recognition results and a preset cooking control strategy, the cooking state can be adjusted accordingly to ensure that the ingredients reach their optimal cooking state after the cooking process. For example, a maturity threshold, such as 98%, can be preset in the cooking control strategy. When the maturity of the ingredients in the state recognition results reaches or exceeds this threshold, an emergency stop command or power adjustment command can be sent to the underlying controller of the cooking equipment to prevent overcooking.

[0072] In one optional implementation, the cooking state of the target cooking process is controlled according to the state recognition result and the cooking control strategy. This may include: acquiring the current food temperature, the remaining cooking time, and user preference data; and generating cooking control instructions based on the cooking control strategy, according to the state recognition result, the current food temperature, the remaining cooking time, and user preference data.

[0073] The current food temperature can be the real-time temperature obtained through a preset sensor, such as the core temperature of the food measured by a temperature probe. The remaining cooking time can be the length of time between the current moment and the end time in a preset cooking program, such as a preset recipe on the cooking device. User preference data can be the user's preference for the optimal cooking state of the food.

[0074] Specifically, the above implementation can achieve multimodal decision-making based on cooking control strategies. The cooking control commands used to control the cooking process are not only determined based on the state recognition results, but also integrate the current food temperature, remaining cooking time, and user preference data.

[0075] Further optionally, based on the above implementation, before generating cooking control instructions based on the cooking control strategy, the status recognition result, the current ingredient temperature, the remaining cooking time, and user preference data, the method may include: issuing a check prompt to the user if the status recognition result does not match the current ingredient temperature. For example, if the status recognition result indicates that the ingredient's maturity is 95%, but the probe temperature is far from reaching the safe temperature to ensure the ingredient is fully cooked, the user may be prompted to check.

[0076] Further optionally, based on the above implementation, before generating cooking control instructions based on the cooking control strategy, the status recognition result, the current ingredient temperature, the remaining cooking time, and user preference data, the method may include: issuing a check prompt to the user if the status recognition result does not match the remaining cooking time. For example, if the status recognition result shows that the ingredient's maturity is 95%, it can be predicted that it will take 1 minute to reach 100% maturity, while the remaining cooking time determined according to the preset recipe is 10 minutes, then the user can be prompted to check. Further optionally, a preset recipe update instruction can be generated based on the user's check result to update the preset recipe to a setting that matches the user's preferences.

[0077] In one optional implementation, the state recognition result may include: an ingredient state prediction curve. The ingredient state prediction curve can be obtained by predicting the changes in the cooking state of the ingredient over a future period of time based on the dynamic fusion features of the ingredient image sequence. It is used to describe the changes of preset parameters representing the cooking state of the ingredient over time. The parameters may be, for example, the maturity of the ingredient, the degree of completion of cooking, etc.

[0078] In an optional implementation, the cooking state regulation of the target cooking process based on the state recognition result and the cooking regulation strategy may include: determining the predicted time distance between the current food state and the target food state based on the food state prediction curve; and obtaining a cooking regulation instruction that matches the predicted time distance from the cooking regulation strategy.

[0079] Here, the current ingredient state can be the cooking state of the ingredient identified at the current moment, and the target ingredient state can be the optimal cooking state that the ingredient needs to achieve through the cooking process. The predicted time distance can be the time length between the future moment corresponding to the target ingredient state in the ingredient state prediction curve and the current moment. Cooking control commands can be used to regulate the cooking state of the target cooking process; for example, they can be commands to control the actions of cooking equipment.

[0080] Specifically, the state recognition result can include not only the current cooking state of the ingredients, but also the predicted changes in the cooking state of the ingredients at future times based on the dynamic changes of the features, thus obtaining an ingredient state prediction curve. Furthermore, based on the ingredient state prediction curve, the predicted time when the target ingredient state is reached can be determined, thus obtaining the predicted time distance between that time and the current time. Based on the predicted time distance, matching cooking control instructions can be obtained accordingly. These instructions allow for pre-adjustment of cooking actions before the ingredients fully reach the target state, avoiding overcooking caused by responding only after the ingredients have reached the target state.

[0081] In an optional implementation, obtaining a cooking control command that matches the predicted time distance in the cooking control strategy may include: obtaining a heat preservation and penetration command when the predicted time distance is within a first duration range; and obtaining a stop heating command when the predicted time distance is 0.

[0082] The first time range can be a relatively short period of time. The heat preservation and penetration command can be a command to reduce the heating power of the cooking equipment to a certain extent. For example, if it is predicted that the food's doneness will reach 100% within the next 10 seconds, the heating power can be reduced to 30% to allow heat to slowly transfer to the center of the food, preventing the surface of the food from overheating and allowing the cooking to enter the pre-termination stage. If the predicted time distance has reached 0, the cooking equipment can be controlled to immediately stop heating via a stop heating command. Optionally, it can also include controlling the cooking equipment to immediately start the cavity cooling fan to quickly reduce the ambient temperature of the food and prevent residual heat from continuing to heat the food.

[0083] In one optional implementation, the cooking state regulation of the target cooking process based on the state recognition result and the cooking regulation strategy may include: obtaining the ingredient cooking rate based on the ingredient state prediction curve; and obtaining a cooking regulation command that matches the ingredient cooking rate.

[0084] The food cooking rate can be the rate at which the food's cooking state changes at the current moment. Based on the food cooking rate, corresponding cooking control commands can be obtained to adjust the cooking actions of the cooking equipment, ensuring that the rate of change in the food's cooking state is within an ideal range. Specifically, if the food cooking rate exceeds a preset cooking rate threshold, commands to lower the temperature and / or adjust the shelf position can be obtained. The preset cooking rate threshold can be a pre-defined maximum allowable rate of change in the cooking state; lowering the temperature can slow down the change in cooking state; adjusting the shelf position can be a command to adjust the position of the food, moving it away from the heat source to lower the temperature.

[0085] In an optional implementation, after regulating the cooking state of the target cooking process based on the state recognition result and the cooking regulation strategy, the method may further include: obtaining user feedback data on the cooking state regulation; and updating the cooking regulation strategy based on the user feedback data.

[0086] Among them, user feedback data can be data describing the user's satisfaction with the results of cooking state control after the control is implemented.

[0087] Specifically, after each cooking status adjustment, a feedback channel can be provided to the user to obtain user feedback data. For example, a feedback interface can be pushed to the user's terminal, providing feedback options such as "too rare," "perfect," and "well-done." User feedback data can be used to update the cooking control strategy. Optionally, user feedback data can be correlated with dynamic fusion features for subsequent status recognition results obtained based on these features. Further, a user preference model can be constructed based on the correlation between user feedback data and dynamic fusion features to dynamically adjust the cooking control strategy according to different users' preferences. For example, if a user repeatedly selects "perfect" even when there are many charred spots on the steak surface, the system will learn that the user prefers a crispier texture and appropriately increase the doneness threshold in the future.

[0088] The above implementation method acquires food image sequences in real time during the target cooking process, obtains the state recognition result of the target cooking process based on the dynamic fusion features of at least two food image sequences, and then regulates the cooking state of the target cooking process according to the state recognition result and cooking control strategy. This solves the problems of limited application scope, low reliability and lack of time-series dynamic analysis capability of existing food cooking state recognition technology, realizes accurate recognition of the dynamic change trend of food state during the cooking process, improves the accuracy and reliability of the recognition result, and makes precise control decisions based on the dynamic change trend of food, thereby optimizing the automatic cooking effect.

[0089] According to another aspect of the embodiments of the present invention, Figure 4 This is a schematic diagram of a cooking process monitoring device provided in an embodiment of the present invention, as shown below. Figure 4 As shown, the device includes an image acquisition module 410, a state recognition module 420, and a state control module 430, wherein:

[0090] Image acquisition module 410 is used to acquire food image sequences during the target cooking process;

[0091] The state recognition module 420 is used to obtain the state recognition result of the target cooking process based on the dynamic fusion features of the food image sequence;

[0092] The state control module 430 is used to control the cooking state of the target cooking process according to the state recognition result and the cooking control strategy.

[0093] In an optional implementation, the image acquisition module 410 may include: an action detection unit for detecting the current cooking action during the target cooking process; and an acquisition triggering unit for triggering an image frame acquisition action to acquire the food image sequence when it is determined that the current cooking action does not include image interference actions.

[0094] In an optional implementation, the state recognition module 420 may include: a model input unit, used to input the current image frame and at least one historical image frame of the food image sequence into a state recognition model; and a model output unit, used to obtain the state recognition result output by the state recognition model; wherein the state recognition model is used to obtain the dynamic fusion features of the food image sequence based on the input current image frame and the historical image frame, and output the state recognition result; the dynamic fusion features include local detail features, global channel features, and temporal dynamic features.

[0095] In an optional implementation, the state recognition module 420 may further include: a model configuration unit, configured to configure functional parameters and dynamic weights for the local feature branches, global feature branches, and temporal feature branches of the state recognition model, respectively; and a model training unit, configured to train a target model on the state recognition model based on preset training samples, so as to update the functional parameter configuration and / or the dynamic weight configuration through the target model training.

[0096] In an optional implementation, the state recognition module 420 may further include: a sample acquisition unit for acquiring a training image sequence with a preset number of samples; a sample processing unit for performing sample preprocessing on the training image sequence; the sample preprocessing includes scene feature enhancement processing; and a sample annotation unit for annotating the food ingredient states on the training image data according to the cooking start state frame and cooking completion state frame in the training image sequence, thereby obtaining the preset training samples.

[0097] In an optional implementation, the state recognition result may include: a food ingredient state prediction curve; the state control module 430 may include: a prediction time unit, used to determine the predicted time distance between the current food ingredient state and the target food ingredient state based on the food ingredient state prediction curve; and an instruction matching unit, used to obtain a cooking control instruction that matches the predicted time distance in the cooking control strategy; the cooking control instruction is used to control the cooking state of the target cooking process.

[0098] In an optional implementation, the device may further include: a feedback update module, configured to acquire user feedback data on the cooking state control; and update the cooking control strategy based on the user feedback data.

[0099] According to another aspect of the embodiments of the present invention, Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present invention, such as... Figure 5As shown, the electronic device includes a processor 510, a memory 520, an input device 530, and an output device 540; the number of processors 510 in the electronic device can be one or more. Figure 5 Taking a processor 510 as an example; the processor 510, memory 520, input device 530, and output device 540 in the electronic device can be connected via a bus or other means. Figure 5 Taking the example of a connection between China and Israel via a bus.

[0100] The memory 520, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the cooking process monitoring method in this embodiment of the invention (e.g., the image acquisition module 410, the status recognition module 420, and the status control module 430 in the cooking process monitoring device). The processor 510 executes various functional applications and data processing of the electronic device by running the software programs, instructions, and modules stored in the memory 520, thereby realizing the aforementioned cooking process monitoring method.

[0101] The memory 520 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function; the data storage area may store data created based on terminal usage. Furthermore, the memory 520 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 520 may further include memory remotely located relative to the processor 510, which can be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0102] Input device 530 can be used to receive input digital or character information, and generate key signal inputs related to user settings and function control of the electronic device. It can also be a camera for acquiring images and a sound pickup device for acquiring audio data. Output device 540 may include display devices such as a screen, and audio devices such as a speaker. It should be noted that the specific composition of input device 530 and output device 540 can be set according to actual conditions. Processor 510 executes various functional applications and data processing of the electronic device by running software programs, instructions, and modules stored in memory 520.

[0103] According to another aspect of the present invention, the present invention also provides a computer-readable storage medium comprising a stored program, wherein the program, when executed, performs a cooking process monitoring method as follows:

[0104] Acquire food image sequences during the target cooking process;

[0105] Based on the dynamic fusion features of the food image sequence, the state recognition result of the target cooking process is obtained;

[0106] Based on the state recognition results and cooking control strategies, the cooking state of the target cooking process is controlled.

[0107] Of course, the computer-executable instructions provided in the embodiments of the present invention are not limited to the method operations described above, but can also perform related operations in the cooking process monitoring method provided in any embodiment of the present invention.

[0108] Based on the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0109] It is worth noting that in the embodiments of the cooking process monitoring device described above, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of the present invention.

[0110] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for monitoring the cooking process, characterized in that, include: Acquire food image sequences during the target cooking process; Based on the dynamic fusion features of the food image sequence, the state recognition result of the target cooking process is obtained; Based on the state recognition results and cooking control strategies, the cooking state of the target cooking process is controlled.

2. The method according to claim 1, characterized in that, The process of acquiring food image sequences during the target cooking process includes: Detect the current cooking action during the target cooking process; If it is determined that the current cooking action does not include image interference actions, an image frame acquisition action is triggered to obtain the food image sequence.

3. The method according to claim 1, characterized in that, The step of obtaining the state recognition result of the target cooking process based on the dynamic fusion features of the food image sequence includes: The current image frame and at least one historical image frame of the food image sequence are input into the state recognition model; Obtain the state recognition result output by the state recognition model; The state recognition model is used to obtain the dynamic fusion features of the food image sequence based on the input current image frame and the historical image frame, and output the state recognition result; the dynamic fusion features include local detail features, global channel features and temporal dynamic features.

4. The method according to claim 3, characterized in that, Before obtaining the state recognition result of the target cooking process based on the dynamic fusion features of the food image sequence, the method further includes: Functional parameters and dynamic weights are configured for the local feature branch, global feature branch, and temporal feature branch of the state recognition model, respectively. The state recognition model is trained using preset training samples to update the functional parameter configuration and / or the dynamic weight configuration through the target model training.

5. The method according to claim 4, characterized in that, Before training the state recognition model on the target model based on the preset training samples, the method further includes: Obtain a training image sequence with a preset number of samples; The training image sequence is subjected to sample preprocessing; the sample preprocessing includes scene feature enhancement processing. Based on the cooking start state frame and cooking completion state frame in the training image sequence, the food ingredient status is labeled on the training image data to obtain the preset training sample.

6. The method according to claim 1, characterized in that, The state recognition results include: food ingredient state prediction curves; The step of regulating the cooking state of the target cooking process based on the state recognition result and the cooking regulation strategy includes: The predicted time distance between the current food state and the target food state is determined based on the food state prediction curve. The cooking control strategy acquires a cooking control instruction that matches the predicted time distance; the cooking control instruction is used to control the cooking state of the target cooking process.

7. The method according to claim 6, characterized in that, After regulating the cooking state of the target cooking process based on the state recognition result and the cooking regulation strategy, the method further includes: Obtain user feedback data on the cooking state control; The cooking control strategy is updated based on the user feedback data.

8. A cooking process monitoring device, characterized in that, include: The image acquisition module is used to acquire image sequences of ingredients during the target cooking process; The state recognition module is used to obtain the state recognition result of the target cooking process based on the dynamic fusion features of the food image sequence; The state control module is used to control the cooking state of the target cooking process based on the state recognition results and the cooking control strategy.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program, when executed, performs the cooking process monitoring method according to any one of claims 1 to 7.

10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the cooking process monitoring method according to any one of claims 1 to 7 through the computer program.

Images