A control method and device of a multi-mode cooking machine, the cooking machine and a storage medium

Abstract

The application provides a control method of a multi-modal cooking machine: the method comprises the following steps: after obtaining a cooking task, determining attribute data of a to-be-processed food material corresponding to the cooking task; wherein the cooking task comprises an identification of a dish and cooking requirement information; obtaining cooking data related to the cooking task from a server based on the identification of the dish; wherein the cooking data is used for realizing cooking of the food material; determining a target cooking strategy corresponding to the cooking task based on the attribute data of the to-be-processed food material, the cooking requirement information and the cooking data; and performing a cooking operation based on the target cooking strategy and multi-modal first data related to the to-be-processed food material, so as to complete the cooking of the to-be-processed food material. The application also provides a control device, a cooking machine and a storage medium.

Description

Technical Field

[0001] This application relates to the control technology of multimodal cooking machines in the field of intelligent technology, and particularly to a control method, device, cooking machine and storage medium for a multimodal cooking machine. Background Technology

[0002] With the development of artificial intelligence technology, intelligent cooking robots have become a core product in kitchen automation. Currently, cooking robots on the market can achieve basic functions such as automatic stir-frying and timed heating; some have even incorporated temperature and weighing sensors to control basic cooking parameters. With advancements in computer vision and multi-sensor fusion technology, cooking has begun to be achieved through image recognition and gas detection. However, the cooking solutions in these technologies have limitations; a given recipe can only be deployed locally or in the cloud, resulting in insufficient precision in controlling the recipe and thus affecting cooking efficiency. Summary of the Invention

[0003] This application provides a control method, device, cooking machine, and storage medium for a multimodal cooking machine, which solves the problem of insufficient control accuracy of the determined cooking recipe in related technologies, thereby improving cooking efficiency.

[0004] The technical solution of this application embodiment is implemented as follows: A control method for a multimodal cooking machine, the method comprising: After obtaining the cooking task, determine the attribute data of the ingredients to be processed corresponding to the cooking task; wherein, the cooking task includes the dish's identifier and cooking requirement information; Based on the identifier of the dish, cooking data related to the cooking task is obtained from the server; wherein, the cooking data is used to cook the ingredients; Based on the attribute data of the ingredients to be processed, the cooking requirement information, and the cooking data, the target cooking strategy corresponding to the cooking task is determined; The cooking operation is performed based on the target cooking strategy and the first multimodal data related to the ingredients to be processed, so as to complete the cooking of the ingredients to be processed.

[0005] In the above scheme, determining the target cooking strategy corresponding to the cooking task based on the attribute data of the ingredients to be processed, the cooking requirement information, and the cooking data includes: If it is determined that the attribute data of the ingredient to be processed matches the basic attribute data corresponding to the cooking task, the target cooking strategy is determined based on the attribute data of the ingredient to be processed, the cooking requirement information, and the historical cooking data corresponding to the cooking task; wherein, the cooking data includes the historical cooking data.

[0006] The method in the above scheme further includes: If it is determined that the attribute data of the ingredient to be processed does not match the basic attribute data corresponding to the cooking task, the attribute data of the ingredient to be processed is updated based on the standard ingredient attribute data corresponding to the cooking task. The target cooking strategy is determined based on the updated attribute data of the ingredients to be processed, the cooking requirement information, and the historical cooking data; wherein the cooking data includes standard ingredient attribute data and the historical cooking data.

[0007] In the above scheme, the step of performing the cooking operation based on the target cooking strategy and the first multimodal data related to the ingredients to be processed includes: During the cooking operation based on the target cooking strategy, multiple sensors collect multimodal first data related to the food to be processed within the pot; wherein, the multiple first data represent the current state of the food to be processed; each sensor is of a different type; The first degree of cooking of the food to be processed in the pot is determined based on the multiple first data. If the first maturity level meets the first target maturity threshold, a feeding operation is performed on the ingredients to be processed in the pot. If the cooking parameters of the ingredients to be processed are determined to meet the target requirements, a prompt message will be output to indicate that the cooking is complete.

[0008] In the above scheme, determining the first degree of doneness of the food to be processed in the pot based on the plurality of first data includes: Determine the first feature information for each first data point; The weight coefficient corresponding to each first data point is determined based on the cooking scenario corresponding to the cooking task. The first fusion feature is determined based on multiple first feature information and the weight coefficient corresponding to each first data; The first degree of maturity of the food ingredient to be processed is determined based on the first fusion feature.

[0009] In the above scheme, determining the first feature information of each first data point includes: Process multiple sets of the first data; For each processed first data, feature extraction is performed to obtain multiple first feature information.

[0010] In the above scheme, determining the first degree of maturity of the food to be processed in the pot based on the first fusion feature includes: Obtain multiple initial feature information of the food to be processed corresponding to the multiple sensors; Initial fusion features are determined based on the aforementioned multiple initial feature information; A second fusion feature is determined based on the first fusion feature and the initial fusion feature; The first maturity level is determined based on the second fusion feature.

[0011] In the above scheme, the step of performing a feeding operation on the ingredients to be processed in the pot body if the first maturity level meets the first target maturity level threshold includes: If all N first maturity values ​​determined based on the N second fusion features within the target time period meet the first target maturity threshold, a feeding operation is performed on the ingredients to be processed in the pot.

[0012] In the above scheme, the feeding operation for the ingredients to be processed in the pot includes: If the first maturity meets the first target maturity threshold, multiple second data related to the food to be processed are collected inside the pot through the multiple sensors; Determine the second feature information for each second data point; The third fusion feature is determined based on multiple second feature information and the weight coefficient corresponding to each second data; The second degree of maturity of the food to be processed in the pot is determined based on the third fusion feature. If the second maturity level meets the second target maturity threshold, a feeding operation is performed on the ingredients to be processed inside the pot.

[0013] In the above scheme, determining the second degree of maturity of the food to be processed in the pot based on the third fusion feature includes: The fourth fusion feature is determined based on the third fusion feature and the initial fusion feature; The second degree of maturity of the food to be processed in the pot is determined based on the fourth fusion feature.

[0014] In the above scheme, if the second maturity meets the second target maturity threshold, the step of performing a feeding operation on the ingredients to be processed in the pot includes: If all N second maturity values ​​determined based on the N fourth fusion features within the target time period meet the second target maturity threshold, a feeding operation is performed on the ingredients to be processed in the pot.

[0015] The method in the above scheme further includes: During the process of adding the ingredients to be processed into the pot based on the target cooking strategy, sensor data collected by multiple sensors corresponding to each cooking stage and / or the currently added ingredients are acquired at different cooking stages. Cooking operations corresponding to the different cooking stages are performed based on multiple sensor data.

[0016] A control device, the device comprising: The first determining unit is used to determine the attribute data of the ingredients to be processed corresponding to the cooking task after obtaining the cooking task; wherein, the cooking task includes the identification of the dish and cooking requirement information; An acquisition unit is configured to acquire cooking data related to the cooking task from a server based on the identifier of the dish; wherein the cooking data is used to cook the ingredients; The second determining unit is used to determine the target cooking strategy corresponding to the cooking task based on the attribute data of the ingredients to be processed, the cooking requirement information, and the cooking data. The processing unit is configured to perform cooking operations based on the target cooking strategy and multimodal first data related to the ingredients to be processed, so as to complete the cooking of the ingredients to be processed.

[0017] A cooking machine, the cooking machine comprising: a processor, a memory, and a communication bus; The communication bus is used to realize the communication connection between the processor and the memory; The processor is used to execute the control program of the multimodal cooking machine in the memory to implement the steps of the above-described control method for the multimodal cooking machine.

[0018] A computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the steps of the above-described control method for a multimodal cooking machine.

[0019] A computer program product comprising a computer program that, when executed by a processor, implements the aforementioned control method for a multimodal cooking machine.

[0020] The multimodal cooking machine control method, device, cooking machine, and storage medium provided in this application embodiment include: after acquiring a cooking task including the identifier of the dish and cooking requirement information, determining the attribute data of the ingredients to be processed corresponding to the cooking task; acquiring cooking data related to the cooking task for cooking the ingredients from the server based on the identifier of the dish; determining the target cooking strategy corresponding to the cooking task based on the attribute data of the ingredients to be processed, the cooking requirement information, and the cooking data; and performing a cooking operation based on the target cooking strategy and the first multimodal data related to the ingredients to be processed to complete the cooking of the ingredients to be processed. In this way, when cooking the ingredients, the cooking strategy can be determined based on the cooking data related to the current cooking task from the server, the attribute data of the ingredients corresponding to the current cooking task, and the cooking requirements. Then, the cooking of the ingredients is achieved by combining the first multimodal data related to the ingredients to be processed. Obviously, the cooking strategy for cooking the ingredients is determined based on the cooking data obtained from the server, the attribute data of the ingredients, and the cooking requirements. Both local data and cloud data are taken into account, thereby solving the problem of insufficient control accuracy of the cooking recipe determined in related technologies, and thus improving cooking efficiency. Attached Figure Description

[0021] Figure 1 A flowchart illustrating a control method for a multimodal cooking machine provided in an embodiment of this application; Figure 2 A flowchart illustrating another control method for a multimodal cooking machine provided in an embodiment of this application; Figure 3 A schematic flowchart illustrating another control method for a multimodal cooking machine provided in an embodiment of this application; Figure 4 A schematic diagram of a cooking robot system in a control method for a multimodal cooking machine provided in an embodiment of this application; Figure 5 A schematic diagram of the structure of a control device provided for an embodiment of this application; Figure 6 This is a schematic diagram of the structure of a cooking machine provided for an embodiment of this application. Detailed Implementation

[0022] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0023] It should be understood that the phrases "embodiments of this application" or "foreign embodiments" throughout the specification mean that a specific feature, structure, or characteristic related to an embodiment is included in at least one embodiment of this application. Therefore, "embodiments of this application" or "in the foreign embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0024] Unless otherwise specified, any step in the embodiments of this application performed by the electronic device may be executed by the processor of the electronic device. It is also worth noting that the embodiments of this application do not limit the order in which the electronic device performs the following steps. Furthermore, the methods used to process data in different embodiments may be the same or different methods. It should also be noted that any step in the embodiments of this application can be executed independently by the electronic device; that is, when the electronic device performs any step in the following embodiments, it may not depend on the execution of other steps.

[0025] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of this application.

[0026] This application provides a control method for a multimodal cooking machine, which can be applied to cooking machines. (Refer to...) Figure 1 As shown, the method may include the following steps: Step 101: After obtaining the cooking task, determine the attribute data of the ingredients to be processed corresponding to the cooking task.

[0027] The cooking task includes the identification of the dishes and information on cooking requirements.

[0028] In this embodiment, the cooking task can be triggered by other local devices or a server, or it can be triggered directly by the user on the cooking machine; the server may include a cloud device. The dish identifier can refer to identification information that can characterize the dish to be cooked in the cooking task. In one feasible embodiment, the dish identifier can refer to the name of the dish.

[0029] In addition, cooking requirement information can refer to a cooking requirement dataset; in one feasible embodiment, cooking requirement information may include user preferences (taste requirements), ingredient names, weights, cutting specifications, marinating requirements, pre-treatment blanching and frying requirements, oil usage, timing of ingredient addition, hot oil temperature, seasoning usage, timing of seasoning addition, stir-frying actions and speed during the cooking process, temperature control and heating power during the cooking process, etc.

[0030] It should be noted that the ingredients to be processed can refer to pre-cut ingredients; attribute data can refer to the variety of ingredients, the rules for cutting ingredients, the weight of ingredients, the state of ingredients, etc. In one feasible embodiment, the state of ingredients can include dry, high moisture content, fluffy, firm, etc.

[0031] In other embodiments of this application, the weight of the food to be processed can be determined by a weight sensor installed on the cooking machine; and images of the food to be processed can be acquired by an image sensor installed on the cooking machine, and other attribute data can be obtained by analyzing the images of the food to be processed. Of course, images can also be acquired by a video sensor.

[0032] Step 102: Obtain cooking data related to the cooking task from the server based on the dish's identifier.

[0033] Among them, cooking data is used to cook the ingredients.

[0034] Specifically, after obtaining the identifier of a dish, the cooking machine can retrieve the cooking data matching that identifier from the server. It should be noted that the server can store a large amount of recipe-related data, and this data is stored in a correspondence between the dish names and the recipe information.

[0035] In other embodiments of this application, cooking data may include historical cooking data; or, cooking data may include historical cooking data and standard ingredient attribute data.

[0036] Step 103: Based on the attribute data of the ingredients to be processed, the cooking requirements information, and the cooking data, determine the target cooking strategy corresponding to the cooking task.

[0037] In this embodiment, after the cooking machine obtains the cooking data from the server, it can jointly determine the target cooking strategy matching the current cooking task based on the attribute data of the ingredients to be processed, the cooking requirement information, and the cooking data. It should be noted that the target cooking strategy may refer to a target recipe.

[0038] It should be noted that the cooking machine can combine the attribute data of the ingredients to be processed, cooking requirements information, and cooking data to obtain the target recipe that can complete the cooking task from the local machine, and then proceed with the subsequent cooking operations. Specifically, the cooking machine can first determine whether the attribute data of the ingredients to be processed matches the basic attribute data corresponding to the cooking task, and then determine the target recipe based on the determination result, combined with the cooking requirements information and cooking data.

[0039] Step 104: Perform cooking operations based on the target cooking strategy and the first multimodal data related to the ingredients to be processed, in order to complete the cooking of the ingredients to be processed.

[0040] In this embodiment, after determining the target recipe, the cooking machine can perform corresponding cooking operations on the ingredients according to the preset cooking actions corresponding to the target recipe and in conjunction with the multimodal first data related to the ingredients to be processed, thereby realizing the cooking of the ingredients to be processed and obtaining the corresponding dish. It should be noted that the solution of this application, by combining a lightweight edge-cloud collaborative architecture, eliminates the need for a high-performance processor in the local processor of the cooking machine, greatly reducing hardware costs and significantly improving the quality of the dishes, thus enhancing the performance of human-computer interaction.

[0041] The first data can be multiple data points related to the food to be processed, collected from multiple different types of sensors across multiple modalities. It should be noted that the first data points for different modalities can be collected from different types of sensors.

[0042] Based on the foregoing embodiments, in other embodiments of this application, reference is made to... Figure 2 As shown, step 103 above can be achieved in the following way: Step 103A: If it is determined that the attribute data of the ingredients to be processed matches the basic attribute data corresponding to the cooking task, determine the target cooking strategy based on the attribute data of the ingredients to be processed, the cooking requirement information, and the historical cooking data corresponding to the cooking task.

[0043] The cooking data includes historical cooking data.

[0044] Specifically, the cooking machine can first match the attribute data of the ingredients to be processed with the basic attribute data corresponding to the cooking task stored locally. If the attribute data of the ingredients to be processed matches the basic attribute data, the target recipe can be determined based on the attribute data of the ingredients to be processed and the historical cooking data obtained from the cloud.

[0045] It's important to note that historical cooking data refers to the data used in the past cooking of dishes using the corresponding ingredients. By incorporating this historical cooking data when determining the target recipe, the resulting recipe is better suited to the current cooking task, ensuring the final dish meets the user's taste preferences. Furthermore, historical cooking data can include high-frequency cooking records of the cooking machine, commonly used ingredient quantities, fixed cutting habits, and historical success parameters.

[0046] In other embodiments of this application, reference is made to Figure 2 As shown, the method may further include: Step 103B: If it is determined that the attribute data of the ingredient to be processed does not match the basic attribute data corresponding to the cooking task, update the attribute data of the ingredient to be processed based on the standard ingredient attribute data corresponding to the cooking task.

[0047] Specifically, if the attribute data of the ingredient to be processed does not match the basic attribute data, then the attribute data of the ingredient to be processed needs to be updated with the standard ingredient attribute data that matches the name of the dish, obtained from the server. It should be noted that the basic attribute data may include standard ingredient type, standard cutting method, standard weight range, standard initial cooking parameters, etc.; in addition, the server stores the standard ingredient type, standard cutting method, standard weight range, and standard initial cooking parameters for each dish.

[0048] In other embodiments of this application, invalid attribute data can be directly replaced and the cooking program can be re-initialized without requiring the user to perform any operations; alternatively, sensors can be continuously enabled to monitor the cooking status of ingredients in real time, and the standard parameters sent from the cloud can be dynamically fine-tuned through local real-time ripeness feedback to solve the deviation problem.

[0049] In this embodiment, a mismatch between the attribute data and basic attribute data of the food to be processed may include the inability to determine the type of food, invalid recognition of the food cutting form (inability to distinguish between whole slices, shredded, diced, and sliced ​​states, etc.), inability to obtain effective food weight (abnormal fluctuations in weight sensor data, vibration interference, false judgments due to no load, etc.), and failure of image features (obstruction by oil fumes, abnormal lighting, reflection interference, etc.). It should be noted that the type of food can be determined through a visual food classification model; that is, if the confidence level output by the visual food classification model is lower than a preset threshold, it is considered that the type of food cannot be accurately determined.

[0050] In other embodiments of this application, if the specific ingredient type cannot be identified, the standard ingredient ratio can be locked based on the recipe name selected by the user. The cloud returns the fixed main ingredient and side dish types corresponding to the recipe, directly overriding the unknown ingredient parameters locally. At the same time, the cloud synchronously sends the standard heating curve, cooking characteristic threshold, and ingredient addition sequence rules corresponding to the dish, so that the local control module operates according to the standard ingredient characteristics of the recipe, avoiding parameter errors caused by unknown ingredient identification.

[0051] In other embodiments of this application, if the cutting status recognition fails, the user's most frequent cutting method for the same dish in history is retrieved as the default form; if there is no historical data, the industry default standard form is called (meat is shredded / sliced, vegetables are sliced, and root vegetables are chunks). The cloud corrects three sets of core parameters locally based on the default cutting form: (1) Correcting heating power: finely chopped ingredients have reduced peak power and extended constant temperature time; large pieces of ingredients have increased initial heating power; (2) Correcting stirring speed: finely chopped ingredients have reduced stirring speed to prevent them from breaking apart; large pieces of ingredients have increased stirring frequency to ensure uniform heating; (3) Correcting the maturity judgment threshold: the multimodal ripening judgment standard is recalibrated based on the contact area of ​​the ingredients.

[0052] In other embodiments of this application, if weight recognition fails, the cloud employs a two-layer fallback weighting strategy to correct local cooking parameters: firstly, it retrieves the average weight of the same dish from the user's historical records as the default portion size; secondly, it retrieves the median of the recommended portion size range from the cloud-based recipe standard; thus, based on the corrected standard weight, the cloud recalculates and sends the following parameters to the local system: heating power ratio per unit ingredient, total cooking time, moisture evaporation threshold, and multi-level ingredient feeding intervals. The local adaptive control module directly overrides the original initialization parameters, achieving adaptive correction of reduced power and shorter cooking time for small portions, and increased power and longer cooking time for large portions.

[0053] Step 103C: Determine the target cooking strategy based on the updated attribute data of the ingredients to be processed, cooking requirement information, and historical cooking data.

[0054] The cooking data includes standard ingredient attribute data and historical cooking data.

[0055] Specifically, after the attribute data of the ingredients to be processed is updated, the target recipe can be determined based on the updated attribute data of the ingredients to be processed, the cooking requirements information, and the historical cooking data obtained from the server.

[0056] In other embodiments of this application, reference is made to Figure 2 As shown, step 104 can be achieved in the following way: Step 104A: During the cooking operation based on the target cooking strategy, multiple first data related to the ingredients to be processed in the pot are collected by multiple sensors.

[0057] Among them, multiple primary data represent the current state of the food to be processed; each sensor belongs to a different category.

[0058] Specifically, multiple sensors can include vision sensors, temperature sensors, weight sensors, gas sensors, sound sensors, motion sensors, and other sensors capable of acquiring information in multiple modalities.

[0059] It should be noted that the vision sensor can be a high-definition wide-angle camera, mounted on top of the pot, with a frame rate of 30fps, equipped with an oil-proof glass cover and LED supplementary light, used to capture images of food and identify the type of food, its cutting status, quantity, and color and texture changes during cooking. The temperature sensor can be a negative temperature coefficient (NTC) thermistor, installed at the center of the bottom of the pot, with a sampling frequency of 10Hz, a measurement range of 0-300℃, and an accuracy of ±1℃. It is also equipped with an infrared temperature sensor for non-contact measurement of the food surface temperature. The weight sensor can be a high-precision resistance strain gauge load cell, installed below the pot support and below the food container, with a range of 0-10kg and an accuracy of ±5g, used to measure the total weight of the food and the amount of moisture evaporated in real time. The gas sensor can employ a volatile organic compound (VOC) sensor array, including various sensitive elements, installed at the exhaust vent with a sampling frequency of 1Hz. It detects VOCs generated during cooking, reflecting the degree of doneness, aroma development, and whether the food is burnt. The sound sensor uses a microelectromechanical system (MEMS) microphone, installed on the side of the pot, with a sampling frequency of 44.1kHz. It collects the sounds of stirring and food popping, aiding in determining the food's condition. The motion sensor can be an inertial sensor, installed on moving parts such as the stirring blade, stirring arm, and pot body, with a sampling frequency of 1kHz. It collects the status of each moving mechanism, determining whether the food is burnt, food is stuck, or there is equipment malfunction.

[0060] Step 104B: Determine the first degree of maturity of the ingredients to be processed in the pot based on multiple first data.

[0061] Specifically, multiple data points collected by multiple sensors can be processed to determine the initial degree of cooking of the food inside the pot.

[0062] It should be noted that the first feature information of the first data collected by each sensor can be determined first, and then the first degree of cooking of the food to be processed in the pot can be determined based on multiple first feature information.

[0063] Step 104C: If the first maturity meets the first target maturity threshold, perform the feeding operation on the ingredients to be processed in the pot.

[0064] The first target maturity threshold can be a pre-set range of maturity levels, or it can be set according to actual needs, specific application scenarios, and different dishes.

[0065] In this embodiment, if the first maturity of the food to be processed in the pot is determined to be within the range of the first target maturity threshold, it means that the maturity of the food to be processed in the pot has reached a certain requirement, and then the operation of adding seasonings to the food to be processed in the pot can be performed. Of course, when the first maturity of the food to be processed in the pot meets the first target maturity threshold, it can be further determined whether the timing for adding ingredients has been met. If the timing for adding ingredients is determined to be met, then the operation of adding seasonings can be performed.

[0066] Step 104D: If it is determined that the cooking parameters of the ingredients to be processed meet the target requirements, output a prompt message to indicate that cooking is complete.

[0067] Specifically, during the cooking process, various parameters of the ingredients inside the pot can be continuously monitored. If all parameters meet the target requirements, it means that the dish has been prepared. At this time, a prompt message can be output to indicate that the cooking is complete, and operations such as taking the dish out of the pot can be performed.

[0068] It should be noted that after the dish is cooked, a cooking review can be performed, allowing for recipe optimization based on factors such as cooking effect and efficiency, combined with human evaluation results. Additionally, the cooking process and results can be stored locally on the cooking machine and, for example... Figure 4 The recipe database of the edge-cloud collaboration layer server, as shown, allows for continuous accumulation of experience and refinement of cooking skills and recipes. This enables the support of new dishes and functions without the need to replace hardware, thus extending the lifespan of the cooking machine.

[0069] In other embodiments of this application, reference is made to Figure 3 As shown, step 104B above, which determines the first degree of cooking of the food to be processed in the pot based on multiple first data points, may include: Step 104B1: Determine the first feature information for each first data point.

[0070] In this process, feature extraction can be performed on each first data point to obtain the first feature information of each first data point.

[0071] In one feasible embodiment, it can be achieved by, as follows Figure 4The multimodal fusion module in the control layer of the multimodal cooking robot system shown realizes feature extraction; specifically, different first feature information can be extracted for different first data collected by different sensors; lightweight MobileNetV3 is used to extract spatial features for visual data; and 1D-CNN is used to extract temporal features for temperature, weight, gas and sound data.

[0072] Step 104B2: Determine the weight coefficient corresponding to each first data point based on the cooking scenario corresponding to the cooking task.

[0073] Specifically, the weighting coefficients for each type of sensor are different for different cooking scenarios.

[0074] In one feasible embodiment, for the scenario of stewing in water, the weight of the temperature sensor and the gas sensor can be set to be the highest; for the scenario of stir-frying meat, the weight of the visual sensor, the weight of the weight sensor, and the odor sensor can be set to be the highest; for the scenario where the oil fumes greatly obstruct the camera, the weight of the visual sensor can be set to be lower, and the weight of the temperature sensor, the gas sensor, and the sound sensor can be set to be higher; for the scenario of stir-frying with little oil, the weight sensor can be set to be higher.

[0075] Step 104B3: Determine the first fusion feature based on multiple first feature information and the weight coefficient corresponding to each first data.

[0076] Specifically, the first fusion feature can be obtained by weighting and summing all the first feature information according to the weight coefficient corresponding to each first data point. It should be noted that each first feature information can be weighted according to the weight coefficient corresponding to each first data point, and the weighted features can be concatenated to obtain the global fusion feature vector (i.e., the first fusion feature).

[0077] Step 104B4: Determine the first degree of maturity of the ingredients to be processed in the pot based on the first fusion feature.

[0078] Among them, it can be achieved through Figure 4 The maturity recognition module shown processes the first fused feature to obtain the first maturity level. It should be noted that the first maturity level can be obtained by using the first fused feature as input information for a specific model, processing the first fused feature using that specific model, and then outputting the first maturity level. Furthermore, this specific model can be a Transformer network model.

[0079] It should be noted that the maturity value that the model can output can range from 0 to 100; where 0 is unripe and 100 is fully ripe; the corresponding maturity levels can include unripe, three-quarters ripe, five-quarters ripe, seven-quarters ripe, fully ripe, overripe, etc.

[0080] In other embodiments of this application, if the proportion of burnt gas, sudden temperature rise, and food caramelization detected in the first fusion feature exceeds the standard, the model directly determines the risk of overcooking, triggers power reduction in advance, and does not perform subsequent determination of the first target maturity threshold.

[0081] In other embodiments of this application, the first feature information for determining each first data may include: Process multiple sets of primary data; Multiple first feature information are extracted from each processed first data.

[0082] Specifically, multiple sets of first data can be preprocessed, and feature information of each preprocessed first data set can be extracted to obtain multiple sets of first feature information. Furthermore, a corresponding model can be used to extract the first feature information (i.e., sequential feature information and image feature information) of each first data set.

[0083] It should be noted that each piece of first data can undergo preprocessing including time synchronization, outlier removal, and normalization. In one feasible embodiment, timestamps can be unified, and all first data can be aligned within a 10-100ms window. Outliers with delays exceeding 200ms are discarded to avoid timing misalignments. For example, temperature / weight: 100Hz high-frequency sampling; visual: 5-30fps frame images; gas, microphone sound: 1Hz low-frequency sampling. Temperature: Sudden changes greater than 30℃ are considered interference from oil fume obstruction, and this temperature data is discarded; weight: Instantaneous ±100g is considered stirring vibration error, and this weight data is filtered; visual: Frames with large areas of oil fume obstruction or brightness below a threshold are marked as invalid; gas: Peak values ​​of kitchen fumes and external odors are directly truncated; sound: Opening lids and collision noise segments are deleted. All first data are mapped to the 0-100 range to eliminate dimensional differences: temperature 0-300℃, weight 0-5kg, gas concentration 0-10000ppm, sound decibels, and image pixel features are standardized.

[0084] In other embodiments of this application, determining the first degree of maturity of the food to be processed in the pot based on the first fusion feature may include: Acquire multiple initial feature information of the food to be processed, corresponding to multiple sensors; Initial fusion features are determined based on multiple initial feature information; The second fusion feature is determined based on the first fusion feature and the initial fusion feature; The first maturity level is determined based on the second fusion feature.

[0085] It should be noted that after determining the first fusion feature, the first fusion feature can be processed with the initial fusion feature corresponding to the initial state of the food to be processed, and the first maturity can be determined based on the processing result.

[0086] Specifically, initial data of the ingredients to be processed can be acquired from multiple sensors, and features can be extracted from each initial data point to obtain initial features. All initial features are then fused to obtain initial fused features. Next, the difference between the first fused feature and the initial fused features is calculated to obtain a second fused feature, which is then processed and output to determine the first degree of ripeness. Alternatively, the initial fused features can be determined and stored in the cooking machine at the beginning of cooking. In this way, the differential feature processing can eliminate judgment biases caused by initial weight, initial temperature, and initial appearance of the ingredients, improving the accuracy of the final determined ripeness.

[0087] In other embodiments of this application, reference is made to Figure 3 As shown, step 104C may include: Step 104C1: If all N first maturity values ​​determined based on the N second fusion features determined within the target time period meet the first target maturity threshold, perform a feeding operation on the ingredients to be processed in the pot.

[0088] Specifically, N second fusion features can be continuously determined within the target duration. When all N first maturity values ​​obtained from these N second fusion features meet the first target maturity threshold, the ingredients to be processed in the pot can be added. It should be noted that by capturing the changing patterns of features during the continuous cooking process, rather than a single static data point, the accuracy of the obtained maturity values ​​can be further improved, resulting in higher quality cooked dishes.

[0089] In other embodiments of this application, the above-mentioned feeding operation of the ingredients to be processed in the pot may include: Multiple secondary data related to the food to be processed are collected inside the pot using multiple devices; Determine the second feature information for each second data point; The third fusion feature is determined based on multiple second feature information and the weight coefficient corresponding to each second data; The second degree of maturity of the food to be processed inside the pot is determined based on the third fusion feature; If the second maturity level meets the second target maturity threshold, the feeding operation is performed on the ingredients to be processed in the pot.

[0090] It should be noted that once the initial maturity of the ingredients in the pot meets the first target maturity threshold, it can be further determined whether the state of the ingredients in the pot has reached the optimal time for addition. If the time has been reached, the addition operation is performed. The weight coefficient corresponding to each second data point can be determined based on the cooking scenario corresponding to the cooking task. Furthermore, sensors continuously monitor data during the addition process, such as weight sensors monitoring the weight of seasonings, temperature sensors monitoring changes in the pot temperature, and visual sensors monitoring the state of oil fumes and the coloring of the ingredients. If the seasoning and coloring effects do not meet customer expectations, the system automatically corrects by increasing the amount of ingredients added.

[0091] Specifically, features can be extracted from the second data collected by each sensor to obtain multiple second feature information, and all the second feature information can be weighted and fused to obtain a third fused feature. Then, the third fused feature is processed to output the second maturity of the ingredients to be processed in the pot. If the second maturity is determined to meet the second target maturity threshold, it is determined that the feeding time has been reached, and the ingredients to be processed in the pot can be fed. It should be noted that for different dishes, the ingredient timing of each dish can be broken down, and different cooking adaptation nodes for different ingredients can be defined: hard ingredients (shredded meat, potatoes, meat) need to be cooked at high temperature first, soft ingredients (green peppers, green vegetables, mushrooms) need to be stir-fried quickly later, and seasonings (salt, light soy sauce, sauce) need to be added before serving, etc. That is, a corresponding second target maturity threshold can be preset for each type of ingredient.

[0092] In other embodiments of this application, multiple ingredient addition operations can be performed during the cooking process. That is, after each addition, the cooking timing node can be updated synchronously, and data from multiple sensors can be collected again to iteratively determine the timing of adding the next level of ingredients, forming a closed-loop cooking control process of "sensing-determining-adding-re-sensing". It should be noted that if the data from individual sensors deviates from the set value during the addition process, data from other sensors can be used for cross-validation to correct the failure of individual sensors.

[0093] In this way, relying on real-time data from five modal sensors (vision, temperature, weight, gas, and sound), combined with the cooking sequence logic of ingredients in recipes, the system can dynamically determine the timing of ingredient addition based on the actual cooking state of the ingredients. This eliminates the need for manually preset fixed addition times and can adapt to different ingredient quantities, initial states, and cooking temperatures, accurately determining the optimal addition points for various auxiliary ingredients, side dishes, and seasonings. It automatically delays addition points for frozen ingredients and extra-large portions to ensure the main ingredient is fully cooked; and it pre-adapts addition times for room-temperature fresh ingredients and small portions to prevent overcooking and aging. Simultaneously, it dynamically matches the cooking temperature in real-time, shortening the addition interval for high-temperature stir-frying scenarios and delaying addition points for low-temperature slow-cooking scenarios, adapting to all cooking needs.

[0094] In other embodiments of this application, the determination of the second degree of cooking of the food to be processed in the pot based on the third fusion feature includes: The fourth fusion feature is determined based on the third fusion feature and the initial fusion feature; The second degree of maturity of the food to be processed in the pot is determined based on the fourth fusion feature.

[0095] It should be noted that after determining the third fusion feature, the third fusion feature can be processed with the initial fusion feature corresponding to the initial state of the food to be processed, and the second maturity can be determined based on the processing result.

[0096] Specifically, the fourth fusion feature can be obtained by performing a difference operation between the third fusion feature and the initial fusion feature, and then processed to output the second maturity level. In this way, the processing of differential features can eliminate the judgment bias caused by the initial weight, initial temperature, and initial appearance of the ingredients, thereby improving the accuracy of the final determination of the feeding timing.

[0097] In other embodiments of this application, if the second maturity meets the second target maturity threshold, performing a feeding operation on the ingredients to be processed in the pot may include: If all N second maturity values ​​determined based on the N fourth fusion features within the target time period meet the second target maturity threshold, then the feeding operation is performed on the ingredients to be processed in the pot.

[0098] Specifically, N fourth fusion features can be continuously determined within the target duration. When all N second maturity levels obtained from these N fourth fusion features meet the second target maturity threshold, the feeding timing is determined, and the ingredients to be processed in the pot can be fed. It should be noted that by capturing the changing patterns of features during continuous cooking, rather than a single static data point, the accuracy of the feeding timing can be further improved, avoiding problems such as incorrect, premature, or delayed feeding caused by instantaneous data fluctuations. Furthermore, the value of N can be determined based on specific needs and actual application scenarios.

[0099] In other embodiments of this application, the method may further include: During the process of adding the ingredients to be processed into the pot based on the target cooking strategy, sensor data collected by multiple sensors corresponding to each cooking stage and / or the currently added ingredients are acquired at different cooking stages. Cooking operations corresponding to different cooking stages are executed based on data from multiple sensors.

[0100] The different cooking stages can include the hot pan stage, the hot oil stage, the sautéing stage, and the addition of ingredients stage. For each cooking stage, corresponding cooking operations can be performed based on sensor data collected by multiple sensors.

[0101] In one feasible embodiment, during the preheating stage, a temperature sensor monitors the pot and, based on ambient temperature and humidity, adjusts the preheating temperature accordingly. In winter, when the ambient temperature is low, a preheating temperature compensation T0 is applied to ensure a high initial temperature for the hot oil and consistent heating effect. During the oil heating stage, especially for animal fats, if the initial temperature deviates, visual recognition of the oil's heat signature and patterns can be used to identify the uniformity of the hot oil. If the oil heating process deviates from the set timing, the heating power is dynamically adjusted to achieve the best cooking results. During the stir-frying stage, a sound sensor monitors the sound of the stir-frying ingredients. When the moisture in the ingredients has evaporated, the sound volume detected by the sound sensor decreases to the target value. During the meat addition stage, a video sensor monitors the state of the meat being broken up, and a temperature sensor monitors the pot temperature, adjusting the stirring speed and heating speed to ensure even cooking. During the chili addition stage, a visual sensor monitors the position of the chilies in the pot and dynamically corrects the stirring posture to ensure better and more even heating of the ingredients.

[0102] It should be noted that during the entire cooking process of the dish, it can be done through... Figure 4 The adaptive control module automatically adjusts cooking parameters such as heating power, stirring speed, and stirring direction to achieve optimal cooking results. It also features a scorching warning mechanism; when the gas sensor detects scorching gases or the temperature rise rate exceeds a threshold, it immediately reduces heating power and increases stirring speed. Furthermore, the intelligent ingredient recognition function eliminates the need for manual recipe selection and parameter input, reducing operational steps and achieving truly fully automatic cooking. It dynamically adjusts heating power and exhaust airflow based on real-time cooking status, resulting in lower average energy consumption and reduced exhaust energy consumption compared to fixed power control.

[0103] In other embodiments of this application, such as Figure 4 The multimodal cooking robot system to which the method provided in this application is applicable also has an execution layer and a collaboration layer with other devices. The execution layer may include a heating module, a stirring module, a food ingredient feeding module, a powder feeding module, and a liquid feeding module. The collaboration layer with other devices may include a Kitchen Display System (KIDS), a rice steamer, a rice dispenser, a steam oven, a deep fryer, a dishwasher, and a fresh air exhaust system. The cooking robot can interact with these devices in the collaboration layer to improve overall food preparation efficiency. KIDS enables core digital functions for the kitchen, including paperless order processing, real-time order processing, intelligent order allocation, and progress tracking.

[0104] The control method for the multimodal cooking machine provided in the embodiments of this application can determine the cooking strategy based on the cooking data related to the current cooking task on the server, the attribute data of the ingredients corresponding to the current cooking task, and the cooking requirements when cooking ingredients. This enables the cooking of ingredients. Obviously, the cooking strategy for cooking ingredients is determined based on the cooking data obtained from the server, the attribute data of the ingredients, and the cooking requirements. Both local data and cloud data are taken into account, thereby solving the problem of insufficient control accuracy of the cooking recipe determined in related technologies, and thus improving cooking efficiency.

[0105] Based on the foregoing embodiments, embodiments of this application provide a control device, referring to... Figure 5 As shown, the control device 2 may include: a first determining unit 21, an acquiring unit 22, a second determining unit 23, and a processing unit 24, wherein: The first determining unit 21 is used to determine the attribute data of the ingredients to be processed corresponding to the cooking task after obtaining the cooking task; wherein, the cooking task includes the identification of the dish and cooking requirement information; The acquisition unit 22 is used to acquire cooking data related to the cooking task from the server based on the identifier of the dish; wherein, the cooking data is used to cook the ingredients; The second determining unit 23 is used to determine the target cooking strategy corresponding to the cooking task based on the attribute data of the ingredients to be processed, cooking requirement information and cooking data. The processing unit 24 is used to perform cooking operations based on the target cooking strategy and multimodal first data related to the ingredients to be processed, so as to complete the cooking of the ingredients to be processed.

[0106] In other embodiments of this application, the second determining unit 23 is further configured to determine a target cooking strategy based on the attribute data of the ingredient to be processed, cooking requirement information and historical cooking data corresponding to the cooking task if it is determined that the attribute data of the ingredient to be processed matches the basic attribute data corresponding to the cooking task; wherein, the cooking data includes historical cooking data.

[0107] In other embodiments of this application, the second determining unit 23 is further configured to perform the following steps: If it is determined that the attribute data of the ingredient to be processed does not match the basic attribute data corresponding to the cooking task, update the attribute data of the ingredient to be processed based on the standard ingredient attribute data corresponding to the cooking task. Based on the updated attribute data of the ingredients to be processed, cooking requirement information, and historical cooking data, the target cooking strategy is determined; the cooking data includes standard ingredient attribute data and historical cooking data.

[0108] In other embodiments of this application, the processing unit 24 is further configured to perform the following steps: During the cooking operation based on the target cooking strategy, multiple sensors collect multiple multimodal first data related to the food to be processed inside the pot; wherein, the multiple first data represent the current state of the food to be processed; each sensor is of a different type; The initial degree of cooking of the food to be processed inside the pot is determined based on multiple primary data points; If the first maturity level meets the first target maturity threshold, the feeding operation is performed on the ingredients to be processed in the pot. If the cooking parameters of the ingredients to be processed are determined to meet the target requirements, output a prompt message to indicate that cooking is complete.

[0109] In other embodiments of this application, the processing unit 24 is further configured to perform the following steps: Determine the first feature information for each first data point; The weight coefficient for each first data point is determined based on the cooking scenario corresponding to the cooking task. The first fusion feature is determined based on multiple first feature information and the weight coefficient corresponding to each first data; The first degree of maturity of the food to be processed in the pot is determined based on the first fusion feature.

[0110] In other embodiments of this application, the processing unit 24 is further configured to perform the following steps: Process multiple sets of primary data; Multiple first feature information are extracted from each processed first data.

[0111] In other embodiments of this application, the processing unit 24 is further configured to perform the following steps: Acquire multiple initial feature information of the food to be processed, corresponding to multiple sensors; Initial fusion features are determined based on multiple initial feature information; The second fusion feature is determined based on the first fusion feature and the initial fusion feature; The first maturity level is determined based on the second fusion feature.

[0112] In other embodiments of this application, the processing unit 24 is further configured to perform a feeding operation on the ingredients to be processed in the pot if all N first maturity values ​​determined based on N second fusion features determined within a target time period meet the first target maturity threshold.

[0113] In other embodiments of this application, the processing unit 24 is further configured to perform the following steps: If the first maturity level meets the first target maturity threshold, multiple second data related to the food to be processed are collected inside the pot through multiple sensors; Determine the second feature information for each second data point; The third fusion feature is determined based on multiple second feature information and the weight coefficient corresponding to each second data; The second degree of maturity of the food to be processed inside the pot is determined based on the third fusion feature; If the second maturity level meets the second target maturity threshold, the feeding operation is performed on the ingredients to be processed in the pot.

[0114] In other embodiments of this application, the processing unit 24 is further configured to perform the following steps: The fourth fusion feature is determined based on the third fusion feature and the initial fusion feature; The second degree of maturity of the food to be processed in the pot is determined based on the fourth fusion feature.

[0115] In other embodiments of this application, the processing unit 24 is further configured to perform a feeding operation on the ingredients to be processed in the pot if the N second maturity values ​​determined based on the N fourth fusion features determined within the target time period all meet the second target maturity threshold.

[0116] In other embodiments of this application, the processing unit 24 is further configured to perform the following steps: During the process of adding the ingredients to be processed into the pot based on the target cooking strategy, sensor data collected by multiple sensors corresponding to each cooking stage and / or the currently added ingredients are acquired at different cooking stages. Cooking operations corresponding to different cooking stages are executed based on data from multiple sensors.

[0117] It should be noted that the specific implementation process of the steps performed by each unit in the embodiments of this application can be referred to Figure 1-3 The implementation process of the control method for the multimodal cooking machine provided in the corresponding embodiment will not be described in detail here.

[0118] The control device provided in the embodiments of this application can determine a cooking strategy based on cooking data related to the current cooking task from the server, the attribute data of the ingredients corresponding to the current cooking task, and the cooking requirements when cooking ingredients. Then, it combines multimodal first data related to the ingredients to be processed to realize the cooking of the ingredients. Obviously, the cooking strategy for cooking ingredients is determined based on cooking data obtained from the server, the attribute data of the ingredients, and the cooking requirements. Both local data and cloud data are taken into account, thereby solving the problem that the control accuracy of the cooking recipe determined in related technologies is not high enough, and thus improving cooking efficiency.

[0119] Based on the foregoing embodiments, embodiments of this application provide a cooking machine that can be applied to... Figure 1-3 In the control method of the multimodal cooking machine provided in the corresponding embodiment, refer to Figure 6As shown, the cooking machine 3 may include: a processor 31, a memory 32, and a communication bus 33, wherein: Communication bus 33 is used to realize the communication connection between processor 31 and memory 32; The memory 32 is used to store computer programs that can run on the processor 31; Processor 31 is used to run computer programs to perform the following steps: After obtaining the cooking task, determine the attribute data of the ingredients to be processed corresponding to the cooking task; the cooking task includes the dish's identifier and cooking requirement information; Based on the dish's identifier, cooking data related to the cooking task is obtained from the server; this cooking data is used to cook the ingredients. Based on the attribute data of the ingredients to be processed, cooking requirements information, and cooking data, determine the target cooking strategy corresponding to the cooking task; Cooking operations are performed based on the target cooking strategy and multimodal first data related to the ingredients to be processed, in order to complete the cooking of the ingredients to be processed.

[0120] In other embodiments of this application, the processor 31 is used to run computer programs and can also perform the following steps: If the attribute data of the ingredients to be processed is determined to match the basic attribute data corresponding to the cooking task, the target cooking strategy is determined based on the attribute data of the ingredients to be processed, the cooking requirement information, and the historical cooking data corresponding to the cooking task; wherein, the cooking data includes historical cooking data.

[0121] In other embodiments of this application, the processor 31 is used to run computer programs and can also perform the following steps: If it is determined that the attribute data of the ingredient to be processed does not match the basic attribute data corresponding to the cooking task, update the attribute data of the ingredient to be processed based on the standard ingredient attribute data corresponding to the cooking task. Based on the updated attribute data of the ingredients to be processed, cooking requirement information, and historical cooking data, the target cooking strategy is determined; the cooking data includes standard ingredient attribute data and historical cooking data.

[0122] In other embodiments of this application, the processor 31 is used to run computer programs and can also perform the following steps: During the cooking operation based on the target cooking strategy, multiple sensors collect multiple multimodal first data related to the food to be processed inside the pot; wherein, the multiple first data represent the current state of the food to be processed; each sensor is of a different type; The initial degree of cooking of the food to be processed inside the pot is determined based on multiple primary data points; If the first maturity level meets the first target maturity threshold, the feeding operation is performed on the ingredients to be processed in the pot. If the cooking parameters of the ingredients to be processed are determined to meet the target requirements, output a prompt message to indicate that cooking is complete.

[0123] In other embodiments of this application, the processor 31 is used to run computer programs and can also perform the following steps: Determine the first feature information for each first data point; The weight coefficient for each first data point is determined based on the cooking scenario corresponding to the cooking task. The first fusion feature is determined based on multiple first feature information and the weight coefficient corresponding to each first data; The first degree of maturity of the food to be processed in the pot is determined based on the first fusion feature.

[0124] In other embodiments of this application, the processor 31 is used to run computer programs and can also perform the following steps: Process multiple sets of primary data; Multiple first feature information are extracted from each processed first data.

[0125] In other embodiments of this application, the processor 31 is used to run computer programs and can also perform the following steps: Acquire multiple initial feature information of the food to be processed, corresponding to multiple sensors; Initial fusion features are determined based on multiple initial feature information; The second fusion feature is determined based on the first fusion feature and the initial fusion feature; The first maturity level is determined based on the second fusion feature.

[0126] In other embodiments of this application, the processor 31 is used to run computer programs and can also perform the following steps: If all N first maturity values ​​determined based on the N second fusion features within the target time period meet the first target maturity threshold, then a feeding operation is performed on the ingredients to be processed in the pot.

[0127] In other embodiments of this application, the processor 31 is used to run computer programs and can also perform the following steps: If the first maturity level meets the first target maturity threshold, multiple second data related to the food to be processed are collected inside the pot through multiple sensors; Determine the second feature information for each second data point; The third fusion feature is determined based on multiple second feature information and the weight coefficient corresponding to each second data; The second degree of maturity of the food to be processed inside the pot is determined based on the third fusion feature; If the second maturity level meets the second target maturity threshold, the feeding operation is performed on the ingredients to be processed in the pot.

[0128] In other embodiments of this application, the processor 31 is used to run computer programs and can also perform the following steps: The fourth fusion feature is determined based on the third fusion feature and the initial fusion feature; The second degree of maturity of the food to be processed in the pot is determined based on the fourth fusion feature.

[0129] In other embodiments of this application, the processor 31 is used to run computer programs and can also perform the following steps: If all N second maturity values ​​determined based on the N fourth fusion features within the target time period meet the second target maturity threshold, then the feeding operation is performed on the ingredients to be processed in the pot.

[0130] In other embodiments of this application, the processor 31 is used to run computer programs and can also perform the following steps: During the process of adding the ingredients to be processed into the pot based on the target cooking strategy, sensor data collected by multiple sensors corresponding to each cooking stage and / or the currently added ingredients are acquired at different cooking stages. Cooking operations corresponding to different cooking stages are executed based on data from multiple sensors.

[0131] It should be noted that a detailed description of the steps performed by the processor can be found in [reference needed]. Figure 1-3 The control method for the multimodal cooking machine provided in the corresponding embodiment will not be described in detail here.

[0132] The cooking machine provided in the embodiments of this application can determine a cooking strategy based on cooking data related to the current cooking task from the server, the attribute data of the ingredients corresponding to the current cooking task, and the cooking requirements when cooking ingredients. Then, it combines multimodal first data related to the ingredients to be processed to realize the cooking of the ingredients. Obviously, the cooking strategy for cooking ingredients is determined based on cooking data obtained from the server, the attribute data of the ingredients, and the cooking requirements. Both local data and cloud data are taken into account, thereby solving the problem of insufficient control accuracy of the cooking recipe determined in related technologies, and thus improving cooking efficiency.

[0133] Based on the foregoing embodiments, embodiments of this application provide a computer-readable storage medium storing one or more programs, which can be executed by one or more processors 31 to implement... Figure 1-3 The corresponding embodiment provides the steps of the control method for a multimodal cooking machine.

[0134] Based on the foregoing embodiments, embodiments of this application provide a computer program product, including a computer program that can be executed by a processor 31 to perform... Figure 1-3 The corresponding embodiment provides the steps of the control method for a multimodal cooking machine.

[0135] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0136] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0137] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0138] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0139] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A control method for a multimodal cooking machine, characterized in that, The method includes: After obtaining the cooking task, determine the attribute data of the ingredients to be processed corresponding to the cooking task; wherein, the cooking task includes the dish's identifier and cooking requirement information; Based on the identifier of the dish, cooking data related to the cooking task is obtained from the server; wherein, the cooking data is used to cook the ingredients; Based on the attribute data of the ingredients to be processed, the cooking requirement information, and the cooking data, the target cooking strategy corresponding to the cooking task is determined; The cooking operation is performed based on the target cooking strategy and the first multimodal data related to the ingredients to be processed, so as to complete the cooking of the ingredients to be processed.

2. The method according to claim 1, characterized in that, The step of determining the target cooking strategy corresponding to the cooking task based on the attribute data of the ingredients to be processed, the cooking requirement information, and the cooking data includes: If it is determined that the attribute data of the ingredient to be processed matches the basic attribute data corresponding to the cooking task, the target cooking strategy is determined based on the attribute data of the ingredient to be processed, the cooking requirement information, and the historical cooking data corresponding to the cooking task; wherein, the cooking data includes the historical cooking data.

3. The method according to claim 2, characterized in that, The method further includes: If it is determined that the attribute data of the ingredient to be processed does not match the basic attribute data corresponding to the cooking task, the attribute data of the ingredient to be processed is updated based on the standard ingredient attribute data corresponding to the cooking task. The target cooking strategy is determined based on the updated attribute data of the ingredients to be processed, the cooking requirement information, and the historical cooking data; wherein the cooking data includes standard ingredient attribute data and the historical cooking data.

4. The method according to claim 1, characterized in that, The cooking operation based on the target cooking strategy and multimodal first data related to the ingredients to be processed includes: During the cooking operation based on the target cooking strategy, multiple sensors collect multimodal first data related to the food to be processed within the pot; wherein, the multiple first data represent the current state of the food to be processed; each sensor is of a different type; The first degree of cooking of the food to be processed in the pot is determined based on the multiple first data. If the first maturity level meets the first target maturity threshold, a feeding operation is performed on the ingredients to be processed in the pot. If the cooking parameters of the ingredients to be processed are determined to meet the target requirements, a prompt message will be output to indicate that the cooking is complete.

5. The method according to claim 4, characterized in that, Determining the first degree of doneness of the food to be processed inside the pot based on the plurality of first data includes: Determine the first feature information for each first data point; The weight coefficient corresponding to each first data point is determined based on the cooking scenario corresponding to the cooking task. The first fusion feature is determined based on multiple first feature information and the weight coefficient corresponding to each first data; The first degree of maturity of the food to be processed in the pot is determined based on the first fusion feature; Accordingly, determining the first feature information for each first data point includes: Process multiple sets of the first data; For each processed first data, feature extraction is performed to obtain multiple first feature information.

6. The method according to claim 5, characterized in that, The determination of the first degree of doneness of the food to be processed in the pot based on the first fusion feature includes: Obtain multiple initial feature information of the food to be processed corresponding to the multiple sensors; Initial fusion features are determined based on the aforementioned multiple initial feature information; A second fusion feature is determined based on the first fusion feature and the initial fusion feature; The first maturity level is determined based on the second fusion feature; Accordingly, if the first maturity level meets the first target maturity threshold, the step of performing a feeding operation on the ingredients to be processed in the pot includes: If all N first maturity values ​​determined based on the N second fusion features within the target time period meet the first target maturity threshold, a feeding operation is performed on the ingredients to be processed in the pot.

7. The method according to claim 4 or 6, characterized in that, The feeding operation for the ingredients to be processed in the pot includes: If the first maturity meets the first target maturity threshold, multiple second data related to the food to be processed are collected inside the pot through the multiple sensors; Determine the second feature information for each second data point; The third fusion feature is determined based on multiple second feature information and the weight coefficient corresponding to each second data; The second degree of maturity of the food to be processed in the pot is determined based on the third fusion feature. If the second maturity level meets the second target maturity threshold, a feeding operation is performed on the ingredients to be processed inside the pot.

8. The method according to claim 7, characterized in that, The determination of the second degree of cooking of the food to be processed in the pot based on the third fusion feature includes: The fourth fusion feature is determined based on the third fusion feature and the initial fusion feature; The second degree of maturity of the food to be processed in the pot is determined based on the fourth fusion feature. Accordingly, if the second maturity level meets the second target maturity threshold, the feeding operation is performed on the ingredients to be processed in the pot, including: If all N second maturity values ​​determined based on the N fourth fusion features within the target time period meet the second target maturity threshold, a feeding operation is performed on the ingredients to be processed in the pot.

9. The method according to claim 1, characterized in that, The method further includes: During the process of adding the ingredients to be processed into the pot based on the target cooking strategy, sensor data collected by multiple sensors corresponding to each cooking stage and / or the currently added ingredients are acquired at different cooking stages. Cooking operations corresponding to the different cooking stages are performed based on multiple sensor data.

10. A control device, characterized in that, The device includes: The first determining unit is used to determine the attribute data of the ingredients to be processed corresponding to the cooking task after obtaining the cooking task; wherein, the cooking task includes the identification of the dish and cooking requirement information; An acquisition unit is configured to acquire cooking data related to the cooking task from a server based on the identifier of the dish; wherein the cooking data is used to cook the ingredients; The second determining unit is used to determine the target cooking strategy corresponding to the cooking task based on the attribute data of the ingredients to be processed, the cooking requirement information, and the cooking data. The processing unit is configured to perform cooking operations based on the target cooking strategy and multimodal first data related to the ingredients to be processed, so as to complete the cooking of the ingredients to be processed.

11. A cooking machine, characterized in that, The cooking machine includes: a processor, a memory, and a communication bus; The communication bus is used to realize the communication connection between the processor and the memory; The processor is used to execute the control program of the multimodal cooking machine in the memory to implement the steps of the control method of the multimodal cooking machine as described in any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the steps of the control method for the multimodal cooking machine as described in any one of claims 1-9.

Images