Multi-mode sensing intelligent cooking system and control method thereof

Abstract

The invention relates to the technical field of intelligent kitchen appliances and artificial intelligence, in particular to a multi-modal sensing intelligent cooking system and a control method thereof, and the method comprises the steps: obtaining multi-modal sensing data of a to-be-cooked food material, inputting the multi-modal sensing data into a multi-modal time sequence fusion network, extracting spatial-temporal characteristics through a visual flow branch, and obtaining a multi-modal sensing data fusion network; time sequence dynamic features are extracted through a non-visual flow branch, adaptive weighted fusion is carried out through an attention fusion layer, and a cooking state quantitative index is obtained; inputting the cooking state quantitative index into an adaptive decision control engine, and generating a hierarchical control instruction by comparing the deviation between the current cooking state and the target state sequence; controlling a precise execution module to execute cooking operation according to the bottom execution parameters; according to the intelligent cooking equipment, self-adaptive adjustment in the cooking process and accurate reproduction of the degree of heating are achieved, and the automation level and the product production stability of the intelligent cooking equipment are remarkably improved.

Description

Technical Field

[0001] This invention relates to the fields of smart kitchen appliances and artificial intelligence technology, and in particular to a multimodal sensing smart cooking system and its control method. Background Technology

[0002] Cooking is a fundamental yet complex human activity, heavily reliant on the chef's senses (sight, smell, touch) and experiential judgment. With the development of automation and artificial intelligence technologies, automating the cooking process to improve efficiency, ensure consistent quality, reduce labor costs, and popularize standardized cuisine has become an important area of ​​technological research. Intelligent cooking equipment is evolving from simple timed heating towards "robot chefs" capable of sensing, decision-making, and execution.

[0003] This evolutionary path mainly involves technological development at two levels: the perception level and the control level.

[0004] At the perception level, early automated cooking equipment primarily relied on single temperature or time sensors, using preset temperature-time curves for open-loop control. This approach couldn't handle disturbances such as differences in ingredients and variations in water volume, resulting in coarse control. With the popularization of computer vision, some research began incorporating cameras, utilizing the color and texture features of food surfaces to determine doneness. However, this approach relies solely on visual information, lacking quantitative perception of key physicochemical processes such as temperature gradients and weight changes, resulting in limited information dimensions and susceptibility to misjudgments in complex cooking methods (such as stewing and stir-frying). The industry has recognized the necessity of multi-sensor fusion, using cameras, thermometers, and gas sensors to monitor food status. However, current technologies only briefly mention the coexistence of multiple sensors, failing to address the core challenge of precise temporal synchronization and effective feature fusion of multi-source heterogeneous data. Data processing is often parallel and isolated, failing to form a unified temporal feature representation capable of characterizing the dynamic evolution of the cooking process.

[0005] At the control level, most high-end smart kitchen appliances on the market currently adopt a preset program control mode, based on limited experimental data, to pre-set a fixed control curve for each dish. Its fundamental flaw lies in its "open-loop" nature: once the initial conditions (amount of ingredients, initial temperature, thermal conductivity of the cookware) deviate from the preset, or an unexpected event occurs during the process (localized overheating), the system cannot detect and adjust, leading to cooking failure. Some studies have introduced threshold-based feedback control, which automatically reduces the heat or adds broth when the temperature inside the pot exceeds the set value. This type of solution is a single-point, single-variable, reactive control, with simple and crude logic. It can only compensate after a certain extreme situation occurs and cannot perform forward-looking, multi-variable collaborative, and overall state-based optimization control.

[0006] The most similar existing technical solution uses a camera to capture real-time images of the pot, and a pre-trained convolutional neural network to classify the images, identifying discrete stages such as adding ingredients, stir-frying, reducing sauce, and removing from the pot. Based on the identified stages, it calls corresponding control parameters from a pre-set instruction library to operate the equipment. However, this technical solution has the following drawbacks: Its perception dimension is singular and shallow, relying solely on RGB visual information, making it sensitive to changes in lighting and unable to perceive key physicochemical changes such as temperature distribution and moisture evaporation rate; its state recognition is discrete and coarse, outputting only a few discrete stage labels, losing continuous, quantified state information of the cooking process, resulting in abrupt control and an inability to achieve smooth, fine transitions; its decision-making logic is rigid, essentially a lookup table method, lacking the ability to dynamically adjust based on real-time quantified states and to optimize multi-actuator collaboration; and it lacks temporal context modeling, as its convolutional neural network model processes single-frame images and does not fully consider the temporal dynamics of cooking state changes. Summary of the Invention

[0007] To address the aforementioned technical problems, this invention proposes a multimodal sensing intelligent cooking system and its control method. By constructing a closed-loop control framework encompassing perception, decision-making, and execution, it achieves precise, robust, and human-like automated operation of complex cooking processes such as Chinese stir-frying.

[0008] On one hand, embodiments of the present invention provide a control method for a multimodal sensing intelligent cooking system, the method comprising the following steps:

[0009] Acquire multimodal sensing data of the food to be cooked, including color visual data, infrared visual data, three-dimensional visual data, odor sensor data, weight sensor data, temperature sensor data, and pot body attitude data collected by an inertial measurement unit integrated into the pot handle;

[0010] The multimodal sensing data is input into a multimodal temporal fusion network. The color visual data and infrared visual data are subjected to three-dimensional convolution operation through the visual flow branch to extract spatiotemporal features. The temperature sensor data, weight sensor data, odor sensor data and pot body posture data are processed by a long short-term memory network through the non-visual flow branch to extract temporal dynamic features. The spatiotemporal features and temporal dynamic features are adaptively weighted and fused through the attention fusion layer to obtain the cooking state quantitative index.

[0011] The quantitative indicators of the cooking state are input into the adaptive decision control engine. By comparing the deviation between the current cooking state and the preset target state sequence in the digital recipe, hierarchical control instructions are generated. The hierarchical control instructions include high-level action intentions and low-level execution parameters. The target state sequence includes target temperature curve, target ingredient timing and target stir-frying frequency.

[0012] The cooking operation is controlled by the precision execution module based on the underlying execution parameters. The precision execution module includes an intelligent stir-frying mechanism, a precision temperature control system, and an adaptive feeding system. The intelligent stir-frying mechanism performs stir-frying actions based on the underlying execution parameters. The precision temperature control system independently adjusts the power of different areas of the pot bottom based on the power distribution parameters of the multi-zone electromagnetic heating coil. The adaptive feeding system quantitatively feeds solid and liquid materials based on the feeder control parameters.

[0013] Optionally, acquiring the multimodal sensing data of the ingredients to be cooked includes:

[0014] Color image data of the food is acquired by a color vision unit that is vertically mounted on the top of the pot body. The color vision unit is a global shutter industrial color camera.

[0015] Two-dimensional temperature field distribution data inside the pot are collected by an infrared vision unit that is tilted and installed above the side of the pot body. The infrared vision unit is an uncooled microthermometer.

[0016] The three-dimensional point cloud data of the ingredients is acquired by a three-dimensional vision unit installed directly above the pot body. The three-dimensional vision unit adopts a structured light depth camera or a time-of-flight depth camera.

[0017] The concentration data of volatile flavor substances are collected by an odor sensor array integrated inside the pot lid. The odor sensor array is a metal oxide semiconductor gas sensor.

[0018] The total weight of the cookware is collected by a weight sensor integrated into the cookware support structure.

[0019] The attitude data of the pot body is collected by an inertial measurement unit integrated into the handle of the pot. The inertial measurement unit includes a three-axis accelerometer and a three-axis gyroscope.

[0020] The infrared thermal imaging array collects temperature field distribution data of the bottom of the pot, and the center temperature sensor embedded in the geometric center of the bottom of the pot and the edge temperature sensor embedded in the edge area of ​​the bottom of the pot collect temperature data of key points of the pot. The infrared thermal imaging array, the center temperature sensor and the edge temperature sensor work together to form a redundant temperature measurement system.

[0021] Optionally, the step of adaptively weighting and fusing spatiotemporal features and temporal dynamic features through an attention fusion layer to obtain a quantitative index of cooking state includes:

[0022] Visual spatiotemporal feature maps and non-visual temporal dynamic features are concatenated along the feature dimension to obtain a multimodal joint feature vector;

[0023] The contribution weights of each modality data in the current cooking stage are calculated using an attention mechanism, and the contribution weights are dynamically adjusted according to the cooking stage.

[0024] The multimodal joint feature vector is weighted and fused according to the contribution weights to obtain the fused feature representation;

[0025] The fusion features are input into the fully connected layer, and the ripeness index, coking risk value, soup viscosity index and mixing uniformity score are output through regression calculation. The value range of each index is normalized to 0 to 1.

[0026] Optionally, the precise temperature control system independently adjusts the power of different areas of the pot bottom according to the power distribution parameters of the multi-zone electromagnetic heating coils, including:

[0027] Multiple independent excitation coils arranged in a honeycomb pattern are placed directly below the bottom of the pot. Each excitation coil is controlled by an independent insulated gate bipolar transistor drive circuit.

[0028] Based on the dynamic thermal field simulation mode, the thermal field migration effect is simulated by rapidly switching the power timing of adjacent coils, thereby achieving millisecond-level precise intervention on the local thermal effect on the bottom of the pot.

[0029] Optionally, the intelligent stir-frying mechanism performs stir-frying actions according to underlying execution parameters, including:

[0030] A curved shovel head is assembled on the end flange of a six-axis industrial robotic arm. The curved shovel head is made of food-grade stainless steel and its arc surface fits the curvature of the pot bottom.

[0031] A six-dimensional torque sensor is integrated at the connection between the shovel handle and the flange to sense the contact force between the shovel head and the food and the bottom of the pot in real time.

[0032] Based on the contact force feedback, the force-position hybrid control is executed. When abnormal resistance is detected, the shovel angle and force are adaptively adjusted to identify the sticking state and perform targeted shoveling and throwing actions.

[0033] Optionally, after controlling the precision execution module to perform the cooking operation according to the underlying execution parameters, the method further includes:

[0034] Multimodal perception data is collected in real time during the execution process, and the multimodal perception data is fed back to the multimodal time series fusion network to update the quantitative indicators of cooking status, forming a closed-loop control of perception-decision-execution.

[0035] On the other hand, embodiments of the present invention provide a multimodal sensing intelligent cooking system, comprising:

[0036] The multimodal sensing module is used to acquire multimodal sensing data of the food to be cooked. The multimodal sensing data includes color visual data, infrared visual data, three-dimensional visual data, odor sensor data, weight sensor data, temperature sensor data, and pot body attitude data collected by the inertial measurement unit integrated into the pot handle.

[0037] The central processing and control module is used to input the multimodal sensing data into a multimodal temporal fusion network. It performs 3D convolution operations on color visual data and infrared visual data through a visual flow branch to extract spatiotemporal features. It also performs long short-term memory network processing on temperature sensor data, weight sensor data, odor sensor data, and pot posture data through a non-visual flow branch to extract temporal dynamic features. Finally, it performs adaptive weighted fusion of spatiotemporal features and temporal dynamic features through an attention fusion layer to obtain a quantitative index of the cooking state. The module also inputs the quantitative index of the cooking state into an adaptive decision control engine. By comparing the deviation between the current cooking state and the preset target state sequence in the digital recipe, it generates hierarchical control instructions. These hierarchical control instructions include high-level action intentions and low-level execution parameters. The target state sequence includes a target temperature curve, a target ingredient feeding sequence, and a target stirring frequency.

[0038] The precision execution module is used to control the precision execution module to perform cooking operations according to the underlying execution parameters. The precision execution module includes an intelligent stir-frying mechanism, a precision temperature control system, and an adaptive feeding system. The intelligent stir-frying mechanism performs stir-frying actions according to the underlying execution parameters. The precision temperature control system independently adjusts the power of different areas of the pot bottom according to the power distribution parameters of the multi-zone electromagnetic heating coil. The adaptive feeding system performs quantitative feeding of solid and liquid materials according to the feeder control parameters.

[0039] Optionally, the multimodal sensing module includes:

[0040] The color vision unit is vertically mounted on the top of the pot body and uses a global shutter industrial color camera.

[0041] The infrared vision unit is installed at an angle on the upper side of the pot body and uses an uncooled microthermometer.

[0042] The three-dimensional vision unit is installed directly above the pot body and uses a structured light depth camera or a time-of-flight depth camera.

[0043] An odor sensor array, integrated inside the pot lid, employs a metal oxide semiconductor gas sensor;

[0044] Weight sensor integrated into the cookware support structure;

[0045] An inertial measurement unit, integrated into the handle, includes a three-axis accelerometer and a three-axis gyroscope;

[0046] The temperature sensing unit includes an infrared thermal imaging array, a center temperature sensor embedded in the geometric center of the bottom of the pot, and an edge temperature sensor embedded in the edge region of the bottom of the pot. The infrared thermal imaging array is used to collect temperature field distribution data of the bottom of the pot, and the center temperature sensor and the edge temperature sensor are used to collect temperature data of key points of the pot. The infrared thermal imaging array, the center temperature sensor, and the edge temperature sensor work together to form a redundant temperature measurement system.

[0047] On the other hand, embodiments of the present invention provide a multimodal sensing intelligent cooking system, comprising:

[0048] At least one processor;

[0049] At least one memory for storing at least one program;

[0050] When the at least one program is executed by the at least one processor, the at least one processor performs the method described above.

[0051] On the other hand, embodiments of the present invention provide a computer-readable storage medium storing a processor-executable program, which, when executed by a processor, is used to perform the above-described method.

[0052] The embodiments of the present invention have the following beneficial effects:

[0053] This invention constructs a multimodal perception module, integrating multiple sensors such as color vision, infrared thermal imaging, 3D vision, odor sensing, and weight sensing. This enables comprehensive perception of the visual appearance, temperature field distribution, 3D motion, volatile flavor compounds, and weight changes during the cooking process, overcoming the limitations of existing technologies that rely on a single perception dimension. By designing a multimodal temporal fusion network and employing a dual-stream architecture to process visual and non-visual temporal data separately, and introducing an attention mechanism to dynamically and adaptively allocate modal contributions, the invention achieves deep fusion of multi-source heterogeneous data at the feature level. This allows the system to integrate all sensory information, like an experienced chef, to identify key cooking status indicators such as the doneness of ingredients, risk of charring, broth condition, and mixing uniformity in real time and quantitatively.

[0054] This invention constructs an adaptive decision control engine, transforming traditional recipes into structured sequences of target states. Employing a hierarchical hybrid control strategy, it dynamically generates high-level action intentions by real-time comparison of the deviation between the current identified state and the target state, further parsing them into low-level execution parameters. This achieves closed-loop intelligent control from perception and understanding to decision-making, overcoming the shortcomings of existing technologies such as rigid decision-making logic and lack of adaptability. By integrating an intelligent stir-frying mechanism, a multi-zone electromagnetic heating system, and an adaptive feeding system, it achieves refined and spatially differentiated coordinated control of stir-frying actions, heating power, and feeding timing. This enables precise execution of complex instructions, such as targeted stir-frying and cooling of locally overheated areas, effectively simulating the operational skills of a master chef.

[0055] This invention achieves adaptive adjustment of the cooking process and precise reproduction of heat by constructing a closed-loop control framework that integrates perception, decision-making, and execution. It significantly improves the automation level and output stability of intelligent cooking equipment, and is particularly suitable for Chinese stir-fries that require strict heat control and have complex processes. It can achieve precise control of "wok hei" (wok aroma) and flavor. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 This is a flowchart illustrating the steps of a control method for a multimodal sensing intelligent cooking system provided in an embodiment of the present invention.

[0058] Figure 2 This is a structural block diagram of a multimodal sensing intelligent cooking system provided in an embodiment of the present invention;

[0059] Figure 3 This is a schematic diagram of the layout of the multimodal sensing module provided in an embodiment of the present invention;

[0060] Figure 4 This is a software architecture diagram of the central processing and control module provided in an embodiment of the present invention;

[0061] Figure 5 This is a schematic diagram of the working principle of the adaptive decision control engine provided in this embodiment of the invention;

[0062] Figure 6 This is a schematic diagram of the mechanical structure of the precision execution module provided in an embodiment of the present invention. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0064] It should be noted that although the device diagram is divided into modules and the flowchart shows the logical order, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart.

[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0066] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a full understanding of embodiments of the invention. However, those skilled in the art will recognize that the technical solutions of the invention can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of the invention.

[0067] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0068] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily need to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0069] refer to Figure 1 and Figure 2 , Figure 1 A control method for a multimodal sensing intelligent cooking system provided in this embodiment of the invention includes the following steps:

[0070] S100, acquire multimodal perception data of the food to be cooked, including color vision data, infrared vision data, three-dimensional vision data, odor sensor data, weight sensor data, temperature sensor data, and pot body attitude data collected by the inertial measurement unit integrated in the pot handle.

[0071] S200, the multimodal perception data is input into the multimodal temporal fusion network. The color visual data and infrared visual data are subjected to three-dimensional convolution operation through the visual flow branch to extract spatiotemporal features. The temperature sensor data, weight sensor data, odor sensor data and pot body posture data are processed by long short-term memory network through the non-visual flow branch to extract temporal dynamic features. The spatiotemporal features and temporal dynamic features are adaptively weighted and fused through the attention fusion layer to obtain the cooking state quantitative index.

[0072] S300, the cooking state quantification index is input into the adaptive decision control engine. By comparing the deviation between the current cooking state and the preset target state sequence in the digital recipe, a hierarchical control instruction is generated. The hierarchical control instruction includes high-level action intention and low-level execution parameters. The target state sequence includes target temperature curve, target ingredient timing and target stir-frying frequency.

[0073] S400, according to the underlying execution parameters, the precise execution module is controlled to perform cooking operations. The precise execution module includes an intelligent stir-frying mechanism, a precise temperature control system, and an adaptive feeding system. The intelligent stir-frying mechanism performs stir-frying actions according to the underlying execution parameters. The precise temperature control system independently adjusts the power of different areas of the pot bottom according to the power distribution parameters of the multi-zone electromagnetic heating coil. The adaptive feeding system performs quantitative feeding of solid and liquid materials according to the feeder control parameters.

[0074] This invention proposes a multimodal sensing intelligent cooking system and its control method. The system achieves deep perception of the cooking process across all dimensions and modes through a multimodal sensing module. It establishes a deep learning-based multimodal state recognition model and an adaptive decision control engine through a central processing and control module. The system achieves precise, collaborative, and biomimetic multi-actuator linkage control through a precision execution module. Finally, it constructs a closed-loop control framework for the entire chain of perception, decision-making, and execution, enabling adaptive adjustment of the cooking process and precise reproduction of the cooking temperature.

[0075] In this embodiment, system initialization is performed first. The user selects a target recipe, and the system loads the corresponding target state sequence from the digital recipe knowledge base. The intelligent stir-frying mechanism resets to the standby position, and the multi-zone electromagnetic heating system enters standby mode.

[0076] In some embodiments, acquiring multimodal sensing data of the ingredients to be cooked includes:

[0077] S110 acquires color image data of the ingredients through a color vision unit vertically mounted above the pot. The color vision unit uses a global shutter industrial color camera with a resolution of 4K ultra-high definition (3840×2160 pixels). All pixels of the sensor are exposed simultaneously, effectively avoiding image distortion caused by rolling shutters when shooting fast stir-frying. It can clearly distinguish subtle color changes of ingredients, such as the Maillard reaction from bright red to brown, as well as microscopic details such as the shrinkage of scallions, the texture of meat, and the shape of bubbles. The lens is rigidly fixed about 30 cm above the pot for a vertical downward view. It is equipped with a high-temperature resistant and oil-proof quartz glass protective cover and a micro heating ring on the edge to prevent steam condensation and ensure image clarity.

[0078] S120 collects two-dimensional temperature field distribution data inside the pot through an infrared vision unit that is tilted and installed on the upper side of the pot body. The infrared vision unit adopts an uncooled microthermometer with a working wavelength of 8-14μm. It can penetrate water vapor to obtain a two-dimensional temperature field of no less than 640×512 pixels with a temperature measurement accuracy of ±2℃. It is recommended to install it at a 45° tilt to the side to avoid interference with the line of sight of the main camera. It can locate local hot or cold areas in real time through pseudo-color images and estimate the internal cooking degree of large pieces of food based on surface temperature gradient and machine learning model.

[0079] S130: The three-dimensional point cloud data of the ingredients is collected by a three-dimensional vision unit installed directly above the pot. The three-dimensional vision unit adopts a structured light depth camera or a time-of-flight depth camera. By analyzing the deformation of the structured light pattern or measuring the time of flight of the light pulse, high-precision three-dimensional point cloud data of the scene inside the pot is obtained. Based on the point cloud, the volume and stacking height of the ingredients are calculated in real time to help verify whether the amount of ingredients meets the recipe standard. At the same time, through continuous frame point cloud registration, the height of the ingredients being thrown, the spatial displacement trajectory and the uniformity of the three-dimensional distribution are quantitatively analyzed during the stir-frying process.

[0080] S140: The concentration data of volatile flavor substances is collected by an odor sensor array integrated inside the pot lid. The odor sensor array adopts a metal oxide semiconductor gas sensor array, which integrates 4-8 sensing units that are sensitive to characteristic volatile compounds such as ethanol, aldehydes, hydrogen sulfide, and ammonia. After miniaturization, it is embedded inside the pot lid or the inlet of the exhaust channel to directly capture cooking steam. The array response fingerprint is analyzed by pattern recognition algorithm to qualitatively or semi-quantitatively sense the changes in the concentration of flavor substances such as aldehydes and ketones.

[0081] S150 collects the total weight data of the cookware through a weight sensor integrated into the cookware support structure. The weight sensor is a high-precision strain gauge weighing sensor with a range covering the weight from an empty pot to a full pot. The overall accuracy is better than 0.1%. It is integrated into the load-bearing shaft of the cookware support structure or heating platform. By continuously monitoring the decrease curve of the total weight and its first derivative, the moisture evaporation rate is calculated in real time. The weight step signal after the feeding command is detected, the feeding completion time is accurately determined and the feeding weight is recorded.

[0082] S160 collects cookware temperature data through a center temperature sensor embedded in the geometric center of the bottom of the pot and an edge temperature sensor embedded in the edge area of ​​the bottom of the pot. The temperature sensor adopts a PT100 platinum resistance temperature sensor, which utilizes its electrical characteristics of high accuracy, good stability and good linearity in a wide temperature range. The probe is tightly coupled to the pot wall with high-temperature thermally conductive silicone grease. The temperature difference between the center and the edge directly reflects the uniformity of heating of the cookware, which is the core basis for guiding the multi-zone electromagnetic coil to realize the dynamic heating strategy.

[0083] In addition, the inertial measurement unit (IMU) integrated into the pot handle collects real-time motion attitude data of the pot body in three-dimensional space, including triaxial acceleration and triaxial angular velocity. This data is used to quantitatively analyze dynamic characteristics such as the frequency of tossing, the amplitude of stirring, and the tilt angle of the pot body. In conjunction with sensory data such as vision, temperature, and weight, it provides key motion information for assessing the uniformity of stirring and detecting abnormal sticking.

[0084] refer to Figure 3 The multimodal sensing module layout diagram shows: the pot body is positioned above a multi-zone electromagnetic heating device, with a weight sensor integrated into the bottom support structure; center and edge temperature sensors are embedded in the bottom of the pot; a color vision unit is vertically mounted on the top of the pot body to capture microscopic details such as the color, shape, and bubbles in the broth; an infrared vision unit is mounted at a 45° angle on the upper side to collect the two-dimensional temperature field distribution inside the pot, enabling quantification of cooking temperature; a three-dimensional vision unit can be optionally installed on the top to acquire three-dimensional motion data such as food volume, tossing height, and evenness of stirring; an odor sensor array is integrated inside the lid to detect volatile flavor compounds during cooking. An inertial measurement unit (IMU) is integrated at the handle to collect pot posture data, including tossing frequency, tilt angle, and acceleration. All sensor data is converged in real time to the central processing unit via wired or wireless means to complete the fusion perception and real-time calculation of multimodal information such as color, infrared, three-dimensional, temperature, weight, odor, and posture.

[0085] In some embodiments, the pot body attitude data is used for:

[0086] The frequency and amplitude of tossing are analyzed using accelerometer signals and compared with the target stir-frying frequency.

[0087] The gyroscope signal is used to analyze the changes in the pot's attitude angle, and the uniformity of food distribution is assessed by combining it with three-dimensional visual data.

[0088] By detecting abnormal high-frequency vibrations, the sticking condition can be identified, and corresponding scraping action or water injection intervention can be triggered.

[0089] In some embodiments, inputting the multimodal sensing data into a multimodal temporal fusion network includes:

[0090] S210, input the color image sequence and thermal imaging sequence within a continuous time window into the visual flow branch. The duration of the time window is 5 seconds and the sampling rate is 10 frames per second. The spatiotemporal features of the input color image sequence and thermal imaging sequence are extracted by a three-dimensional convolutional neural network to obtain a visual spatiotemporal feature map that fuses appearance features and motion features.

[0091] S220: Acquire cookware center temperature data, cookware edge temperature data, weight change rate data, and odor feature vector data that are strictly time-synchronized with visual data; input the cookware center temperature data, cookware edge temperature data, weight change rate data, and odor feature vector data into a long short-term memory network; capture long-term dependencies in the time-series data through a gating mechanism; and output non-visual time-series dynamic features.

[0092] S230, the visual spatiotemporal feature map and the non-visual temporal dynamic features are concatenated in the feature dimension to obtain a multimodal joint feature vector. The contribution weight of each modality data in the current cooking stage is calculated through an attention mechanism. The contribution weight is dynamically adjusted according to the cooking stage. The multimodal joint feature vector is weighted and fused according to the contribution weight to obtain a fused feature representation.

[0093] S240, the fusion features are input into the fully connected layer, and the ripeness index, coking risk value, soup viscosity index and mixing uniformity score are output through regression calculation. The value range of each index is normalized to 0 to 1.

[0094] refer to Figure 4The central processing and control module software architecture is divided into five layers: The first layer is the data acquisition and synchronization layer, which receives real-time raw data from six types of multimodal sources, including color images, thermal imaging, temperature, weight, odor, and posture. This data is then aligned by a millisecond-level unified time synchronization and timestamp synchronization module. The second layer is the preprocessing and feature extraction layer. After preprocessing such as denoising, correction, and enhancement, image data is used to extract color histograms, local binary pattern textures, and convolutional neural network semantic features. Thermal imaging extracts average temperature, temperature gradient, and the proportion of high-temperature areas. Non-visual sensor data is processed by Kalman filtering to extract pot temperature, weight change rate, etc. The first layer consists of the pot's motion features and odor fingerprint feature vectors. The second layer is a multimodal fusion and state recognition layer. Visual information is extracted through a 3D convolutional neural network to extract spatiotemporal features, while non-visual temporal information is extracted through a long short-term memory network to extract dynamic change features. The two are weighted and fused in the attention fusion layer, and the fused features output the cooking stage classification and continuous state quantification values. The third layer is an adaptive decision control layer, which compares the current recognition state with the preset stage target state in the digital recipe. The decision engine generates high-level action intentions based on the deviation. The fourth layer is a low-level control instruction generation layer, which transforms the high-level intentions into specific execution instructions.

[0095] In some embodiments, inputting the cooking state quantification index into the adaptive decision control engine includes:

[0096] S310, load the target state sequence from the digital recipe knowledge base. The target state sequence decomposes the cooking process into a preheating stage, a stir-frying stage, a braising stage, a sauce reduction stage, and a serving stage. Each stage corresponds to a set of multimodal quantization thresholds. The multimodal quantization thresholds include a pot bottom temperature threshold, a charring risk threshold, an ingredient color threshold, a soup viscosity threshold, a target stir-frying frequency threshold, and a weight change rate threshold.

[0097] S320, calculate the deviation vector between the current cooking state quantification index and the multimodal quantification threshold of the current stage target state;

[0098] S330, based on the deviation vector, a high-level action intention is generated through expert system rules or model prediction control algorithms. The high-level action intention includes semantic instructions such as increasing local heating effect, reducing heating power, performing feeding, switching stir-frying mode, and triggering unloading.

[0099] refer to Figure 5The adaptive decision control engine works as follows: The engine has a built-in digital recipe knowledge base, transforming traditional text recipes into structured target state sequences. Each cooking stage corresponds to a set of multimodal target thresholds, forming a quantifiable and traceable stage-based control benchmark. The high-level strategy is responsible for generating decision intentions. By comparing the deviation between the current identified state and the target state of the current stage in real time, it uses expert system rules or predictive optimization algorithms to output semantic high-level action intentions. The low-level control is responsible for instruction execution and motion planning, translating the high-level intentions into specific equipment instructions: For the intelligent stir-frying mechanism, it generates the shovel trajectory and joint motion parameters through inverse kinematics solutions, achieving precise actions such as bottoming, tossing, and scraping; for the precision temperature control system, it resolves the power distribution and start-stop sequence of the electromagnetic coils, achieving regionalized, instantaneous response heat field adjustment.

[0100] In some embodiments, controlling the precision execution module to perform cooking operations based on the underlying execution parameters includes:

[0101] S410 controls the intelligent stir-frying mechanism to perform stir-frying actions according to the underlying execution parameters. The intelligent stir-frying mechanism adopts a six-axis articulated industrial robotic arm. The base is fixed to the side or rear of the cooking table. The working range completely covers the pot opening area. The end effector is a customized curved spatula head made of food-grade 304 stainless steel. The spatula shape fits the curvature of the pot bottom. The blunt edge design protects the pot coating. A six-dimensional torque sensor is integrated at the connection between the spatula handle and the flange to sense the three-dimensional torque in real time and realize force-position hybrid control. It can automatically identify the sticking state based on resistance feedback and adjust the spatula angle and force.

[0102] S420, according to the power distribution parameters of the multi-zone electromagnetic heating coils, controls the precise temperature control system to independently adjust the power of different areas of the pot bottom. The precise temperature control system adopts multi-zone electromagnetic heating technology. Multiple independently controlled flat spiral excitation coils are arranged in a honeycomb pattern directly below the pot bottom, completely covering the projected area of ​​the pot bottom. Each coil is driven by an independent insulated gate bipolar transistor with high-frequency chopper control technology for fine power adjustment. It supports three modes: global power synchronization, independent temperature control of zones, and dynamic thermal field simulation.

[0103] S430 controls the adaptive feeding system to quantitatively add solid and liquid materials according to the feeder control parameters. The adaptive feeding system adopts a gantry or rotating tower structure, spanning or surrounding the pot, and is equipped with several solid and liquid silos. The bottom of the solid silos is controlled by a screw feeder or piezoelectric vibrating feeder driven by a micro servo motor to control quantitative conveying. Each liquid silo is connected to a precision metering pump or high-speed solenoid valve to realize quantitative addition of liquid seasonings. It supports both open-loop quantitative and closed-loop feedback modes.

[0104] refer to Figure 6The schematic diagram of the precision execution module's mechanical structure is shown below. The intelligent stir-frying mechanism employs a six-axis articulated industrial robotic arm. The base is fixed to the side or rear of the cooking counter, and its working range completely covers the pot's rim area. The end effector is a custom-designed curved spatula head made of food-grade 304 stainless steel. The spatula shape conforms to the curvature of the pot's bottom, and the blunt edge design protects the pot's coating. A six-dimensional torque sensor is integrated at the connection between the spatula handle and the flange, sensing three-dimensional torque in real time to achieve force-position hybrid control. It can automatically identify the sticking state based on resistance feedback and adjust the spatula angle and force accordingly. The precision temperature control system uses multi-zone electromagnetic heating technology. Multiple independently controlled flat spiral excitation coils are arranged in a honeycomb pattern directly below the pot's bottom, completely covering the projected area of ​​the pot's bottom. Each coil is driven by an independent insulated-gate bipolar transistor (IGBT) circuit with high-frequency chopper control technology for fine power adjustment, supporting three modes: global power synchronization, independent zone temperature control, and dynamic thermal field simulation. The adaptive feeding system adopts a gantry or rotating tower structure, spanning or surrounding the pot, and is equipped with several solid and liquid silos. The bottom of the solid silos is controlled by a screw feeder or piezoelectric vibratory feeder driven by a micro servo motor to control quantitative feeding. Each liquid silo is connected to a precision metering pump or a high-speed solenoid valve to realize quantitative addition of liquid seasonings.

[0105] Taking the preparation of "braised pork" as an example, the complete workflow of the system is as follows:

[0106] Initialization phase: The user selects a recipe, and the system loads the corresponding target state sequence. The intelligent stir-fry mechanism resets to the standby position, and the electromagnetic heating plate enters standby mode.

[0107] Preheating stage: The decision engine issues a "global power 100%" command, and the infrared vision unit monitors the temperature of the bottom of the pot in real time. When the center temperature reaches the preset temperature threshold (pre-calibrated in the range of 150-250℃ according to the characteristics of the ingredients, preferably 200℃), the system automatically moves to the next stage.

[0108] Stir-frying stage: The decision engine issues a "slow stir-fry mode" command, and the robotic arm performs push-pull stir-frying actions. The color vision unit monitors the color changes on the surface of the ingredients. When the color feature parameters that represent the ripeness of the ingredients (such as the H component of the HSV color space) reach the target threshold, the system automatically triggers the addition of seasonings.

[0109] During the simmering stage: The decision engine switches to "low heat mode," controlling the electromagnetic heating plate to operate at reduced power, lowering the heating power to 20-40% of the rated power, preferably 30%. An array of odor sensors monitors changes in the concentration of flavor substances, and a weight sensor monitors the evaporation rate of the broth, providing control data for the subsequent reduction stage.

[0110] During the reduction stage: The system monitors the viscosity index of the broth in real time through a state recognition model and performs refined closed-loop control: When the viscosity index rises to the first preset threshold (calibrated in the range of 0.7-0.9, preferably 0.8), the decision engine automatically switches to "rapid stirring mode" and reduces the power of the electromagnetic heating plate to an extremely low level (such as 5-10% of the rated power), using a combination of simmering and stirring to reduce the broth; during this process, if the increase in viscosity is detected to be stagnant, the decision engine will execute a "pulse heating" command, briefly turning on low-power heating to break through the reduction bottleneck and prevent the broth from failing due to excessively low temperature; when the viscosity index rises to the second preset threshold (calibrated in the range of 0.9-0.98, preferably 0.95), the system turns off the heating and provides a voice prompt indicating that cooking is complete.

[0111] During the unloading stage: The robotic arm executes the preset "unloading and pouring" trajectory to pour the dish into the container specified by the user.

[0112] Stir-fry uniformity control based on pot posture perception

[0113] In this embodiment, an inertial measurement unit (IMU) integrated into the pot handle collects real-time triaxial acceleration and triaxial angular velocity data of the pot body during the cooking process. The sampling frequency is 100Hz, and the measurement ranges are ±16g and ±2000° / s, respectively. This data, along with color vision data, infrared vision data, weight sensor data, and temperature sensor data from the multimodal sensing module, is input to the central processing and control module. It participates in the processing of a long short-term memory network (LSTM) in the non-visual flow branch to extract temporal dynamic features related to the pot body's motion.

[0114] Taking the typical stir-frying process of "Kung Pao Chicken" as an example, the system executes the following control process during the stir-frying stage:

[0115] SS1. Monitoring of the frequency and amplitude of tossing the food:

[0116] The Z-axis acceleration signal from the IMU is bandpass filtered (0.5–10Hz), and the tossing frequency and amplitude of the pot are calculated in real time using a peak detection algorithm. When the tossing frequency is detected to be lower than the target frequency (e.g., the preset target is 1.2Hz, and the current measured value is 0.6Hz), the adaptive decision control engine determines that the stirring force is insufficient and then generates the underlying execution parameters to "increase the speed of the robotic arm's end effector," thereby increasing the speed of the shovel head of the intelligent stirring mechanism by 20% until the tossing frequency fed back by the IMU returns to the target range.

[0117] SS2. Evaluation of ingredient mixing uniformity:

[0118] By integrating the triaxial angular velocities collected by the IMU to obtain the sequence of pot attitude angle changes, and combining it with the food point cloud distribution data collected by the 3D vision unit, the system constructs a "quantitative model of food spatial distribution uniformity". If the system detects that the pot is consistently biased to one side during multiple consecutive stir-fries (e.g., the pitch angle is consistently greater than 15°), and the 3D vision shows that the food is concentrated on the front side of the pot, it is determined that there is a blind spot for stir-frying. The decision engine then generates a "correct shovel trajectory" command, causing the robotic arm to add a "lateral scraping" motion to the original stir-frying path to push the accumulated food towards the center of the pot.

[0119] SS3. Sticking Status Recognition and Adaptive Response: During the sauce reduction stage, if the IMU detects abnormal high-frequency micro-vibrations (characteristic frequency 50–150Hz, amplitude exceeding 3 times that of normal stir-frying) in the pan during the robotic arm's tossing motion, and simultaneously the resistance of the spatula head fed back by the six-dimensional torque sensor exceeds a threshold (e.g., Z-axis resistance > 15N), the system determines that localized sticking has occurred at the bottom of the pan. At this point, the decision engine stops performing regular stir-frying and switches to "sticking handling mode":

[0120] The precise temperature control system reduces the power of the electromagnetic coil corresponding to the sticky area to 10% of the rated power for 2 seconds, using the thermal shrinkage effect to separate the charred matter from the bottom of the pot.

[0121] The intelligent stir-frying mechanism performs a "shovel action". The shovel head moves along the curved surface of the bottom of the pot at a low angle (15° between the shovel head and the bottom of the pot) and a slow speed (20mm / s). The IMU monitors the changes in the posture of the pot in real time to ensure that the shovel head and the bottom of the pot maintain a constant contact pressure.

[0122] If the IMU still detects abnormal vibration after three bottom-scooping actions, the system automatically triggers the "water injection intervention" command, injecting 5-10 mL of clean water into the sticky pot area through the adaptive feeding system, and using steam impact to assist in the stripping of coke.

[0123] Through the aforementioned IMU-based pot posture perception and closed-loop control, the system can quantitatively evaluate the execution quality of the stir-frying process and actively adjust the control strategy when abnormalities occur. This achieves digital reproduction and adaptive optimization of the traditional cooking technique of "tossing the pot," significantly improving the uniformity of food mixing and the success rate of handling pot sticking abnormalities.

[0124] See Figure 2 This invention provides a multimodal sensing intelligent cooking system, comprising:

[0125] The multimodal sensing module is used to acquire multimodal sensing data of the food to be cooked. The multimodal sensing data includes color visual data, infrared visual data, three-dimensional visual data, odor sensor data, weight sensor data, temperature sensor data, and pot body attitude data collected by the inertial measurement unit integrated into the pot handle.

[0126] The central processing and control module is used to input the multimodal perception data into a multimodal temporal fusion network. It performs 3D convolution operations on color visual data and infrared visual data through a visual flow branch to extract spatiotemporal features. It also performs long short-term memory network processing on temperature sensor data, weight sensor data, odor sensor data, and pot posture data through a non-visual flow branch to extract temporal dynamic features. Finally, it performs adaptive weighted fusion of spatiotemporal features and temporal dynamic features through an attention fusion layer to obtain a quantitative index of the cooking state. The module further inputs the quantitative index of the cooking state into an adaptive decision control engine. By comparing the deviation between the current cooking state and the preset target state sequence in the digital recipe, it generates hierarchical control instructions, which include high-level action intentions and low-level execution parameters.

[0127] The precision execution module is used to control the precision execution module to perform cooking operations according to the underlying execution parameters. The precision execution module includes an intelligent stir-frying mechanism, a precision temperature control system, and an adaptive feeding system. The intelligent stir-frying mechanism performs stir-frying actions according to the underlying execution parameters. The precision temperature control system independently adjusts the power of different areas of the pot bottom according to the power distribution parameters of the multi-zone electromagnetic heating coil. The adaptive feeding system performs quantitative feeding of solid and liquid materials according to the feeder control parameters.

[0128] It is evident that the content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0129] This invention also provides a multimodal sensing intelligent cooking system, comprising:

[0130] At least one processor;

[0131] At least one memory for storing at least one program;

[0132] When the at least one program is executed by the at least one processor, the at least one processor performs the method described above.

[0133] The content of the above method embodiments is applicable to this embodiment. The specific functions implemented in this embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments. Therefore, they will not be repeated here.

[0134] This invention also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0135] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0136] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0137] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0138] This invention also provides a computer program product, including a computer program or computer instructions, which are stored in a memory. A processor of a computer device reads the computer program or computer instructions from the memory and executes the computer program or computer instructions, causing the computer device to perform the above-described method.

[0139] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0140] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0141] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically include computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0142] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

Claims

1. A control method for a multimodal sensing intelligent cooking system, characterized in that, The method includes the following steps: Acquire multimodal sensing data of the food to be cooked, including color visual data, infrared visual data, three-dimensional visual data, odor sensor data, weight sensor data, temperature sensor data, and pot body attitude data collected by an inertial measurement unit integrated into the pot handle; The multimodal sensing data is input into a multimodal temporal fusion network. The color visual data and infrared visual data are subjected to three-dimensional convolution operation through the visual flow branch to extract spatiotemporal features. The temperature sensor data, weight sensor data, odor sensor data and pot body posture data are processed by a long short-term memory network through the non-visual flow branch to extract temporal dynamic features. The spatiotemporal features and temporal dynamic features are adaptively weighted and fused through the attention fusion layer to obtain the cooking state quantitative index. The quantitative indicators of the cooking state are input into the adaptive decision control engine. By comparing the deviation between the current cooking state and the preset target state sequence in the digital recipe, hierarchical control instructions are generated. The hierarchical control instructions include high-level action intentions and low-level execution parameters. The target state sequence includes target temperature curve, target ingredient timing and target stir-frying frequency. The cooking operation is controlled by the precision execution module based on the underlying execution parameters. The precision execution module includes an intelligent stir-frying mechanism, a precision temperature control system, and an adaptive feeding system. The intelligent stir-frying mechanism performs stir-frying actions based on the underlying execution parameters. The precision temperature control system independently adjusts the power of different areas of the pot bottom based on the power distribution parameters of the multi-zone electromagnetic heating coil. The adaptive feeding system quantitatively feeds solid and liquid materials based on the feeder control parameters.

2. The method according to claim 1, characterized in that, The acquisition of multimodal perception data of the ingredients to be cooked includes: Color image data of the food is acquired by a color vision unit that is vertically mounted on the top of the pot body. The color vision unit is a global shutter industrial color camera. Two-dimensional temperature field distribution data inside the pot are collected by an infrared vision unit that is tilted and installed above the side of the pot body. The infrared vision unit is an uncooled microthermometer. The three-dimensional point cloud data of the ingredients is acquired by a three-dimensional vision unit installed directly above the pot body. The three-dimensional vision unit adopts a structured light depth camera or a time-of-flight depth camera. The concentration data of volatile flavor substances are collected by an odor sensor array integrated inside the pot lid. The odor sensor array is a metal oxide semiconductor gas sensor. The total weight of the cookware is collected by a weight sensor integrated into the cookware support structure. The attitude data of the pot body is collected by an inertial measurement unit integrated into the handle of the pot. The inertial measurement unit includes a three-axis accelerometer and a three-axis gyroscope. The infrared thermal imaging array collects temperature field distribution data of the bottom of the pot, and the center temperature sensor embedded in the geometric center of the bottom of the pot and the edge temperature sensor embedded in the edge area of ​​the bottom of the pot collect temperature data of key points of the pot. The infrared thermal imaging array, the center temperature sensor and the edge temperature sensor work together to form a redundant temperature measurement system.

3. The method according to claim 1, characterized in that, The process involves adaptively weighting and fusing spatiotemporal features and temporal dynamic features through an attention fusion layer to obtain quantitative indicators of the cooking state, including: Visual spatiotemporal feature maps and non-visual temporal dynamic features are concatenated along the feature dimension to obtain a multimodal joint feature vector; The contribution weights of each modality data in the current cooking stage are calculated using an attention mechanism, and the contribution weights are dynamically adjusted according to the cooking stage. The multimodal joint feature vector is weighted and fused according to the contribution weights to obtain the fused feature representation; The fusion features are input into the fully connected layer, and the ripeness index, coking risk value, soup viscosity index and mixing uniformity score are output through regression calculation. The value range of each index is normalized to 0 to 1.

4. The method according to claim 1, characterized in that, The precise temperature control system independently adjusts the power of different areas of the pot bottom according to the power distribution parameters of the multi-zone electromagnetic heating coils, including: Multiple independent excitation coils arranged in a honeycomb pattern are placed directly below the bottom of the pot. Each excitation coil is controlled by an independent insulated gate bipolar transistor drive circuit. Based on the dynamic thermal field simulation mode, the thermal field migration effect is simulated by rapidly switching the power timing of adjacent coils, thereby achieving millisecond-level precise intervention on the local thermal effect on the bottom of the pot.

5. The method according to claim 1, characterized in that, The intelligent stir-frying mechanism performs stir-frying actions based on underlying execution parameters, including: A curved shovel head is assembled on the end flange of a six-axis industrial robotic arm. The curved shovel head is made of food-grade stainless steel and its arc surface fits the curvature of the pot bottom. A six-dimensional torque sensor is integrated at the connection between the shovel handle and the flange to sense the contact force between the shovel head and the food and the bottom of the pot in real time. Based on the contact force feedback, the force-position hybrid control is executed. When abnormal resistance is detected, the shovel angle and force are adaptively adjusted to identify the sticking state and perform targeted shoveling and throwing actions.

6. The method according to claim 1, characterized in that, After controlling the precise execution module to perform the cooking operation based on the underlying execution parameters, the method further includes: Multimodal perception data is collected in real time during the execution process, and the multimodal perception data is fed back to the multimodal time series fusion network to update the quantitative indicators of cooking status, forming a closed-loop control of perception-decision-execution.

7. A multimodal sensing intelligent cooking system, characterized in that, include: The multimodal sensing module is used to acquire multimodal sensing data of the food to be cooked. The multimodal sensing data includes color visual data, infrared visual data, three-dimensional visual data, odor sensor data, weight sensor data, temperature sensor data, and pot body attitude data collected by the inertial measurement unit integrated into the pot handle. The central processing and control module is used to input the multimodal sensing data into the multimodal temporal fusion network. It performs three-dimensional convolution operations on color visual data and infrared visual data through the visual flow branch to extract spatiotemporal features. It performs long short-term memory network processing on temperature sensor data, weight sensor data, odor sensor data and pot body posture data through the non-visual flow branch to extract temporal dynamic features. It performs adaptive weighted fusion of spatiotemporal features and temporal dynamic features through the attention fusion layer to obtain quantitative indicators of cooking status. And for inputting the cooking state quantification index into the adaptive decision control engine, by comparing the deviation between the current cooking state and the preset target state sequence in the digital recipe, a hierarchical control instruction is generated. The hierarchical control instruction includes a high-level action intention and a low-level execution parameter. The target state sequence includes a target temperature curve, a target ingredient timing sequence, and a target stir-frying frequency. The precision execution module is used to control the precision execution module to perform cooking operations according to the underlying execution parameters. The precision execution module includes an intelligent stir-frying mechanism, a precision temperature control system, and an adaptive feeding system. The intelligent stir-frying mechanism performs stir-frying actions according to the underlying execution parameters. The precision temperature control system independently adjusts the power of different areas of the pot bottom according to the power distribution parameters of the multi-zone electromagnetic heating coil. The adaptive feeding system performs quantitative feeding of solid and liquid materials according to the feeder control parameters.

8. The system according to claim 7, characterized in that, The multimodal sensing module includes: The color vision unit is vertically mounted on the top of the pot body and uses a global shutter industrial color camera. The infrared vision unit is installed at an angle on the upper side of the pot body and uses an uncooled microthermometer. The three-dimensional vision unit is installed directly above the pot body and uses a structured light depth camera or a time-of-flight depth camera. An odor sensor array, integrated inside the pot lid, employs a metal oxide semiconductor gas sensor; Weight sensor integrated into the cookware support structure; An inertial measurement unit, integrated into the handle, includes a three-axis accelerometer and a three-axis gyroscope; The temperature sensing unit includes an infrared thermal imaging array, a center temperature sensor embedded in the geometric center of the bottom of the pot, and an edge temperature sensor embedded in the edge region of the bottom of the pot. The infrared thermal imaging array is used to collect temperature field distribution data of the bottom of the pot, and the center temperature sensor and the edge temperature sensor are used to collect temperature data of key points of the pot. The infrared thermal imaging array, the center temperature sensor, and the edge temperature sensor work together to form a redundant temperature measurement system.

9. A multimodal sensing intelligent cooking system, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor performs the method as described in any one of claims 1 to 6.

10. A computer-readable storage medium storing a processor-executable program, characterized in that, The processor-executable program, when executed by the processor, is used to perform the method as described in any one of claims 1 to 6.

Images