A remote synchronized cooking method and system

Abstract

The present application relates to a kind of remote synchronous cooking method and system, including: chef end multimodal interaction subsystem, network transmission and synchronization control subsystem, kitchen end intelligent cooking execution subsystem and cooking skill digital twin modeling subsystem.The beneficial effects of the present application are: professional chef cooking action can be remotely reproduced, professional cooking actions such as pan turning, fine feeding and fire adjustment can be completed, at the same time, through double-redundancy transmission channel, PTP precision clock synchronization and delay prediction feedforward compensation, the problem of action asynchronization and misexecution caused by network fluctuation and packet loss is solved.

Description

Technical Field

[0001] This invention relates to the field of intelligent cooking technology, and specifically to a remote synchronous cooking method and system. Background Technology

[0002] With the development of smart home appliance technology, smart cooking machines have gradually entered commercial and home scenarios. Most existing smart cooking machines are based on pre-set fixed programs to perform cooking operations, and can only complete the standardized cooking of simple dishes. They cannot replicate the personalized and refined cooking actions of professional chefs, let alone realize the remote real-time cooking operations of professional chefs.

[0003] In existing technologies, some remote cooking solutions use simple cameras to capture and replicate movements. The motion capture accuracy is low, and it can only capture rough arm movements. It cannot replicate professional cooking movements such as tossing the wok, adding ingredients precisely, and fine-tuning the heat. The motion synchronization delay is high, and the cooking effect is inconsistent. At the same time, there is no network latency compensation. When the network fluctuates or packets are lost, the movements may be lost or erroneously executed, which poses serious security risks. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this application proposes a remote synchronous cooking method and system that can remotely replicate the cooking actions of professional chefs. It can perform professional cooking actions such as tossing the wok, precise ingredient addition, and fine-tuning of the heat. At the same time, by using dual redundant transmission channels, PTP precision clock synchronization, and delay prediction feedforward compensation, the problems of asynchronous actions and erroneous execution caused by network fluctuations and packet loss are solved.

[0005] The following is the technical solution of the present invention: a remote synchronous cooking system, comprising: a chef-end multimodal interaction subsystem, a network transmission and synchronization control subsystem, a kitchen-end intelligent cooking execution subsystem, and a cooking technique digital twin modeling subsystem; The chef-side multimodal interaction subsystem includes a limb motion capture module for collecting chef's limb movement data, a force perception and operation intention perception module for collecting operation force data and electromyographic signals to identify operation intentions, and a VR immersive cooking feedback module for providing immersive feedback to chefs. The network transmission and synchronization control subsystem includes an action data precoding and compression module for compressing and correcting the acquired data, a bidirectional low-latency transmission channel module for establishing a dual-redundant low-latency transmission channel, a synchronization timing control module for synchronizing the clocks at both ends and predicting and compensating for network latency, and a safety redundancy and fault handling module for system safety monitoring and fault handling. The kitchen-side intelligent cooking execution subsystem includes a multi-degree-of-freedom cooking execution module for replicating the chef's cooking actions, an intelligent firepower and temperature control execution module for precisely controlling heating, a precision ingredient feeding execution module for accurately adding ingredients according to intent, and a cooking environment multi-modal acquisition module for collecting multi-modal data of the cooking environment. The cooking technique digital twin modeling subsystem extracts technique features from cooking process data, and constructs and optimizes an executable digital recipe twin model.

[0006] As a preferred embodiment of the present invention, the limb motion capture module includes several inertial motion capture nodes worn on the chef's upper limbs and waist, an optical correction camera for drift correction of the inertial motion capture data, and a motion data preprocessing unit for converting the data to a globally unified coordinate system.

[0007] As a preferred embodiment of the present invention, the force and operation intention perception module includes a six-dimensional force / torque sensor embedded in the simulated kitchen utensils, several surface electromyography sensors attached to the chef's forearm, and an operation intention recognition unit that identifies cooking operation intentions based on motion features, force features and electromyography features through a pre-trained classification model. The intention recognition unit is configured to predict the trajectory of movement in the future time period based on the advanced features of electromyographic signals.

[0008] As a preferred embodiment of the present invention, the synchronization timing control module includes: PTP precision clock synchronization unit based on precision time protocol to synchronize global clocks at both ends; A network latency prediction unit for predicting future one-way network transmission latency based on historical latency data; Feedforward compensation unit is used to perform feedforward compensation on execution instructions sent to the kitchen end based on predicted delay and action prediction data.

[0009] As a preferred embodiment of the present invention, the multi-degree-of-freedom cooking execution module includes: Used to decouple chef's limb movement data into motion decoupling units with multiple degrees of freedom; This is a trajectory planning and adaptation unit used to map decoupled actions to the end effector trajectory of a robotic arm based on the spatial ratio between the chef's end and the kitchen end, and to perform inertial compensation based on the total weight of the cookware and ingredients.

[0010] As a preferred embodiment of the present invention, the digital twin modeling subsystem for cooking techniques includes: A cooking process data acquisition and annotation module for collecting and labeling multi-source data of a single cooking process; A cuisine technique feature extraction module for extracting core technique features of different cuisines from labeled data; A digital recipe twin modeling module used to encapsulate and compile core technical features into executable control instructions; This module is used for iterative optimization of the technique model of a digital recipe model based on multiple evaluation data of cooked products.

[0011] A remote synchronous cooking method includes the following steps: S1. Collect the chef's limb movement data, operational force data and electromyographic signals at the chef's end to identify and predict the chef's intention to perform cooking operations. S2. The collected data and recognition results are encoded and compressed, and transmitted to the kitchen end through a low-latency network channel. During the transmission process, global clock synchronization, network latency prediction and feedforward compensation are performed. S3. Receive and parse the processed data at the kitchen end, decouple and map it into the motion trajectory of the robotic arm to perform cooking actions, control the fire and add ingredients, and collect multimodal data of the cooking environment. S4. The collected multimodal data of the cooking environment is encoded and transmitted back to the chef's end, and presented to the chef through VR devices in the form of panoramic vision, spatial audio and haptic feedback.

[0012] As a preferred embodiment of the present invention, it also includes S5: extracting technique features based on the cooking process data collected in S1 to S3 to construct a digital recipe twin model, and iteratively optimizing the model based on the evaluation of the cooked product.

[0013] In a preferred embodiment of the present invention, step S1, identifying and predicting the intention of the cooking operation, includes: Based on limb movement characteristics, operational force characteristics, and electromyographic characteristics, the intention to toss the wok, stir-fry, add ingredients, or adjust the heat is identified through a classification model. Based on the characteristic of electromyographic signals that precede limb movements, the trajectory of movements and operational intentions can be predicted within the next 200-500ms.

[0014] In a preferred embodiment of the present invention, S2 includes network delay prediction and feedforward compensation, comprising: A time series forecasting model is used to predict future one-way transmission delays based on historical network delay data; Based on the predicted delay and the motion prediction data obtained from electromyography signals, the compensated timestamp of the kitchen-side execution command is calculated.

[0015] The beneficial effects of this invention are: 1. In this invention, the cooking actions of professional chefs can be remotely replicated. Through wearable multimodal motion capture, electromyographic intention prediction and feedforward delay compensation, professional cooking actions such as wok tossing, precise ingredient addition and heat fine adjustment can be completed, solving the core pain point that existing technologies cannot realize remote real-time cooking by professional chefs. 2. In this invention, not only is the chef's limb movement trajectory replicated, but the chef's cooking operation intention is also identified through electromyography signals and force data. Combined with the parameters of pots and ingredients in the kitchen, the movement is adapted and the trajectory is optimized, which greatly improves the system's adaptability and execution success rate. 3. In this invention, the problems of asynchronous actions and erroneous execution caused by network fluctuations and packet loss are solved by using dual redundant transmission channels, PTP precision clock synchronization and delay prediction feedforward compensation, thus achieving high operational stability and security. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the overall architecture of the system of the present invention; Figure 2 This is a schematic diagram of the chef-side multimodal interaction subsystem of the system of the present invention; Figure 3 This is a schematic diagram of the network transmission and synchronization control subsystem of the system of the present invention; Figure 4 This is a schematic diagram of the kitchen-side intelligent cooking execution subsystem of the system of the present invention; Figure 5 This is a schematic diagram of the digital twin modeling subsystem for cooking techniques in the system of the present invention; Figure 6 This is a flowchart illustrating the steps of the method of the present invention; The diagram shows: 1. Chef-side multimodal interaction subsystem; 2. Network transmission and synchronization control subsystem; 3. Kitchen-side intelligent cooking execution subsystem; 4. Cooking technique digital twin modeling subsystem; 101. Body motion capture module; 102. Force and operation intention perception module; 103. VR immersive cooking feedback module; 104. Instruction and parameter configuration module; 201. Data precoding and compression module; 202. Two-way low-latency transmission channel module; 203. Synchronization timing control module; 204. Safety redundancy and fault handling module; 301. Multi-degree-of-freedom cooking execution module; 302. Intelligent firepower and temperature control execution module; 303. Precise ingredient feeding execution module; 304. Cooking environment multimodal acquisition module; 401. Cooking process data acquisition and annotation module; 402. Cuisine technique feature extraction module; 403. Digital recipe twin modeling module; 404. Technique model iterative optimization module. Detailed Implementation

[0017] To make the technical problems solved by the present invention, the technical solutions adopted, and the technical effects achieved clearer, the technical solutions of the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Example 1 like Figures 1 to 5 As shown, a remote synchronous cooking system includes: a chef-side multimodal interaction subsystem 1, a network transmission and synchronization control subsystem 2, a kitchen-side intelligent cooking execution subsystem 3, and a cooking technique digital twin modeling subsystem 4.

[0019] The bidirectional communication terminal of the chef-side multimodal interaction subsystem 1 is connected to the first bidirectional communication terminal of the network transmission and synchronization control subsystem 2; the second bidirectional communication terminal of the network transmission and synchronization control subsystem 2 is connected to the bidirectional communication terminal of the kitchen-side intelligent cooking execution subsystem 3; the cooking technique digital twin modeling subsystem 4 is connected to both the chef-side multimodal interaction subsystem 1 and the kitchen-side intelligent cooking execution subsystem 3.

[0020] In this embodiment, as Figure 2 As shown, the chef-side multimodal interaction subsystem 1 includes a body motion capture module 101, a force and operational intent perception module 102, a VR immersive cooking feedback module 103, and an instruction and parameter configuration module 104. The data output of the body motion capture module 101 is connected to the data input of the force and operational intent perception module 102, and the data output of the force and operational intent perception module 102 is connected to the input of the network transmission and synchronization control subsystem 2. The input of the VR immersive cooking feedback module 103 is connected to the feedback output of the network transmission and synchronization control subsystem 2. The output of the instruction and parameter configuration module 104 is connected to the security control terminal of the network transmission and synchronization control subsystem 2 and the input of the cooking technique digital twin modeling subsystem 4, respectively.

[0021] The limb motion capture module 101 is used to collect full-dimensional limb motion data related to the chef's cooking actions, including the spatial position, angle, angular velocity, and acceleration data of the upper arm, forearm, wrist, palm, knuckles, and waist. The limb motion capture module 101 is equipped with 12 wearable inertial motion capture nodes, 4 optically calibrated industrial cameras, a motion data preprocessing unit, and a spatial coordinate calibration unit. The 12 inertial motion capture nodes are fixed to the chef's bilateral upper arms, bilateral forearms, bilateral wrists, palms, thumbs, index fingers, middle fingers, and waist, respectively, and communicate with the motion data preprocessing unit via a CAN bus. The 4 optically calibrated industrial cameras are installed at the four corners of the chef's operating space and communicate with the motion data preprocessing unit via gigabit Ethernet. The output of the motion data preprocessing unit is connected to the input of the network transmission and synchronization control subsystem 2. The sampling rate of the inertial motion capture node is 200Hz; the resolution of the optical correction industrial camera is 2560×1440, the refresh rate is 120Hz, and it is used to correct the cumulative drift of the inertial motion capture in real time. The correction frequency is 30Hz; the motion data preprocessing unit performs noise reduction, Kalman filtering, zero bias calibration, and coordinate system transformation on the collected raw motion capture data, converting the motion capture data from the chef's local coordinate system to the global unified coordinate system, and outputting standardized limb movement timing data.

[0022] The force and operational intent perception module 102 is used to collect operational force data and electromyographic signals during the chef's cooking process, identify the chef's cooking operational intent, and perform action prediction and feedforward compensation. The force and operational intent perception module 102 includes two 6-dimensional force / torque sensors, a 4-channel surface electromyography (sEMG) sensor, an operational intent recognition unit, and an action prediction unit. The 6-dimensional force / torque sensors are embedded in the simulated pot handle and simulated spatula handle operated by the chef, with a sampling rate of 100Hz. The 4-channel sEMG sensor is attached to the flexor, extensor, biceps, and triceps muscles of the chef's forearm, with a sampling rate of 1000Hz. The output terminals of the force / torque sensors and the sEMG sensors are connected to the input terminal of the operational intent recognition unit, and the output terminal of the operational intent recognition unit is connected to the input terminal of the action prediction unit and the network transmission and synchronization control subsystem 2, respectively. The collected force / torque data is denoised and filtered to extract the changes in force, direction of force, and torque changes of the chef's grip on the wok and spatula. The sEMG signal is rectified, filtered, and normalized to extract the root mean square (RMS), integral electromyography (IEMG), and median frequency (MF) features of the electromyography signal. The feature changes of the electromyography signal lead the limb movement by 200-500ms, enabling advance prediction of the movement. Based on the pre-trained LightGBM classification model, the movement features, force sensory features, and electromyography features are fused to identify the chef's cooking operation intentions, including tossing the wok, stir-frying, adding ingredients, adjusting the heat, and resting. The movement prediction unit predicts the chef's movement trajectory and operation intention within the next 200ms based on the features of the electromyography signal and sends the prediction data to the kitchen in advance to compensate for network transmission delay.

[0023] The VR immersive cooking feedback module 103 is used to present the cooking environment, pot status, and ingredient status data collected in real time from the kitchen to the chef in an immersive manner. The VR immersive cooking feedback module 103 includes a binocular VR headset, a spatial audio unit, a multi-physical quantity tactile feedback unit, and a panoramic image stitching unit. The input end of the panoramic image stitching unit is connected to the feedback output end of the network transmission and synchronization control subsystem 2, and the output end of the panoramic image stitching unit is connected to the binocular VR headset. The input end of the spatial audio unit is connected to the audio output end of the network transmission and synchronization control subsystem 2, and the output end is connected to the headphones of the VR headset. The input end of the multi-physical quantity tactile feedback unit is connected to the physical quantity output end of the network transmission and synchronization control subsystem 2, and the output end is connected to the vibration motor of the simulated pot handle and the simulated spatula handle and the Peltier temperature control unit, respectively. The binocular VR headset has a 110° field of view; the panoramic image stitching unit receives images from four industrial cameras in the kitchen, performs real-time distortion correction, feature matching, and panoramic stitching, and outputs a 360° panoramic image, simultaneously overlaying real-time data on pot temperature, oil temperature, and food doneness; the spatial audio unit receives dual-channel audio from the kitchen and performs 3D spatial rendering to reproduce cooking sounds in the kitchen; the multi-physical tactile feedback unit maps the real-time temperature of the pot body to the grip area of ​​the simulated pot handle through the Peltier temperature control unit, and reproduces the vibration feedback during the tossing and stir-frying process through the vibration motor.

[0024] The instruction and parameter configuration module 104 is used by the chef to configure parameters before cooking and input instructions during the cooking process. The instruction and parameter configuration module 104 includes a touch screen, an offline voice recognition unit, a hardware emergency stop button, and a parameter storage unit. The outputs of the touch screen, voice recognition unit, and hardware emergency stop button are connected to the parameter storage unit and the safety control input of the network transmission and synchronization control subsystem 2. The output of the parameter storage unit is connected to the input of the cooking technique digital twin modeling subsystem 4. Before cooking, the chef configures the dish name, cuisine, ingredient type, ingredient weight, cookware specifications, and safety threshold parameters via the touch screen. During cooking, the chef inputs voice commands via the voice recognition unit, such as increasing the heat or adding ingredients. In an emergency, pressing the hardware emergency stop button immediately sends a stop command to the kitchen, cutting off the heating power and locking the robotic arm's movement.

[0025] In this embodiment, as Figure 3As shown, the network transmission and synchronization control subsystem 2 includes an action data precoding and compression module 201, a bidirectional low-latency transmission channel module 202, a synchronization timing control module 203, and a safety redundancy and fault handling module 204. The input of the action data precoding and compression module 201 is connected to the output of the chef-side multimodal interaction subsystem 1, and the output is connected to the input of the bidirectional low-latency transmission channel module 202. The synchronization timing control module 203 is bidirectionally connected to both the bidirectional low-latency transmission channel module 202 and the safety redundancy and fault handling module 204. The output of the safety redundancy and fault handling module 204 is connected to the emergency stop control of the kitchen-side intelligent cooking execution subsystem 3.

[0026] The motion data precoding and compression module 201 is used to extract features and compress and encode motion data, intention data, and prediction data collected from the chef's end. The motion data precoding and compression module 201 includes a motion keyframe extraction unit, a sparse coding unit, a forward error correction coding unit, and a data packetization unit. The keyframe extraction unit extracts keyframes based on the motion's speed, acceleration, and intent changes, including the motion start frame, peak frame, end frame, and intent switching frame. Non-keyframes are predicted using linear interpolation, transmitting only keyframe data and interpolation correction parameters. The sparse coding unit performs sparse coding based on a pre-trained dictionary learning model, compressing the motion time-series data. The forward error correction coding unit uses RS(255,233) forward error correction coding to encode data packets, correcting errors up to 11 bytes. The data packetization unit divides the encoded data into packets with an MTU of 1400 bytes, adding a global timestamp, sequence number, and data type identifier to each data packet for easy sorting, reassembly, and synchronization by the receiving end.

[0027] The bidirectional low-latency transmission channel module 202 is used to establish a bidirectional transmission channel between the chef's end and the kitchen end, realizing the downlink transmission of action commands and the uplink feedback of cooking status data. The bidirectional low-latency transmission channel module 202 includes a main transmission link unit, a backup transmission link unit, a link status monitoring unit, and a redundancy switching unit. The main transmission link unit uses a 5G SA / fiber optic leased line, and the backup transmission link unit uses a 4G CAT1 / wired backup leased line. Both links use the UDP protocol for transmission. The link status monitoring unit collects latency, jitter, and packet loss rate data of both links, and its output is connected to the redundancy switching unit. The output of the redundancy switching unit is connected to both the main transmission link unit and the backup transmission link unit. The link status monitoring unit has a sampling frequency of 10Hz and monitors the transmission quality of both links. When the packet loss rate of the main link is ≥5% or the one-way latency is ≥50ms and persists for more than 100ms, the redundancy switching unit switches to the backup transmission link unit.

[0028] The synchronization timing control module 203 is used to synchronize the global clocks of the chef's end and the kitchen end, predict and compensate for network transmission delays, and improve the consistency of actions performed by the chef's end and the kitchen end. The synchronization timing control module 203 includes a PTP precision clock synchronization unit, a network delay prediction unit, a feedforward compensation unit, and an action smoothing interpolation unit. The PTP precision clock synchronization unit is deployed at both the chef's end and the kitchen end, synchronizing the global clocks through the transmission channel. The input of the network delay prediction unit is connected to the link status monitoring unit, and the output is connected to the feedforward compensation unit. The output of the feedforward compensation unit is connected to the action smoothing interpolation unit. The output of the action smoothing interpolation unit is connected to the input of the kitchen end intelligent cooking execution subsystem 3. The PTP precision clock synchronization unit, based on the IEEE 1588 PTPv2 precision time protocol, performs global clock synchronization between the chef's and kitchen's terminals, adding a unified global timestamp to all data as a benchmark for time synchronization. The network latency prediction unit performs real-time network latency prediction, using the ARIMA(3,1,1) time series prediction model to predict the one-way network transmission latency within the next 100ms based on 100 historical sampling points of one-way latency data. The expression is as follows: , In the above formula, Let be the predicted one-way transmission delay at time t; C is a constant term, obtained by fitting historical delay data. is the autoregressive coefficient, with a value range of [-1, 1]; are the measured one-way delays at times t-1, t-2, and t-3, respectively; The moving average coefficient has a value range of [-1, 1]. The prediction residual at time t-1; Let be the white noise residual at time t.

[0029] The feedforward compensation unit is used for feedforward compensation based on the predicted transmission delay. Based on the motion prediction data for the next 200ms output by the motion prediction unit, feedforward compensation is performed on the execution commands in the kitchen. The execution time of the compensated commands is then reduced. The expression is as follows:

[0030] In the above formula, For the global timestamp of action capture, For fixed timing offset, To predict one-way transmission delay.

[0031] The motion smoothing interpolation unit is used to perform motion smoothing interpolation. When data packet loss or delay jitter occurs, it uses a cubic B-spline interpolation algorithm based on the received keyframe data to smooth the missing motion data.

[0032] The safety redundancy and fault handling module 204 is used to monitor the system's operating status and trigger safety protection strategies when network interruptions, data anomalies, or execution failures occur. The safety redundancy and fault handling module 204 includes a heartbeat detection unit, an abnormal data filtering unit, a safety shutdown strategy unit, and a breakpoint continuation execution unit. The heartbeat detection unit is deployed at both the chef's end and the kitchen end, sending heartbeat packets bidirectionally through a transmission channel. The input of the abnormal data filtering unit is connected to the output of the synchronization timing control module 203, and its output is connected to both the safety shutdown strategy unit and the kitchen-end intelligent cooking execution subsystem 3. The output of the safety shutdown strategy unit is connected to the emergency stop control terminal of the kitchen-end intelligent cooking execution subsystem 3. The heartbeat detection unit performs heartbeat detection, sending heartbeat packets bidirectionally between the chef's end and the kitchen end. If the kitchen end does not receive a heartbeat packet from the chef's end for three consecutive heartbeat cycles, it is determined that the network is interrupted and the safety shutdown policy is immediately triggered. The abnormal data filtering unit is used to filter abnormal data. It performs threshold verification on the received action command data. When the action command exceeds the safe stroke of the robotic arm, the firepower exceeds the safe threshold, or the amount of ingredients exceeds the rated range, the abnormal data is filtered, the command is not executed, and an abnormal alarm is sent to the chef's end. When the safety shutdown policy unit triggers the safety shutdown, it immediately cuts off the power to the heating module, resets the robotic arm, places the pot stably on the stove surface, and shuts down all feeding mechanisms. The breakpoint continuation execution unit resumes the cooking action from the breakpoint based on the global timestamp after the network is restored.

[0033] In this embodiment, as Figure 4 As shown, the intelligent cooking execution subsystem 3 in the kitchen includes a multi-degree-of-freedom cooking execution module 301, an intelligent firepower and temperature control execution module 302, a precise ingredient dispensing execution module 303, and a multi-modal cooking environment acquisition module 304. The input terminals of the multi-degree-of-freedom cooking execution module 301, the intelligent firepower and temperature control execution module 302, and the precise ingredient dispensing execution module 303 are all connected to the output terminal of the synchronous timing control module 203; the output terminal of the multi-modal cooking environment acquisition module 304 is connected to the uplink feedback input terminal of the bidirectional low-latency transmission channel module 202; and the emergency stop control terminals of each module are all connected to the output terminal of the safety redundancy and fault handling module 204.

[0034] The multi-degree-of-freedom cooking execution module 301 receives synchronized motion commands and intention commands, decouples and maps the chef's body movements, plans the motion trajectory of the robotic arm, and replicates the cooking actions. The multi-degree-of-freedom cooking execution module 301 includes a 6-axis collaborative robotic arm, an adaptive cookware gripping mechanism, a motion decoupling unit, a trajectory planning and adaptation unit, and a servo drive unit. The input of the motion decoupling unit is connected to the output of the synchronization timing control module 203, and the output of the motion decoupling unit is connected to the trajectory planning and adaptation unit. The output of the trajectory planning and adaptation unit is connected to the servo drive unit. The output of the servo drive unit is connected to the 6-axis collaborative robotic arm. The adaptive cookware gripping mechanism is fixed to the end flange of the robotic arm. The 6-axis collaborative robotic arm uses the UR10e; the adaptive cookware gripping mechanism has a built-in pressure sensor; the motion decoupling unit decouples the arm and waist motion data collected from the chef's end into degrees of freedom for cooking actions, including vertical lifting Z, horizontal forward / backward movement X, horizontal left / right movement Y, rotation angle around the cookware center A, cookware pitch angle B, and cookware flipping angle C, for a total of 6 degrees of freedom; the trajectory planning and adaptation unit performs proportional mapping and inertia compensation based on the chef's operating space and the robotic arm's workspace in the kitchen. The trajectory mapping expression for the tossing motion is as follows:

[0035] In the above formula, Let be the spatial coordinates of the robotic arm's end effector at time t; Let t be the spatial coordinates of the point where the chef holds the pot handle at time t; This is a spatial coordinate mapping scaling factor, based on the dimensional ratio between the chef's operating space and the robotic arm's workspace. The coordinate offset of the robotic arm's workspace is calibrated based on the initial position of the cookware; These represent the rotation, pitch, and flip angles of the robotic arm's end effector at time t; These represent the rotation, tilt, and flip angles at which the chef holds the pot at time t; These are the angle mapping ratio coefficients, calibrated based on cookware specifications.

[0036] The trajectory planning and adaptation unit optimizes the acceleration of the tossing motion based on the total weight m of the pot and ingredients. The expression is as follows:

[0037] In the above formula, The mapped acceleration, The standard weight of cookware and ingredients is specified. This represents the total weight of the cookware and ingredients.

[0038] The servo drive unit, based on the planned motion trajectory, controls the servo motors of the robotic arm via the EtherCAT bus, improving the accuracy of motion replication. The intelligent heat and temperature control execution module 302 identifies the chef's intention to adjust the heat and controls the heating power. The intelligent heat and temperature control execution module 302 includes an electromagnetic heating module, a multi-point thermocouple temperature measuring array, a heat closed-loop control unit, and an emergency stop protection unit. The input of the heat closed-loop control unit is connected to the output of the synchronous timing control module 203 and the output of the multi-point thermocouple temperature measuring array, and the output is connected to the electromagnetic heating module. The multi-point thermocouple temperature measuring array is installed on the stove surface, the bottom of the pot, and the inner wall of the pot. The input of the emergency stop protection unit is connected to the safety redundancy and fault handling module 204, and the output is connected to the power control terminal of the electromagnetic heating module. The multi-point thermocouple temperature measuring array uses K-type thermocouples; the heat closed-loop control unit uses a PID closed-loop control algorithm based on the target pot temperature. Compared with the measured pot temperature To adjust the heating power P in real time based on the deviation, the PID control expression is as follows:

[0039] , In the above formula, Let be the heating power at time t. Let t be the temperature deviation of the pot at time t. These are the proportional coefficient, integral coefficient, and differential coefficient, respectively. For the target pot temperature, This is the actual measured pot temperature.

[0040] When the pot body temperature exceeds 300℃, the pot body is dry-burned for more than 10 seconds, and an emergency stop command is triggered, the emergency stop protection unit immediately cuts off the heating power supply.

[0041] The precision dispensing execution module 303 is used to identify the chef's dispensing intentions and control the type, weight, timing, and speed of dispensing, replicating the chef's dispensing actions. The precision dispensing execution module 303 includes a multi-channel hopper module, a quantitative dispensing mechanism, a dispensing timing control unit, and a weighing feedback closed-loop unit. The input of the dispensing timing control unit is connected to the output of the synchronous timing control module 203, and the output is connected to the quantitative dispensing mechanism. The input of the weighing feedback closed-loop unit is connected to a weighing sensor inside the hopper, and the output is connected to the dispensing timing control unit. The multi-channel hopper module includes a main hopper, a liquid seasoning hopper, a powder seasoning hopper, and a solid auxiliary material hopper, each with a corresponding quantitative dispensing mechanism. The liquid seasoning hopper uses a high-precision peristaltic pump, the powder seasoning hopper uses a screw metering feeding mechanism, and the solid auxiliary material hopper uses a vibrating feeding mechanism. A high-precision weighing sensor is installed at the bottom of each hopper. The dispensing timing control unit controls the dispensing order and interval of different hoppers according to the chef's dispensing sequence, replicating the chef's dispensing rhythm.

[0042] The cooking environment multimodal acquisition module 304 is used to collect data on the cooking environment, pot status, and food status in the kitchen, and then transmit the encoded and compressed data back to the chef. The cooking environment multimodal acquisition module 304 includes a 360° panoramic camera array, a pot end-view camera, a spatial audio acquisition unit, a multi-physical quantity sensor array, and a food status recognition unit. The outputs of the panoramic camera array, the pot end-view camera, the spatial audio acquisition unit, and the multi-physical quantity sensor array are respectively connected to the food status recognition unit and the encoding and compression unit. The output of the encoding and compression unit is connected to the uplink feedback input of the bidirectional low-latency transmission channel module 202. The panoramic camera array consists of four industrial cameras installed around the cooking counter; the internal inspection camera is a 2K wide-angle camera installed below the range hood, facing the inside of the pot; the spatial audio acquisition unit uses a dual-channel noise-canceling microphone; the multi-physical sensor array includes a temperature and humidity sensor, an oil fume concentration sensor, a pot body thermocouple, and an oil temperature sensor; the food status recognition unit, based on the YOLOv8 deep learning model, performs real-time recognition of food images inside the pot, detecting the color, shape, and doneness of the food, and feeding the doneness data back to the chef; the encoding and compression unit performs H.265 encoding and compression on the acquired video, audio, and sensor data, and transmits it back to the chef in real time through the uplink transmission channel.

[0043] In this embodiment, as Figure 5 As shown, the cooking technique digital twin modeling subsystem 4 includes a cooking process data acquisition and annotation module 401, a cuisine technique feature extraction module 402, a digital recipe twin modeling module 403, and a technique model iterative optimization module 404. The input of the cooking process data acquisition and annotation module 401 is connected to the chef-side multimodal interaction subsystem 1 and the kitchen-side intelligent cooking execution subsystem 3, respectively, and its output is connected to the cuisine technique feature extraction module 402. The output of the cuisine technique feature extraction module 402 is connected to the digital recipe twin modeling module 403. The output of the digital recipe twin modeling module 403 is connected to the technique model iterative optimization module 404. The output of the technique model iterative optimization module 404 is connected to the cuisine cooking method library.

[0044] The cooking process data acquisition and annotation module 401 is used to collect, structure, and manually annotate multi-source data of the entire cooking process, providing an annotated dataset for the technique model. The module includes a multi-source data synchronous acquisition unit, a data structuring processing unit, a manual annotation unit, and a dataset management unit. The multi-source data synchronous acquisition unit, based on a global PTP clock, collects action timing data, force data, heat parameters, pot temperature curves, ingredient timing, ingredient status data, and finished product sensory evaluation data of the entire cooking process. All data is stamped with a unified global timestamp. The data structuring processing unit cleans, deduplicates, and normalizes the collected multi-source data, categorizing it according to dish, cuisine, chef, and cooking stage to form a standardized dataset. The manual annotation unit uses professional chefs to annotate the dataset, including cooking stage divisions, core technique actions, key heat points, key ingredient timing, finished product taste evaluation, and core technique points. The dataset management unit manages the annotated dataset by version, categorizes and stores it, and allows retrieval and retrieval by cuisine, dish, and chef.

[0045] The Cuisine Technique Feature Extraction Module 402 is used to extract the core features of cooking techniques from different cuisines, dishes, and chefs from the labeled cooking dataset, and to analyze the chefs' stir-frying patterns and the characteristics of different cuisines. The Cuisine Technique Feature Extraction Module 402 includes a temporal feature extraction unit, an action pattern clustering unit, a technique correlation analysis unit, and a feature weight calculation unit. The temporal feature extraction unit uses the Dynamic Time Warping (DTW) algorithm to align the action timing, heat timing, and ingredient feeding timing data of different cooking samples to extract temporal features. The action pattern clustering unit uses the DBSCAN density clustering algorithm to cluster the extracted action features and identify the core action patterns of different cuisines, such as the stir-frying and tossing pattern of Sichuan cuisine and the slow-cooking pattern of Cantonese cuisine. The technique correlation analysis unit uses Pearson correlation coefficient analysis to analyze the correlation between action features, parameter features, and the evaluation of the finished product's taste, identifying the core technique features affecting the quality of the dish. The correlation coefficient calculation expression is as follows: , In the above formula, The Pearson correlation coefficient between characteristic X and finished product evaluation Y. Let X be the covariance between the characteristic X and the finished product evaluation Y. Let X be the standard deviation of the characteristic. Let Y be the standard deviation of the finished product evaluation.

[0046] The feature weight calculation unit sets the weights of different techniques and features: core feature weight ≥ 0.7, secondary feature weight 0.3-0.7, and redundant feature weight < 0.3.

[0047] The digital recipe twin modeling module 403 is used to transform the extracted core technical features into an executable digital recipe twin model. The digital recipe twin modeling module 403 includes a model structuring encapsulation unit, an executable logic compilation unit, a model simulation verification unit, and a model output unit. The model structuring encapsulation unit structures the core technical features according to cooking stages, including the preparation stage, preheating stage, ingredient feeding stage, cooking stage, and serving stage. Each stage encapsulates corresponding action parameters, heat parameters, ingredient feeding parameters, timing parameters, and exception handling strategies. The executable logic compilation unit compiles the encapsulated model into control logic that can be directly executed by the kitchen-end cooking machine, including executable instructions for robotic arm trajectory planning, heat control, and ingredient feeding control. The model simulation verification unit uses a digital twin simulation platform to simulate and verify the compiled digital recipe model, simulating the entire cooking process, verifying the feasibility of action execution and the rationality of parameters, and correcting abnormal parameters. The model output unit outputs the verified digital recipe model to the cuisine cooking method library, supporting direct execution.

[0048] The technique model iterative optimization module 404 iterates and optimizes the digital recipe model based on the finished product effect data from multiple cooking sessions, improving the model's replicability and the quality of the dishes. The technique model iterative optimization module 404 includes a finished product effect evaluation unit, a model parameter correction unit, a model iteration unit, and a version management unit. The finished product effect evaluation unit quantitatively evaluates the cooked product from sensory dimensions such as color, aroma, taste, shape, and texture, as well as physicochemical dimensions such as ingredient doneness, salt content, and oil content, obtaining a comprehensive evaluation score. The model parameter correction unit corrects the core parameters of the digital recipe model based on the deviation between the evaluation score and the target score using a gradient descent algorithm. The model iteration unit iterates and optimizes the model based on the corrected data from multiple cooking sessions, with at least 10 iterations. The version management unit manages the version of the iterated model, recording parameter changes and evaluation results for each iteration.

[0049] Example 2 like Figure 6 As shown, a remote synchronous cooking method includes the following steps: S1. Collect the chef's limb movement data, operational force data and electromyographic signals at the chef's end to identify and predict the chef's intention to perform cooking operations. S2. The collected data and recognition results are encoded and compressed, and transmitted to the kitchen end through a low-latency network channel. During the transmission process, global clock synchronization, network latency prediction and feedforward compensation are performed. S3. Receive and parse the processed data at the kitchen end, decouple and map it into the motion trajectory of the robotic arm to perform cooking actions, control the fire and add ingredients, and collect multimodal data of the cooking environment. S4. The collected multimodal data of the cooking environment is encoded and transmitted back to the chef's end, and presented to the chef through VR devices in the form of panoramic vision, spatial audio and haptic feedback.

[0050] It also includes S5, which involves extracting technique features based on the cooking process data collected in steps S1 to S3 to construct a digital recipe twin model, and iteratively optimizing the model based on the evaluation of the finished cooking product.

[0051] In step S1, the chef's limb movement data, operational force data, and electromyographic signals are collected at the chef's end to identify and predict the cooking operation intention, including the following steps: S101, Pre-cooking configuration parameters; Chefs can configure dish names, cuisines, ingredient types, ingredient weights, cookware specifications, and safety threshold parameters through the instruction and parameter configuration module, and perform system calibration and coordinate system alignment. S102. Collect limb movement data; Using 12 inertial motion capture nodes and 4 optical correction cameras, the system collects full-dimensional limb movement data of the chef's upper arm, forearm, wrist, palm, knuckles and waist at a sampling rate of 200Hz. After denoising, filtering and coordinate system transformation, standardized motion timing data is output. S103. Collect force sensation and electromyography data; Operational force data is collected by a 6-dimensional force / torque sensor inside the pot handle and shovel handle at a sampling rate of 100Hz; forearm electromyography signals are collected by a 4-channel sEMG sensor at a sampling rate of 1000Hz, and the leading features of the electromyography signals are extracted. S104. Identify cooking intentions and predict actions; Based on motion features, force sensory features, and electromyographic features, the LightGBM model is used to identify the chef's cooking intentions. Based on the leading characteristics of electromyographic signals, the movement trajectory and operational intention within the next 200ms can be predicted and the prediction data can be output in advance. The VR headset receives 360° panoramic images, spatial audio, and tactile feedback on pot temperature transmitted from the kitchen.

[0052] In step S2, the collected data and recognition results are encoded and compressed, and transmitted to the kitchen end through a low-latency network channel. During the transmission process, global clock synchronization, network latency prediction, and feedforward compensation are performed, including the following steps: S201. Encode and compress the motion data: Extract keyframes of the action, and use sparse coding and RS-FEC coding to compress and correct the action data, intent data and prediction data. Packetize according to MTU=1400 bytes and add global timestamps. S202, Transmission via dual redundant channels: Data transmission is performed via the main link 5G SA / fiber optic leased line; The link status monitoring unit monitors the transmission quality in real time. When the main link is abnormal, it switches to the backup 4GCAT1 / wired leased line within 20ms. S203, Synchronize global clock; Based on the IEEE 1588 PTPv2 protocol, synchronize the global clock between the chef's end and the kitchen end; S204. Perform delay prediction and feedforward compensation; The ARIMA(3,1,1) model is used to predict the one-way transmission delay of the network, and feedforward compensation is performed on the executed instructions based on the predicted delay. S205. Perform smooth interpolation on the data; When data packet loss or latency jitter occurs, a cubic B-spline interpolation algorithm is used to smoothly interpolate the missing data.

[0053] In step S3, the processed data is received and parsed at the kitchen end, decoupled and mapped into a robotic arm motion trajectory for performing cooking actions, controlling the heat and adding ingredients, and collecting multimodal data of the cooking environment, including the following steps: S301, Perform action decoupling and mapping; The received motion data is decoupled into 6 core degrees of freedom, and the expression is calculated through trajectory mapping to map the chef's actions into the motion trajectory of the robotic arm. Inertial compensation and trajectory optimization are performed based on the weight of the pot and ingredients. S302, Perform the tossing and stir-frying actions; The 6-axis collaborative robotic arm is controlled via EtherCAT bus to perform tossing and stir-frying actions according to the planned trajectory. S303, Identify the intention to adjust firepower and adjust the power accordingly; The system identifies the chef's intention to adjust the heat and uses a PID control algorithm to adjust the power of the electromagnetic heating module in real time to control the pot temperature. S304. Identify the feeding intention and feed the material; The system identifies the chef's intention to add ingredients and controls the multi-channel hoppers to add ingredients according to the preset amount, timing, and speed, while ensuring the accuracy of the addition through weighing. S305. Collect cooking process data; The system uses a panoramic camera array, an endoscope camera, and a sensor array to collect real-time data on the cooking environment, pot status, and food status.

[0054] In step S4, the collected multimodal data of the cooking environment is encoded and transmitted back to the chef's end, and presented to the chef through VR devices in the form of panoramic vision, spatial audio and haptic feedback, including the following steps: S401. Encode and transmit the feedback data back; The collected video, audio, and sensor data are encoded and compressed using H.265 and then transmitted back to the chef's end in real time via the uplink transmission channel; S402, Presenting visuals and audio through a VR headset; The four returned images are stitched together in a panoramic view, and a 360° panoramic view is presented to the chef through a VR headset. The kitchen cooking sounds are reproduced through a spatial audio unit. S403, perceive the cooking status through tactile feedback; The real-time temperature of the pot body is mapped to the Peltier unit of the simulated pot handle, and the vibration motor restores the vibration feedback of tossing and stir-frying, allowing the chef to perceive the remote cooking status in real time. S404. Adjust actions based on cooking status; Chefs adjust cooking actions, heat levels, and ingredient addition rhythms based on feedback to ensure the quality and consistency of remote cooking.

[0055] In step S5, based on the cooking process data collected in steps S1 to S3, technique features are extracted to construct a digital recipe twin model, and the model is iteratively optimized based on the evaluation of the finished cooking product, including the following steps: S501, synchronously collect and label data throughout the entire process; Based on the global PTP clock, multi-source data of the entire cooking process is collected, structured and manually labeled by professional chefs to form a standardized dataset; S502, Extracting the characteristics of culinary techniques; The DTW algorithm is used to extract temporal features, the DBSCAN algorithm is used to cluster core action patterns, and Pearson correlation coefficient analysis is used to mine core technique features, filter redundant features, and extract the chef's cooking patterns and culinary characteristics. S503. Establish a digital recipe twin model; The core techniques are encapsulated in a structured manner and compiled into a digital recipe model that can be directly executed by the cooking machine. After verification, the model is output to the cooking method library. S504. Iteratively optimize the recipe twin model; Based on quantitative evaluation data of the finished product effect from multiple cookings, the gradient descent algorithm is used to correct the model parameters and iteratively optimize the model. The digital recipe models are classified and version-managed to form a standardized library of cooking methods for different cuisines.

[0056] This invention enables the remote replication of professional chefs' cooking movements. Through wearable multimodal motion capture, electromyographic intention prediction, and feedforward delay compensation, it can perform professional cooking actions such as wok tossing, precise ingredient addition, and fine-tuning of heat, solving the core pain point of existing technologies that cannot achieve remote real-time cooking by professional chefs. It not only replicates the chef's limb movement trajectory but also identifies the chef's cooking operation intention through electromyographic signals and force data. Combined with the parameters of the pots and ingredients at the kitchen end, it adapts the movements and optimizes the trajectory, greatly improving the system's adaptability and execution success rate. Through dual redundant transmission channels, PTP precision clock synchronization, and delay prediction feedforward compensation, it solves the problems of asynchronous movements and erroneous execution caused by network fluctuations and packet loss, and has high operational stability and security.

[0057] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Clearly, those skilled in the art can make various alterations and variations to the invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of equivalents of the invention, the invention is also intended to include these modifications and variations.

Claims

1. A remote synchronized cooking system, characterized by, include: The system comprises a chef-side multimodal interaction subsystem, a network transmission and synchronization control subsystem, a kitchen-side intelligent cooking execution subsystem, and a digital twin modeling subsystem for cooking techniques. The chef-side multimodal interaction subsystem includes a limb motion capture module for collecting chef's limb movement data, a force perception and operation intention perception module for collecting operation force data and electromyographic signals to identify operation intentions, and a VR immersive cooking feedback module for providing immersive feedback to chefs. The network transmission and synchronization control subsystem includes an action data precoding and compression module for compressing and correcting the acquired data, a bidirectional low-latency transmission channel module for establishing a dual-redundant low-latency transmission channel, a synchronization timing control module for synchronizing the clocks at both ends and predicting and compensating for network latency, and a safety redundancy and fault handling module for system safety monitoring and fault handling. The kitchen-side intelligent cooking execution subsystem includes a multi-degree-of-freedom cooking execution module for replicating the chef's cooking actions, an intelligent firepower and temperature control execution module for precisely controlling heating, a precision ingredient feeding execution module for accurately adding ingredients according to intent, and a cooking environment multi-modal acquisition module for collecting multi-modal data of the cooking environment. The cooking technique digital twin modeling subsystem extracts technique features from cooking process data, and constructs and optimizes an executable digital recipe twin model.

2. The remote synchronized cooking system of claim 1, wherein, The limb motion capture module includes several inertial motion capture nodes worn on the chef's upper limbs and waist, an optical correction camera for drift correction of inertial motion capture data, and a motion data preprocessing unit for converting the data to a globally unified coordinate system.

3. The remote synchronized cooking system of claim 1, wherein, The force and operational intention perception module includes a six-dimensional force / torque sensor embedded in the simulated kitchen utensils, several surface electromyography sensors attached to the chef's forearm, and an operational intention recognition unit that identifies cooking operational intentions based on motion features, force features, and electromyography features through a pre-trained classification model. The intention recognition unit is configured to predict the trajectory of movement in the future time period based on the advanced features of electromyographic signals.

4. The remote synchronized cooking system of claim 1, wherein, The synchronization timing control module includes: PTP precision clock synchronization unit based on precision time protocol to synchronize global clocks at both ends; A network latency prediction unit for predicting future one-way network transmission latency based on historical latency data; Feedforward compensation unit is used to perform feedforward compensation on execution instructions sent to the kitchen end based on predicted delay and action prediction data.

5. The remote synchronized cooking system of claim 1, wherein, The multi-degree-of-freedom cooking execution module includes: Used to decouple chef's limb movement data into motion decoupling units with multiple degrees of freedom; This is a trajectory planning and adaptation unit used to map decoupled actions to the end effector trajectory of a robotic arm based on the spatial ratio between the chef's end and the kitchen end, and to perform inertial compensation based on the total weight of the cookware and ingredients.

6. The remote synchronized cooking system of claim 1, wherein, The digital twin modeling subsystem for cooking techniques includes: A cooking process data acquisition and annotation module for collecting and labeling multi-source data of a single cooking process; A cuisine technique feature extraction module for extracting core technique features of different cuisines from labeled data; A digital recipe twin modeling module used to encapsulate and compile core technical features into executable control instructions; This module is used for iterative optimization of the technique model of a digital recipe model based on multiple evaluation data of cooked products.

7. A method for remote synchronized cooking, suitable for a remote synchronized cooking system according to any one of claims 1-6, characterized in that, Includes the following steps: S1. Collect the chef's limb movement data, operational force data and electromyographic signals at the chef's end to identify and predict the chef's intention to perform cooking operations. S2. The collected data and recognition results are encoded and compressed, and transmitted to the kitchen end through a low-latency network channel. During the transmission process, global clock synchronization, network latency prediction and feedforward compensation are performed. S3. Receive and parse the processed data at the kitchen end, decouple and map it into the motion trajectory of the robotic arm to perform cooking actions, control the fire and add ingredients, and collect multimodal data of the cooking environment. S4. The collected multimodal data of the cooking environment is encoded and transmitted back to the chef's end, and presented to the chef through VR devices in the form of panoramic vision, spatial audio and haptic feedback.

8. The remote synchronized cooking method of claim 7, wherein, It also includes S5, which extracts technical features based on the cooking process data collected from S1 to S3 to build a digital recipe twin model, and iteratively optimizes the model based on the evaluation of the cooked product.

9. The remote synchronized cooking method of claim 7, wherein, In S1, the intention to perform the cooking operation is identified and predicted, including: Based on limb movement characteristics, operational force characteristics, and electromyographic characteristics, the intention to toss the wok, stir-fry, add ingredients, or adjust the heat is identified through a classification model. Based on the characteristic of electromyographic signals that precede limb movements, the trajectory of movements and operational intentions can be predicted within the next 200-500ms.

10. The remote synchronized cooking method of claim 7, wherein, In S2, network latency prediction and feedforward compensation include: A time series forecasting model is used to predict future one-way transmission delays based on historical network delay data; Based on the predicted delay and the motion prediction data obtained from electromyography signals, the compensated timestamp of the kitchen-side execution command is calculated.

Images