Cooking robot-oriented real-time 3D perception and decision service platform
By combining optical 3D point cloud and acoustic vibration data, vibration features are extracted and motion correction is performed, solving the problem that traditional optical 3D sensing technology cannot perceive the internal state of food, and realizing real-time adaptive cooking control of cooking robots.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ZHEJIANG HONGJI MINGDE INFORMATION TECH CO LTD
- Filing Date
- 2026-01-23
- Publication Date
- 2026-05-01
AI Technical Summary
Traditional optical 3D sensing technology cannot directly perceive changes in the internal elasticity and moisture content of food, which limits the intelligent and precise cooking decisions of cooking robots.
By combining optical 3D point cloud data and acoustic vibration data generated by the active contact or excitation of the robotic arm, vibration features are extracted through a data fusion layer to form attribute-enhanced point clouds. Then, finite state automata and vibration feedback closed-loop haptic rendering are used for motion correction to achieve real-time perception and motion adjustment of the internal state of food.
It enables real-time perception of changes in the internal elasticity and moisture of ingredients, improving the adaptability of the cooking robot and the precision of the cooking process, and adapting to changes in different ingredients and cooking scenarios.
Smart Images
Figure CN121963191A_ABST
Abstract
Description
A Real-Time 3D Perception and Decision-Making Service Platform for Cooking Robots Technical Field
[0001] This invention relates to the field of cooking robot technology, and in particular to a real-time 3D perception and decision-making service platform for cooking robots. Background Technology
[0002] With the rapid iteration of artificial intelligence and robotics technologies, cooking robots have evolved from performing simple mechanical actions to making intelligent and precise cooking decisions, becoming a core development direction in the fields of home service and industrialized catering. The core logic lies in transforming traditionally vague cooking techniques into quantifiable and controllable scientific parameters, a process that highly depends on the ability to comprehensively perceive the state of the ingredients.
[0003] Current cooking robots, equipped with 3D vision modules, emit optical signals onto the surface of food and receive reflected signals. Based on triangulation or time-of-flight principles, they calculate spatial coordinates and reconstruct a 3D point cloud model of the food. This technology can effectively extract the surface geometric features of food, including key information such as shape, volume, surface texture, and spatial location. Combined with deep learning algorithms, it can achieve functions such as food type identification, preliminary freshness assessment, cutting path planning, and ingredient placement.
[0004] However, in the process of realizing the technical solution of this application, the inventors of this application discovered that the above-mentioned technology has at least the following technical problems: traditional optical 3D sensing technology can only obtain geometric and texture information of the surface of food, and cannot directly perceive changes in the elasticity and moisture content of the food. Summary of the Invention
[0005] To overcome the above shortcomings, this invention provides a real-time 3D perception and decision-making service platform for cooking robots, aiming to improve the problem that traditional optical 3D perception technology can only obtain geometric and texture information of the surface of food ingredients, and cannot directly perceive changes in the internal elasticity and moisture content of food ingredients.
[0006] This invention provides the following technical solution: a real-time 3D perception and decision-making service platform for cooking robots, comprising: a perception layer for synchronously acquiring optical 3D point cloud data of the cooking scene and acoustic vibration data generated by active contact or excitation of the robotic arm; a data fusion layer for extracting vibration features from the acoustic vibration data and mapping the vibration features to a point cloud region corresponding to the contact area to form an attribute-enhanced point cloud containing physical attributes; a decision layer for outputting preliminary action parameters based on the geometric rate of change and the vibration feature rate of change, according to a preset finite state automaton transition rule; a vibration feedback closed-loop tactile rendering and action fine-tuning layer for correcting the preliminary action parameters through active excitation and vibration response verification to obtain final action parameters; and a robotic arm controller for executing cooking actions according to the final action parameters and feeding back the execution results to the next cycle, forming continuous closed-loop control.
[0007] Preferably, in the perception layer, the step of simultaneously acquiring optical 3D point cloud data of the cooking scene and acoustic vibration data generated by the active contact or excitation of the robotic arm includes: acquiring three-dimensional point cloud data of the cooking scene using a structured light sensor or a time-of-flight sensor; setting bidirectional piezoelectric elements on the end effector of the robotic arm and preset cooking utensils; acquiring time-domain vibration signals through the bidirectional piezoelectric elements when the robotic arm performs contact, tapping, or active excitation actions; and performing frequency domain conversion on the time-domain vibration signals to obtain corresponding vibration spectrum data.
[0008] Preferably, in the data fusion layer, the step of extracting vibration features from the acoustic vibration data includes: determining the frequency component with the largest energy from the vibration spectrum as the main resonant peak frequency; calculating the vibration attenuation rate based on the trend of high-frequency energy changing with frequency in the vibration spectrum; comparing the current main resonant peak frequency with the resonant peak frequency under the reference state to obtain the resonant peak shift; and outputting the main resonant peak frequency, attenuation rate, and resonant peak shift as acoustic vibration features.
[0009] Preferably, in the data fusion layer, the step of forming an attribute-enhanced point cloud containing physical attributes includes: determining a subset of point clouds corresponding to the contact area of the robotic arm; calculating a hardness index based on acoustic vibration characteristics for each point in the subset of point clouds; calculating a humidity estimate based on the energy attenuation of the high-frequency band of the vibration spectrum; and writing the hardness index and humidity estimate as additional physical attributes into the corresponding point cloud points to form an attribute-enhanced point cloud.
[0010] Preferably, the step of calculating the hardness index based on acoustic vibration characteristics includes: obtaining the current main resonance peak frequency; obtaining the reference resonance peak frequency under the reference reference state; calculating the hardness index based on the ratio of the two and a preset proportionality coefficient; the calculation step of humidity estimation includes: calculating the energy attenuation rate of the high-frequency band of the vibration spectrum; calculating the humidity estimate based on the energy attenuation rate and a preset proportionality coefficient.
[0011] Preferably, in the decision layer, the calculation step of the geometric change rate includes: acquiring the attribute-enhanced point cloud of the contact area in adjacent time steps, calculating the displacement of each point in the attribute-enhanced point cloud between adjacent time steps, averaging the displacement to obtain the geometric change rate; in the decision layer, the calculation step of the vibration characteristic change rate includes: calculating the change in resonance peak shift in adjacent time steps, calculating the average change in hardness index and humidity estimate in adjacent time steps, weighting the above changes to obtain the vibration characteristic change rate.
[0012] Preferably, in the decision-making layer, the step of outputting preliminary action parameters according to the preset finite state automaton transition rules includes: presetting a finite state set containing multiple discrete states, each state corresponding to a cooking stage; using the geometric rate of change and the vibration characteristic rate of change as state transition inputs; performing state transition according to the preset state transition rules; and outputting preliminary action parameters corresponding to the current state after the state transition is completed.
[0013] Preferably, in the vibration feedback closed-loop tactile rendering and motion fine-tuning layer, the step of correcting the preliminary motion parameters through active excitation and vibration response verification includes: driving a bidirectional piezoelectric element with a low-amplitude excitation signal to apply micro-vibration pulses to the food in contact; synchronously acquiring the vibration response spectrum of the food to the micro-vibration pulses; estimating the corresponding expected vibration spectrum based on the current hardness index and humidity; calculating the deviation between the vibration response spectrum and the expected vibration spectrum; and fine-tuning the preliminary motion parameters based on the deviation.
[0014] Preferably, the step of fine-tuning the initial motion parameters based on the deviation includes: calculating a force adjustment coefficient based on the deviation; performing a proportional calculation between the force adjustment coefficient and the initial force to obtain the final force; adjusting the initial duration when the deviation is less than a preset threshold; and outputting the fine-tuned motion parameters as the final motion parameters.
[0015] Preferably, the steps of the continuous closed-loop control include: sending the final motion parameters to the robotic arm controller and executing them; transferring the attribute-enhanced point cloud data and motion parameters obtained during the execution process to the next cycle; recalculating the geometric rate of change and the vibration characteristic rate of change based on the updated data in the next cycle; completing the calculation and decision processing in the edge computing unit, and synchronizing and storing the proportional coefficient, weighting coefficient and state transition rules through the cloud.
[0016] The present invention has the following beneficial effects: 1. The present invention introduces bidirectional piezoelectric elements to collect acoustic vibration data through the sensing layer, and calculates the hardness index and humidity estimate based on the main resonance peak frequency and energy attenuation rate in the data fusion layer and maps them to the point cloud to form an attribute enhancement layer, thereby achieving the effect of extending the sensing ability of the internal elasticity and moisture changes of food on the basis of traditional optical 3D point cloud.
[0017] 2. This invention uses a decision layer to drive a finite state automaton to perform state transitions and output preliminary motion parameters by using the geometric rate of change and the vibration characteristic rate of change. Then, the parameters are corrected by a vibration feedback closed-loop tactile rendering and motion fine-tuning layer through active excitation and response spectrum deviation calculation, so that the robotic arm can dynamically adjust motion parameters according to the real-time state of the ingredients during the cooking process.
[0018] 3. This invention achieves continuous closed-loop control by synchronously triggering acoustic vibration acquisition at key nodes, performing real-time attribute enhancement layer formation, rate of change calculation, state transition, and parameter fine-tuning, and cyclically processing the above processes in the edge computing unit. This enables the platform to achieve adaptive adjustment of the entire cooking process and adapt to changes in different ingredients and cooking scenarios. Attached Figure Description
[0019] Figure 1 is a schematic diagram of the overall structure of a real-time 3D perception and decision-making service platform for cooking robots proposed in this invention; Figure 2 is a diagram of the steps for extracting vibration features from acoustic vibration data of a real-time 3D perception and decision-making service platform for cooking robots proposed in this invention. Detailed Implementation
[0020] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0021] Referring to Figures 1 and 2, in the first embodiment of the present invention, the present invention provides a real-time 3D perception and decision-making service platform for cooking robots, comprising: a perception layer for synchronously acquiring optical 3D point cloud data of the cooking scene and acoustic vibration data generated by active contact or excitation of the robotic arm; a data fusion layer for extracting vibration features from the acoustic vibration data and mapping the vibration features to the point cloud region corresponding to the contact area to form an attribute-enhanced point cloud containing physical attributes; a decision layer for outputting preliminary action parameters based on the geometric rate of change and the vibration feature rate of change, according to a preset finite state automaton transition rule; a vibration feedback closed-loop tactile rendering and action fine-tuning layer for correcting the preliminary action parameters through active excitation and vibration response verification to obtain the final action parameters; and a robotic arm controller for executing cooking actions according to the final action parameters and feeding back the execution results to the next cycle to form continuous closed-loop control.
[0022] Specifically, the perception layer simultaneously acquires optical 3D point cloud data and acoustic vibration data within the same time step. The optical 3D point cloud data is represented as a point set: ;in, Indicates a time step. This represents the index of a point in the point cloud. These represent the three-dimensional coordinates of a point in a unified spatial coordinate system.
[0023] Acoustic vibration data were collected by bidirectional piezoelectric elements mounted on the end effector of the robotic arm and the cooking appliance to obtain time-domain vibration signals. The vibration spectrum is obtained by performing a frequency domain transform on the time-domain vibration signal: ;in, This represents the Fast Fourier Transform operator. Represents frequency variables. This represents the vibration amplitude at the corresponding frequency.
[0024] In the data fusion layer, from the vibration spectrum Vibrational features are extracted from the data. These vibrational features include at least the main resonant peak frequency. High-frequency energy attenuation rate α and resonance peak shift Among them, the main resonant peak frequency Defined as the frequency component with the largest amplitude in the vibration spectrum; resonant peak shift Defined as the current main resonant peak frequency and the reference resonant peak frequency under the reference reference state. The difference between them, that is: ;in, This refers to the resonant peak frequency measured and stored under reference conditions.
[0025] The high-frequency energy attenuation rate α is calculated based on the energy distribution of the vibration spectrum in the high-frequency band. The calculation method is as follows: ;in, Indicates satisfaction The number of frequency points of the condition This indicates the high-low frequency boundary threshold frequency. This indicates the preset low-frequency reference frequency.
[0026] The data fusion layer determines the subset of point clouds corresponding to the food contact area based on the contact pose of the robotic arm: The vibration features are then mapped to points within the point cloud subset to generate an attribute-enhanced point cloud. For each spatial point in the point cloud subset... Define the corresponding hardness index With humidity estimation Among them, hardness index The calculation method is as follows: Humidity estimation The calculation method is as follows: ;in, and All of these are proportionality coefficients obtained through benchmark experiments.
[0027] After mapping, the attribute-enhanced point cloud is represented as: ;in, The original point cloud coordinates, and These are additional physical properties.
[0028] In the decision-making layer, the geometric rate of change and the vibration characteristic rate of change are calculated based on attribute enhancement points at adjacent time steps. Defined as the average spatial displacement of points within the contact area between adjacent time steps, it is calculated as follows: ;in, This represents the number of points in a point cloud subset. The rate of change of vibration characteristics. It is calculated based on the changes in vibration characteristics in adjacent time steps, and the calculation method is as follows:
[0029] in, This indicates the change in the average hardness index between adjacent time steps. This represents the change in the average humidity estimate between adjacent time steps. , , These are the preset weighting coefficients.
[0030] The decision-making level will use geometric rate of change With vibration characteristic change rate As the state transition input of a finite state automaton, preliminary action parameters are output according to a preset state transition rule. The preliminary action parameters include at least the preliminary force. Initial speed and initial duration
[0031] In the vibration feedback closed-loop haptic rendering and motion fine-tuning layer, a low-amplitude excitation signal is applied to the bidirectional piezoelectric element based on preliminary motion parameters, and the corresponding vibration response spectrum is collected. Based on the current hardness index and humidity, the expected vibration increase is estimated and generated. Deviation between the two The calculation method is as follows: Based on deviation Correct the initial motion parameters and calculate the force adjustment coefficient: ;in, The preset adjustment coefficient is used to obtain the final force. ;When the deviation degree When the initial duration is less than a preset threshold, Adjustments were made to obtain the final motion parameters.
[0032] The robotic arm controller receives the final motion parameters and executes the corresponding cooking action. At the same time, it feeds back the attribute-enhanced point cloud data and motion parameters obtained during the execution to the next time step to recalculate the geometric rate of change and the vibration characteristic rate of change, thus forming a continuous closed-loop control process.
[0033] Furthermore, in the perception layer, the steps of simultaneously acquiring optical 3D point cloud data of the cooking scene and acoustic vibration data generated by the active contact or excitation of the robotic arm include: acquiring three-dimensional point cloud data of the cooking scene using a structured light sensor or a time-of-flight sensor; setting bidirectional piezoelectric elements on the end effector of the robotic arm and the preset cooking utensils; acquiring time-domain vibration signals through the bidirectional piezoelectric elements when the robotic arm performs contact, tapping, or active excitation actions; and performing frequency domain conversion on the time-domain vibration signals to obtain the corresponding vibration spectrum data.
[0034] Specifically, structured light sensors or time-of-flight sensors are fixedly mounted on the cooking robot body or an independent support, with their field of view covering the main working area of the robotic arm. The sensors acquire depth information of the cooking scene according to a preset sampling frequency, and based on intrinsic and extrinsic parameter calibration results, convert the depth information into 3D point cloud data in a unified coordinate system. After acquisition, the 3D point cloud data undergoes spatial coordinate alignment processing to ensure consistency with the robotic arm's motion coordinate system.
[0035] During point cloud acquisition, filtering and cropping operations are performed on the raw point cloud data to remove outliers and point clouds outside the work area. The filtering operation includes outlier removal based on statistical distribution, and the cropping operation is based on preset spatial boundaries, retaining only the point cloud data located within the cooking operation area, thus obtaining a valid point cloud set for subsequent processing.
[0036] Bidirectional piezoelectric elements are respectively installed on the end effector of the robotic arm and at least one cooking utensil that comes into direct contact with the food. The bidirectional piezoelectric elements are electrically connected to a signal acquisition circuit to achieve bidirectional conversion between mechanical vibration and electrical signals under mechanical contact or active excitation conditions. The installation position and fixing method of the bidirectional piezoelectric elements ensure that they maintain a stable mechanical coupling relationship with the contacted object under stress or vibration conditions.
[0037] When the robotic arm performs contact, tapping, or active actuation actions, the bidirectional piezoelectric element outputs a time-domain vibration signal corresponding to the contact process. The time-domain vibration signal is represented as... The time-domain vibration signal is amplified and bandpass filtered by the signal conditioning circuit to suppress DC components and non-target frequency noise.
[0038] After acquiring the time-domain signal, a frequency-domain transformation is performed on the time-domain vibration signal to obtain the corresponding vibration spectrum data. The frequency-domain transformation uses the Fast Fourier Transform, and its calculation method is as follows.
[0039] ;in, This represents the complex spectral value at frequency f, where j represents the imaginary unit and f represents the frequency variable. Vibration spectrum data is obtained by taking the amplitude of the complex spectrum to obtain its amplitude-frequency characteristics, which are then used for subsequent vibration feature extraction and analysis.
[0040] During the acquisition of optical 3D point cloud data and acoustic vibration data, the perception layer performs time synchronization control on the two types of data to ensure that the point cloud data and vibration spectrum data acquired in the same time step correspond to the same robotic arm contact or excitation event, thereby providing a consistent data foundation for subsequent data fusion and feature mapping.
[0041] Furthermore, in the data fusion layer, the steps for extracting vibration features from acoustic vibration data include: determining the frequency component with the highest energy in the vibration spectrum as the main resonant peak frequency; calculating the vibration attenuation rate based on the trend of high-frequency energy changing with frequency in the vibration spectrum; comparing the current main resonant peak frequency with the resonant peak frequency under the reference state to obtain the resonant peak shift; and outputting the main resonant peak frequency, attenuation rate, and resonant peak shift as acoustic vibration features.
[0042] Specifically, the vibration spectrum is obtained by frequency domain transformation of the time-domain vibration signal, and the vibration spectrum is expressed as a function of amplitude changing with frequency. In the vibration spectrum, by traversing the entire frequency band, the frequency component with the largest amplitude is determined, and its corresponding frequency value is defined as the main resonant peak frequency. To avoid the influence of instantaneous noise on peak value determination, the main resonant peak frequency is calculated multiple times within a preset time window, and the frequency component that appears stably is taken as the main resonant peak frequency of the current time step.
[0043] Vibration attenuation rate is used to characterize the energy variation of the vibration spectrum with frequency in the high-frequency range. Firstly, it is determined based on a preset high-low frequency boundary threshold. The vibration spectrum is divided into low-frequency and high-frequency bands; then, a continuous set of frequency sampling points is selected within the high-frequency band. Calculate the corresponding amplitude Logarithmic relationship: Finally, the logarithmic rate of change of the amplitude in the high-frequency band is averaged to obtain the vibration attenuation rate. The calculation method is as follows: ;in, This indicates the number of frequency points involved in the calculation within the high-frequency band. This indicates a preset reference frequency point used to normalize the amplitude.
[0044] The resonant peak shift is obtained by comparing the current main resonant peak frequency with the resonant peak frequency under the reference state. (Resonant peak frequency under the reference state) To collect and store frequency values under preset reference conditions, these conditions must include at least standardized mechanical contact methods and food samples in known physical states. Resonance peak shift. The calculation method is as follows: ;in, It reflects the degree of frequency change of the current contact state relative to the reference state.
[0045] After completing the above calculations, the main resonant peak frequency will be... Vibration attenuation rate and resonance peak shift Composition of acoustic vibration eigenvectors: The acoustic vibration feature vector is then output to the subsequent data fusion layer for association mapping with the optical 3D point cloud data. The vibration features are acquired according to the same calculation process at each time step to ensure the comparability and consistency of vibration features across different time steps.
[0046] Furthermore, in the data fusion layer, the steps for forming an attribute-enhanced point cloud containing physical attributes include: determining a subset of the point cloud corresponding to the contact area with the robotic arm; calculating a hardness index based on acoustic vibration characteristics for each point in the point cloud subset; calculating a humidity estimate based on the energy attenuation of the high-frequency band of the vibration spectrum; and writing the hardness index and humidity estimate as additional physical attributes into the corresponding point cloud points to form an attribute-enhanced point cloud. The steps for calculating the hardness index based on acoustic vibration characteristics include: obtaining the current main resonance peak frequency; obtaining the reference resonance peak frequency under the reference reference state; and calculating the hardness index based on the ratio of the two and a preset scaling factor. The steps for calculating the humidity estimate include: calculating the energy attenuation rate of the high-frequency band of the vibration spectrum; and calculating the humidity estimate based on the energy attenuation rate and a preset scaling factor.
[0047] Specifically, the point cloud subset corresponding to the contact area of the robotic arm is determined based on the pose information of the robotic arm's end effector in a unified spatial coordinate system. By obtaining the spatial position coordinates of the robotic arm's end effector at the current time step, and constructing a spatial neighborhood with a preset radius centered on this position, points located within this spatial neighborhood are selected from the 3D point cloud data to form the point cloud subset corresponding to the contact area. The point cloud subset is represented as follows: ;in, This represents the point cloud set at the current time step. This indicates the spatial coordinates of the robotic arm's end effector at the aforementioned time step. This represents the preset neighborhood radius.
[0048] After determining the subset of the point cloud, physical properties related to acoustic vibration characteristics are introduced to each point in the subset. The hardness index is used to characterize the mechanical response of the contact area in the current state, and its calculation is based on the relationship between the current dominant resonant peak frequency and the reference resonant peak frequency under the reference state. Specifically, the hardness index... The calculation method is as follows ;in, This indicates the frequency of the main resonant peak extracted at the current time step. This represents the reference resonant peak frequency measured and stored under reference conditions. This represents the scaling factor obtained through calibration. The hardness index remains consistent within the point cloud subset and is used to characterize the overall hardness characteristics of the contact area at the current time step.
[0049] Humidity estimation is used to characterize the absorption characteristics of high-frequency vibration energy in the contact area, and its calculation is based on the energy attenuation in the high-frequency band of the vibration spectrum. First, based on a preset high-frequency starting frequency... Determine the high-frequency analysis band; then calculate the average attenuation rate of the vibration spectrum amplitude as a function of frequency within the high-frequency band to obtain the high-frequency energy attenuation rate. Based on this, humidity estimation The calculation method is as follows ;in, This represents the proportionality coefficient obtained through benchmark experiments.
[0050] After completing the calculation of the hardness index and humidity estimation, the hardness index is... With humidity estimation The physical properties are written to each point in the point cloud subset as additional physical attributes. The resulting point cloud data is represented as follows: ;in, Spatial coordinates of the original point cloud points and These are the hardness index and humidity estimates for the corresponding contact areas.
[0051] Through the above processing, an attribute-enhanced point cloud containing spatial geometric information and physical attribute information is formed. The attribute-enhanced point cloud is continuously updated in subsequent time steps and serves as the basic data input for calculating the geometric rate of change and the rate of change of vibration characteristics, supporting subsequent state determination and motion parameter calculation.
[0052] Furthermore, in the decision layer, the calculation steps for the geometric rate of change include: acquiring the attribute-enhanced point cloud of the contact area in adjacent time steps, calculating the displacement of each point in the attribute-enhanced point cloud between adjacent time steps, averaging the displacements to obtain the geometric rate of change; in the decision layer, the calculation steps for the vibration characteristic rate of change include: calculating the change in the resonant peak shift in adjacent time steps, calculating the average change in the hardness index and humidity estimate in adjacent time steps, weighting the above changes to obtain the vibration characteristic rate of change.
[0053] Specifically, the geometric rate of change is based on the relationship between adjacent time steps t and... The attribute enhancement point cloud of the contact region is calculated. The attribute enhancement point clouds of the contact regions corresponding to synchronization when adjacent are represented as follows: and ,in: ; ;in, Indicates the first The spatial coordinates of a point at a time step This represents the spatial coordinates of the corresponding point at the previous time step.
[0054] When calculating the geometric rate of change, point-level correspondences are performed on subsets of the point cloud in adjacent time steps. These point-level correspondences are determined based on spatial proximity relationships within a unified spatial coordinate system, i.e., for each time step... Each point in In time step The point with the smallest spatial distance in the point cloud is selected as the corresponding point. Then, the displacement vector of the corresponding point between adjacent time steps is calculated, and its displacement magnitude is defined as: After obtaining the displacement modulus of all corresponding points within the contact area, the displacement modulus is averaged to obtain the geometric rate of change. The calculation method is as follows: ;in, This indicates the number of points involved in the calculation within the enhanced point cloud of the contact region attributes. The geometric rate of change characterizes the degree of change in the overall spatial morphology of the contact region between phase time steps.
[0055] The rate of change of vibration characteristics is based on the change of acoustic vibration characteristics and their derived physical properties in adjacent time steps. The quantities are calculated. First, the change in the resonant peak shift between adjacent time steps is calculated as follows: ;in, Representing time awareness The resonance peak shifts. Indicates time step The resonance peak shifts.
[0056] Subsequently, the average change in hardness index relative to humidity estimate is calculated across adjacent time steps. (Change in hardness index) Defined as: Changes in humidity estimation Defined as: ;in, and Representing time steps Estimated hardness index and humidity for the lower contact area. This represents the hardness index and humidity estimate corresponding to the previous time step.
[0057] After obtaining the above changes, the changes in resonance peak shift, hardness index, and humidity estimation are weighted and calculated to obtain the vibration characteristic change rate. The calculation method is as follows: ;in, These are preset weighting coefficients used to balance the contribution ratio of different vibration characteristic changes in the overall rate of change.
[0058] Through the above calculation process, the geometric rate of change and the vibration characteristic rate of change are obtained on the same time scale and used as input quantities for subsequent state transition judgment of finite state automata, which are used to characterize the changes in the contact area in terms of spatial morphology and physical properties.
[0059] Furthermore, in the decision-making layer, the step of outputting preliminary action parameters according to the preset finite state automaton transition rules includes: presetting a finite state set containing multiple discrete states, each state corresponding to a cooking stage; using the geometric rate of change and the vibration characteristic rate of change as state transition inputs; executing the state transition according to the preset state transition rules; and outputting the preliminary action parameters corresponding to the current state after the state transition is completed.
[0060] Specifically, the decision-making layer models the states of the cooking process based on a finite state automaton model. The finite state automaton is represented as: ;in, Represents a finite set of states. Represents the set of inputs for state transitions. Represents the set of state transition rules. This indicates the mapping relationship between the state output and the action parameters.
[0061] Finite State Set It consists of multiple discrete states, each corresponding to a specific cooking stage. The cooking stages are divided according to the interaction between the robotic arm and the ingredients, and include at least a preparation stage, a contact processing stage, and a completion stage. Each discrete state is uniquely identified during system initialization and stored in the state management unit of the decision layer.
[0062] State transition input set At least the geometric rate of change With vibration characteristic change rate At each time step, the decision layer combines the currently calculated geometric rate of change with the vibration characteristic rate of change into a state transition input vector: ;in, Indicates the current time step.
[0063] The set of state transition rules T consists of several deterministic rules, each rule constraining the next state given a state and input conditions. The state transition rules are expressed as: in, Indicates the state at the current time step. This indicates the next state obtained after the corresponding transition condition is met. The transition condition is achieved by setting numerical ranges or thresholds for the geometric rate of change and the vibration characteristic rate of change. Each threshold is preset and stored before system deployment.
[0064] After completing the state transition determination, the decision-making layer bases its decisions on the current state. Mapping relationship between state output and action parameters The corresponding preliminary motion parameters are read from the input. These preliminary motion parameters include at least the preliminary force. Initial speed of motion and initial duration The motion parameters are given in definite values or definite function forms in each state, and serve as inputs for subsequent vibration feedback closed-loop haptic rendering and motion fine-tuning layers.
[0065] Through the modeling and operation of the aforementioned finite state automata, the decision layer completes the state update at each time step based on the combined input of the geometric rate of change and the vibration characteristic rate of change, and outputs the preliminary motion parameters corresponding to the current cooking stage, thereby realizing the staged control of the cooking process.
[0066] Furthermore, in the vibration feedback closed-loop haptic rendering and motion fine-tuning layer, the steps for correcting the initial motion parameters through active excitation and vibration response verification include: driving a bidirectional piezoelectric element with a low-amplitude excitation signal to apply micro-vibration pulses to the food in contact; simultaneously acquiring the vibration response spectrum of the food to the micro-vibration pulses; estimating the corresponding expected vibration spectrum based on the current hardness index and humidity; calculating the deviation between the vibration response spectrum and the expected vibration spectrum; and fine-tuning the initial motion parameters based on the deviation. The steps for fine-tuning the initial motion parameters based on the deviation include: calculating the force adjustment coefficient based on the deviation; proportionally calculating the force adjustment coefficient with the initial force to obtain the final force; adjusting the initial duration when the deviation is less than a preset threshold; and outputting the fine-tuned motion parameters as the final motion parameters.
[0067] Specifically, the active excitation is triggered by the decision-making level and occurs after the initial action parameters are output but before the formal action is executed. The low-amplitude excitation signal is a drive signal with a preset amplitude and duration, which is applied to the aforementioned bidirectional piezoelectric element through the control interface, causing it to apply micro-vibration pulses to the food in contact. The amplitude and frequency of the excitation signal are both limited to a range that does not cause macroscopic displacement of the food, ensuring that the excitation process is only used to obtain physical response information.
[0068] Simultaneously with the application of micro-vibration pulses, the bidirectional piezoelectric element is in signal acquisition mode, synchronously acquiring the vibration response signal induced by the micro-vibration pulses. After signal conditioning and frequency domain transformation, the vibration response signal yields the vibration response spectrum, which is expressed as follows: ;in, Represents frequency variable Indicates frequency The response amplitude at that location.
[0069] The expected vibration spectrum is generated based on the hardness index and humidity estimate corresponding to the current time step. Specifically, based on the hardness index... With humidity estimation And, combined with a preset spectral shape function, construct the expected vibration spectrum corresponding to the current physical state, expressed as:
[0070] in, This represents the spectral mapping function determined jointly by the hardness index and humidity estimation. The mapping function is determined and stored through benchmark experiments before system deployment.
[0071] After obtaining the vibration response spectrum and the expected vibration spectrum, the difference between the two is quantified, and the deviation is calculated. The deviation is used to characterize the degree of consistency between the current preliminary action parameters and the actual physical response, and it is calculated as follows: ;in, These represent the lowest and highest frequencies involved in the deviation calculation, respectively.
[0072] After calculating the deviation, the initial motion parameters are fine-tuned based on the deviation. First, the force adjustment coefficient is calculated based on the deviation. The calculation method is as follows: in, This is the preset adjustment coefficient. Then, the force adjustment coefficient is compared with the initial force. By performing proportional calculations, the final force is obtained: ;When the deviation degree Less than the preset threshold At that time, it is assumed that the current force parameters satisfy the physical response constraints, and under this condition, the initial duration is... Adjustments were made to obtain the final duration. The adjustment method involves linearly correcting the duration within a preset time step range.
[0073] After completing the above fine-tuning, the final force, initial velocity, and final duration are combined as the final motion parameters and output, which are then transmitted to the robotic arm controller for subsequent motion execution. Through this process, motion parameter correction based on active excitation and vibration response verification is achieved.
[0074] Furthermore, the steps of continuous closed-loop control include: sending the final motion parameters to the robotic arm controller and executing them; passing the attribute-enhanced point cloud data and motion parameters obtained during the execution process to the next cycle; recalculating the geometric rate of change and vibration characteristic rate of change based on the updated data in the next cycle; completing the calculation and decision processing in the edge computing unit, and synchronizing and storing the proportional coefficient, weighting coefficient and state transition rules through the cloud.
[0075] Specifically, in one implementation, the final motion parameters are generated after the vibration feedback closed-loop haptic rendering and motion fine-tuning are completed, and then sent to the robotic arm controller via a control interface. The final motion parameters include at least the final force. Final velocity and final duration The robotic arm controller generates corresponding control commands based on the final motion parameters and drives each joint of the robotic arm according to a preset control cycle to complete the corresponding cooking action.
[0076] During the robotic arm's actions, the perception layer continuously collects optical 3D point cloud data and acoustic vibration data related to the current action. After processing by the data fusion layer, a new attribute-enhanced point cloud is formed. The attribute-enhanced point cloud and the action parameters corresponding to the current time step are then passed to the next time step as input data for the next cycle's calculation. This data transfer process is performed under a unified time index to ensure data consistency across different cycles.
[0077] In the next cycle, the system recalculates the geometric rate of change and the vibration characteristic rate of change based on the updated attribute-enhanced point cloud and motion parameters. The geometric rate of change reflects the continuous changes in the spatial morphology of the contact area, while the vibration characteristic rate of change reflects the changes in the physical properties of the contact area over time. The calculation method for the rates of change is consistent with the aforementioned time step, thus ensuring the comparability of calculation results between adjacent cycles.
[0078] The geometric rate of change and the rate of change of vibration characteristics are used as new state transition inputs, which are then passed to the decision-making layer to execute the next state transition and update the action parameters within the framework of a finite state automaton. In this way, the system completes decision-making and control based on the latest sensing results at each time step, achieving continuous periodic updates.
[0079] The calculation of geometric rate of change, vibration characteristic rate of change, finite state automaton state transition, and motion parameter generation are all completed in the edge computing unit. The edge computing unit maintains low-latency communication connections with the sensing layer and the robotic arm controller to meet real-time control requirements.
[0080] Simultaneously, the system stores the proportional coefficients, weighting coefficients, and state transition rules of the finite state automaton used for calculation on a cloud server. The cloud server is used for unified management and version control of these parameters, and synchronizes the updated parameter configurations to the edge computing units. After receiving the synchronized parameters from the cloud, the edge computing units perform subsequent cycle calculations and decision-making processes according to the updated parameters.
[0081] Through the aforementioned continuous closed-loop control process, continuous iteration of perception, decision-making, and execution in the time dimension is achieved, enabling the robotic arm to complete multi-cycle motion execution and parameter updates during the cooking process based on real-time updated attribute enhancement point cloud and vibration feature changes.
[0082] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A real-time 3D perception and decision-making service platform for cooking robots, characterized in that, include: The perception layer is used to simultaneously collect optical 3D point cloud data of the cooking scene as well as acoustic vibration data generated by the active contact or excitation of the robotic arm. The data fusion layer is used to extract vibration features from the acoustic vibration data and map the vibration features to the point cloud region corresponding to the contact area to form an attribute-enhanced point cloud containing physical attributes; the decision layer is used to output preliminary action parameters based on the geometric rate of change and the vibration feature rate of change, according to the preset finite state automaton transition rules; the vibration feedback closed-loop haptic rendering and action fine-tuning layer is used to correct the preliminary action parameters through active excitation and vibration response verification to obtain the final action parameters; The robotic arm controller is used to execute cooking actions according to the final motion parameters and feed the execution results back to the next cycle, forming a continuous closed-loop control.
2. The real-time 3D perception and decision-making service platform for cooking robots according to claim 1, characterized in that, In the perception layer, the steps of simultaneously acquiring optical 3D point cloud data of the cooking scene and acoustic vibration data generated by the active contact or excitation of the robotic arm include: acquiring three-dimensional point cloud data of the cooking scene using a structured light sensor or a time-of-flight sensor; setting bidirectional piezoelectric elements on the end effector of the robotic arm and the preset cooking utensils; acquiring time-domain vibration signals through the bidirectional piezoelectric elements when the robotic arm performs contact, tapping, or active excitation actions; and performing frequency domain conversion on the time-domain vibration signals to obtain the corresponding vibration spectrum data.
3. The real-time 3D perception and decision-making service platform for cooking robots according to claim 1, characterized in that, In the data fusion layer, the step of extracting vibration features from the acoustic vibration data includes: determining the frequency component with the largest energy in the vibration spectrum as the main resonant peak frequency; calculating the vibration attenuation rate based on the trend of high-frequency energy changing with frequency in the vibration spectrum; comparing the current main resonant peak frequency with the resonant peak frequency under the reference state to obtain the resonant peak shift; and outputting the main resonant peak frequency, attenuation rate, and resonant peak shift as acoustic vibration features.
4. The real-time 3D perception and decision-making service platform for cooking robots according to claim 1, characterized in that, In the data fusion layer, the step of forming an attribute-enhanced point cloud containing physical attributes includes: determining a subset of point clouds corresponding to the contact area of the robotic arm; calculating a hardness index based on acoustic vibration characteristics for each point in the subset of point clouds; calculating a humidity estimate based on the energy attenuation of the high-frequency band of the vibration spectrum; and writing the hardness index and humidity estimate as additional physical attributes into the corresponding point cloud points to form an attribute-enhanced point cloud.
5. A real-time 3D perception and decision-making service platform for cooking robots according to claim 4, characterized in that, The steps for calculating the hardness index based on acoustic vibration characteristics include: obtaining the current main resonance peak frequency; obtaining the reference resonance peak frequency under the reference reference state; and calculating the hardness index based on the ratio of the two and a preset proportionality coefficient. The steps for calculating the humidity estimation include: calculating the energy attenuation rate of the high-frequency band of the vibration spectrum; and calculating the humidity estimate based on the energy attenuation rate and a preset proportionality coefficient.
6. A real-time 3D perception and decision-making service platform for cooking robots according to claim 1, characterized in that, In the decision layer, the calculation steps for the geometric change rate include: acquiring the attribute-enhanced point cloud of the contact area in adjacent time steps, calculating the displacement of each point in the attribute-enhanced point cloud between adjacent time steps, averaging the displacement to obtain the geometric change rate; in the decision layer, the calculation steps for the vibration characteristic change rate include: calculating the change in resonance peak shift in adjacent time steps, calculating the average change in hardness index and humidity estimate in adjacent time steps, weighting the above changes to obtain the vibration characteristic change rate.
7. A real-time 3D perception and decision-making service platform for cooking robots according to claim 1, characterized in that, In the decision-making layer, the step of outputting preliminary action parameters according to the preset finite state automaton transition rules includes: presetting a finite state set containing multiple discrete states, each state corresponding to a cooking stage; using the geometric rate of change and the vibration characteristic rate of change as state transition inputs; performing state transition according to the preset state transition rules; and outputting preliminary action parameters corresponding to the current state after the state transition is completed.
8. A real-time 3D perception and decision-making service platform for cooking robots according to claim 1, characterized in that, In the vibration feedback closed-loop tactile rendering and motion fine-tuning layer, the step of correcting the preliminary motion parameters through active excitation and vibration response verification includes: driving a bidirectional piezoelectric element with a low-amplitude excitation signal to apply micro-vibration pulses to the food in contact; synchronously acquiring the vibration response spectrum of the food to the micro-vibration pulses; estimating the corresponding expected vibration spectrum based on the current hardness index and humidity; calculating the deviation between the vibration response spectrum and the expected vibration spectrum; and fine-tuning the preliminary motion parameters based on the deviation.
9. A real-time 3D perception and decision-making service platform for cooking robots according to claim 8, characterized in that, The step of fine-tuning the initial motion parameters based on the deviation includes: calculating the force adjustment coefficient based on the deviation; performing a proportional calculation between the force adjustment coefficient and the initial force to obtain the final force; adjusting the initial duration when the deviation is less than a preset threshold; and outputting the fine-tuned motion parameters as the final motion parameters.
10. A real-time 3D perception and decision-making service platform for cooking robots according to claim 1, characterized in that, The steps of the continuous closed-loop control include: sending the final motion parameters to the robotic arm controller and executing them; transmitting the attribute-enhanced point cloud data and motion parameters obtained during the execution process to the next cycle; recalculating the geometric rate of change and the vibration characteristic rate of change based on the updated data in the next cycle; completing the calculation and decision processing in the edge computing unit, and synchronizing and storing the proportional coefficient, weighting coefficient and state transition rules through the cloud.