Humanoid robot body intelligent cooperative control system based on multi-mode perception fusion

Abstract

The invention relates to the technical field of robot cooperative control, in particular to a humanoid robot body intelligent cooperative control system based on multi-modal sensing fusion, and the system comprises a multi-modal sensing fusion module which recognizes a target boundary position, analyzes a pressure change, and combines with a posture to extract audio features to generate an environment sensing graph; the dynamic time sequence adjustment module optimizes an action rhythm adjustment detail generation execution plan, the cross-modal behavior correction module corrects an offset optimization track generation coordination sequence, the task priority distribution module analyzes task distribution to generate an execution list, and the time sequence conflict correction module optimizes a path adjustment conflict generation coordination path. According to the method, an environment perception graph is constructed through matching and fusion of multi-source perception data, the action sequence and interval are dynamically adjusted to optimize an execution chain, trajectory offset is corrected to improve action precision, nearest response and load balancing are achieved through real-time task allocation, conflict blocking is reduced through path rearrangement and time sequence coordination, and a perception decision execution closed loop is formed; and the identification precision and the cooperation efficiency are improved.

Description

Technical Field

[0001] This invention relates to the field of robot collaborative control technology, and in particular to a humanoid robot embodied intelligent collaborative control system based on multimodal perception fusion. Background Technology

[0002] The field of robot cooperative control technology encompasses research directions aimed at achieving sensory communication, motion coordination, task allocation, and information fusion among multiple robots operating in the same environment. The core content involves multi-dimensional acquisition and interaction of environmental information, status data, and behavioral commands to enable collaborative operation between individual robots and groups in dynamic environments. Its research primarily involves human-robot interaction methods, motion control strategies, sensory fusion mechanisms, and the construction of autonomous decision-making systems. It focuses on the issues of motion synchronization, path coordination, resource sharing, and dynamic optimization for different types of robots performing complex tasks, thus forming the fundamental technical system for cooperative control in multi-agent systems.

[0003] Among them, the humanoid robot embodied intelligent collaborative control system based on multimodal perception fusion refers to a collaborative control mechanism for humanoid robots under multi-channel perception conditions by integrating multiple sensory information such as vision, hearing, touch, and motion posture. Addressing the problems of perception coordination, behavior matching, and motion planning of humanoid robots in group operations, it employs cross-modal data mapping, state association recognition, and action intent calculation to transform multi-source sensory information into embodied behavior control signals, achieving information sharing and action coordination among multiple individual robots to ensure consistency between the perception layer, cognitive layer, and execution layer in the control process.

[0004] Existing technologies suffer from problems such as signal asynchrony and state update lag during perception fusion, resulting in untimely response to environmental changes; action execution relies on a fixed rhythm and lacks adaptive adjustment, which easily leads to error accumulation; trajectory correction is insufficient, and position offset continues to expand; task allocation does not consider spatial distribution and real-time load, resulting in uneven resource utilization; group path coordination is lacking, and task overlap is frequent, causing execution delays and decreased collaboration efficiency. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by proposing a humanoid robot embodied intelligent collaborative control system based on multimodal perception fusion.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a humanoid robot embodied intelligent collaborative control system based on multimodal perception fusion, the system comprising, The multimodal perception fusion module visually identifies the boundaries and positions of target objects, extracts force and pressure change features, analyzes tilt angles and rotation amplitudes by combining posture information, extracts frequency wave points and amplitude changes in audio, integrates the temporal and spatial distribution of multimodal data, and generates an environmental perception map. The dynamic timing adjustment module, based on the environmental perception map, analyzes the trajectory of the target object's state change, extracts the execution time nodes and sequence of the robot's actions, optimizes the time allocation and execution rhythm between actions, adjusts the execution priority and revises the robot's action chain, and generates an optimized action execution plan. The cross-modal behavior correction module calls the optimized action execution plan, extracts the offset value and action amplitude in the robot's behavior trajectory, analyzes the difference between the dynamic position and trajectory of the target object, adjusts the offset direction and amplitude, optimizes the trajectory accuracy, corrects abnormal actions, and generates a sequence of coordinated behavior actions. The task priority allocation module extracts the robot's current task status and position distribution based on the behavior coordination action sequence, analyzes the distance and task matching between the robot and the task target object, prioritizes the allocation of key tasks to the robot with the highest matching degree, adjusts the robot's task content and order, and generates a robot task execution list. The timing conflict correction module calls the robot task execution list, extracts the spatial distribution and time nodes in the task path, analyzes the path intersection domain and time conflict, adjusts the task order and path distribution of the conflict points, optimizes the timing distribution and spatial coordination of task execution, and generates a robot collaborative execution path.

[0007] As a further aspect of the present invention, The environmental perception map includes target state characteristics, environmental dynamic parameters, and spatiotemporal distribution information; The optimized action execution plan includes action timing parameters, rhythm coordination factors, and priority configuration; The behavior coordination action sequence includes trajectory correction parameters, amplitude adjustment factors, and action continuity constraints; The robot task execution list includes a task matching index, a division of labor scheduling scheme, and an execution path mapping. The robot collaborative execution path includes a timing coordination diagram, a spatial obstacle avoidance model, and group synchronization parameters.

[0008] As a further aspect of the present invention, the multimodal perception fusion module includes: The visual recognition submodule acquires the target object image and boundary region from the visual input, detects the continuity and morphological stability of the contour line, filters the region with complete boundary and marks the target position range, detects pose changes for adjacent images and extracts tilt angle and rotation direction information, establishes the pose change trajectory of the object, and generates boundary pose information. The force fusion submodule calls the pressure sensing matrix of the contact surface based on the boundary posture information, detects the force change and distribution characteristics of the contact area, determines the correspondence between each contact point and the posture change direction, summarizes the force differences between different contact points, determines the pressure change trend and records its extension in the time series, and generates contact strain characteristics. The audio coordination submodule calls the frequency and amplitude sequence of the audio signal according to the contact strain characteristics, detects the frequency fluctuation range and amplitude change range, determines the time synchronization relationship between audio changes and force distribution, integrates the dynamic correspondence trajectory of vision, force perception and audio, summarizes the coordinated distribution of each perception item in time and space, and generates an environmental perception map.

[0009] As a further aspect of the present invention, the dynamic timing adjustment module includes: The temporal extraction submodule calls the environmental perception map to identify the posture and position changes of the target object in a continuous time period, records the correspondence between robot actions and state changes, marks the start and end nodes of each action, sorts out the sequence of events in the time series, forms the time mapping path of the action chain, and generates the action temporal structure. The rhythm sequencing submodule reads the time sequence of each action node based on the action timing structure, detects action connection segments, adjusts the pause duration between actions according to the continuity of state changes, rearranges the time rhythm, revises the connection sequence of continuous actions, forms an execution chain with consistent rhythm, and generates a rhythm connection sequence. The action priority submodule analyzes the overall structure of the action chain based on the rhythm connection sequence, identifies key nodes that affect the connection of actions in the continuous time sequence, adjusts the priority of actions according to their occurrence position and associated frequency, improves the connection details and triggering order between adjacent actions, integrates the updated action chain, and generates an optimized action execution plan.

[0010] As a further aspect of the present invention, the cross-modal behavior correction module includes: The offset detection submodule, based on the optimized action execution plan, extracts the robot's motion trajectory during execution, observes the changes in the running path and direction of each action segment, identifies trajectory nodes that deviate from the target position, and records their distribution in the action chain. By comparing the correspondence between the trajectory and the target position, it identifies the action segments where offsets have occurred and generates offset distribution features.

[0011] The trajectory correction submodule reads the spatial direction information of the abnormal trajectory based on the offset distribution characteristics, analyzes the coherence relationship between nodes, and determines the path segments caused by the offset. It then reorganizes the trajectory connection sequence according to the offset direction, adjusts the extension direction of the motion segments, and makes the motion path refit the target trajectory, generating a path correction result.

[0012] The motion coordination submodule traces back the overall rhythm of the motion chain based on the path correction results, sorts out the connection and transition relationship between each motion, adjusts the rhythm sequence and amplitude changes of the motion, so that the corrected path maintains continuity and coordination in time, integrates trajectory and motion rhythm information, and generates a behavior coordination motion sequence.

[0013] As a further aspect of the present invention, the task priority allocation module includes: The task recognition submodule calls the behavior coordination action sequence, reads the robot's current task status and spatial position, sorts out the correspondence between each task node and the target object, marks the tasks being executed and those to be assigned, integrates the distribution of tasks in the scene, establishes a correspondence record between tasks and robot operating status, and generates a task status distribution map. The distance matching submodule extracts the positional relationship between the robot and the task target based on the task state distribution map, analyzes the compatibility between task requirements and robot functions, identifies matching objects that are close in space, judges the task affiliation of robots that meet the execution conditions, determines the order of task allocation, and generates task matching results. The task allocation and coordination submodule summarizes the overall task distribution of the robot based on the task matching results, checks the connection and execution order of the task chain, identifies the overlapping or gap positions between tasks, sorts out and adjusts the sequence of uncoordinated task segments, re-plans the task allocation structure, integrates the updated execution sequence, and generates a robot task execution list.

[0014] As a further aspect of the present invention, the timing conflict correction module includes: The path extraction submodule calls the robot task execution list, reads the spatial distribution and time nodes of each task path, sorts out the movement direction and stage connection of different robots in the execution process, marks the intersection of the starting point and ending point of the path, integrates the correspondence between time and space, forms the overall temporal structure of task operation, and generates a path temporal distribution map. The conflict identification submodule compares the running trajectories of multiple task paths based on the path time sequence distribution map, observes the temporal overlap and spatial intersection of task nodes, identifies task segments that appear in the same area at the same time, summarizes the locations and corresponding time periods of conflicts, organizes the distribution relationship of conflict nodes, and generates a task conflict list. The sequence adjustment submodule sorts out the time sequence of task execution according to the task conflict list, locates the key nodes affected, rearranges the connection rhythm and path direction between tasks, adjusts the execution order to reduce time overlap, reorganizes the action sequence of multiple robots in the shared space, integrates the updated task time sequence relationship, and generates a robot collaborative execution path.

[0015] Compared with the prior art, the advantages and positive effects of the present invention are as follows: In this invention, an environmental perception map is constructed through spatiotemporal matching and information fusion of multi-source sensing data, enabling continuous identification of target states and environmental changes. The sequence and interval of actions are dynamically adjusted based on state trajectories to form a rhythmically coordinated execution chain, and action accuracy and continuity are maintained through trajectory offset and anomaly point correction. Task allocation is based on real-time matching of location and state, achieving proximity response and load balancing; during the group execution phase, conflicts and blockages are reduced through path rearrangement and timing coordination. Overall, a closed-loop connection between perception, decision-making, and execution is formed, improving environmental recognition accuracy, action stability, and collaborative efficiency. Attached Figure Description

[0016] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a flowchart illustrating the acquisition process of the multimodal perception fusion module of the present invention. Figure 3 This is a flowchart illustrating the acquisition process of the dynamic timing adjustment module of the present invention. Figure 4 This is a flowchart illustrating the acquisition process of the cross-modal behavior correction module of the present invention. Figure 5 This is a flowchart illustrating the acquisition process of the task priority allocation module of the present invention. Figure 6 This is a flowchart illustrating the acquisition process of the timing conflict correction module of the present invention. Detailed Implementation

[0017] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0018] In this embodiment of the invention, sometimes the subscript such as W1 is written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0019] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0020] Please see Figure 1 This invention provides a technical solution: a humanoid robot embodied intelligent collaborative control system based on multimodal perception fusion, the system comprising: The multimodal perception fusion module visually identifies the boundary and position of the target object, obtains the pressure change characteristics of the force contact area, analyzes the trend of tilt angle change and rotation amplitude by combining posture information, extracts frequency fluctuation points and amplitude change ranges from audio data, compares the distribution of each perception data in time and space, and integrates them into target state features and environmental dynamic information to generate an environmental perception map. The dynamic timing adjustment module calls the environmental perception map, analyzes the trajectory of the target object's state change, extracts the execution time nodes and sequence of the robot's actions, reorders the action time nodes based on the trend of the object's state change, optimizes the time interval and execution rhythm between actions, and on this basis, revises the priority of the robot's action chain and adjusts the details to generate an optimized action execution plan. The cross-modal behavior correction module extracts the offset amplitude and motion change range in the robot's behavior trajectory based on the optimized motion execution plan, analyzes the difference between the offset value and the dynamic position of the target object, filters out abnormal points in the motion trajectory and makes adjustments, corrects the offset direction and amplitude, optimizes the change law of motion amplitude, makes the behavior trajectory more accurate, and generates a sequence of coordinated behavior actions. The task priority allocation module calls the behavior coordination action sequence, extracts the robot's current task status and position distribution, analyzes the distance and task matching between the robot and the task target object, prioritizes the allocation of robots that are closer and meet the task requirements to execute key tasks, and adjusts the task division and execution order of other robots to generate a robot task execution list. The timing conflict correction module calls the robot task execution list, extracts the spatial distribution and time nodes in the task path, analyzes the path intersection area and time conflict point, adjusts the task order and path direction of the conflict node, optimizes the timing distribution and spatial coordination of group task execution, and generates a robot collaborative execution path.

[0021] The environmental perception map includes target state characteristics, environmental dynamic parameters, and spatiotemporal distribution information. The optimized action execution plan includes action timing parameters, rhythm coordination factors, and priority configuration. The behavior coordination action sequence includes trajectory correction parameters, amplitude adjustment factors, and action continuity constraints. The robot task execution list includes task matching index, division of labor scheduling scheme, and execution path mapping. The robot collaborative execution path includes timing coordination map, spatial obstacle avoidance model, and group synchronization parameters.

[0022] Please see Figure 2 The multimodal perception fusion module includes: The visual recognition submodule acquires the target object image and boundary region from the visual input, detects the continuity and morphological stability of the contour line, filters the region with complete boundary and marks the target position range, detects pose changes for adjacent images and extracts tilt angle and rotation direction information, establishes the pose change trajectory of the object, and generates boundary pose information. To acquire the target object image and boundary region from the visual input, specifically, to retrieve the image from a 1920*1080 resolution visual sensor. The captured frame image data is processed by grayscale conversion and Gaussian blurring, followed by edge detection to extract pixel gradients, and a high threshold is set. low threshold This process forms an initial set of pixels for the contour line. Then, iterates through the pixel set, detecting the coordinate distance between adjacent pixels in the contour line. If the distance is greater than 3 pixels, it is marked as a breakpoint. The coordinates are then counted. Time outline Number of breakpoints And calculate its total length. Pixels, simultaneously retrieve Outline of time ( , (pixels), calculate the rate of change of shape. Set a morphological stability threshold The threshold Based on the average (0.5%) rate of change of adjacent frame length in 1000 stable capture experiments, plus three standard deviations ( ) settings, i.e. ,because Determine morphological stability and set a boundary integrity threshold. The threshold Based on 95% of the minimum outline length (1450 pixels) of 100 complete circuit board samples from historical data, i.e. Pixel, current Therefore, this region was selected, and its position range was determined to be [100, 200, 350, 400] using the minimum bounding rectangle method, and recorded. Attitude angle at moment Subsequently, retrieve Time (e.g.) The acquired frame images are used to repeat the above detection and calibration process to obtain... The position range [102, 205, 352, 405] and attitude angle at any given time Calculate the change in tilt angle To determine the direction of rotation, because The rotation direction is set to clockwise. and The attitude angle [15.5, 15.8] and rotation direction [clockwise] at time 15.5 are stored in the attitude sequence. This is used to establish the trajectory of the object's attitude change and generate boundary attitude information.

[0023] The force fusion submodule calls the pressure sensing matrix of the contact surface based on the boundary posture information, detects the force change and distribution characteristics of the contact area, determines the correspondence between each contact point and the posture change direction, summarizes the force differences between different contact points, determines the pressure change trend and records its extension in the time series, and generates contact strain characteristics. The pressure sensing matrix of the contact surface is retrieved based on the boundary attitude information. Specifically, this involves calling... and Boundary pose information at time ( , (Rotation direction = clockwise), and accesses 5 pressure sensors (P1-P5) installed on the inside of the robotic arm's gripper fingers. and Pressure readings at various times are used to construct a pressure sensing matrix. At time [time], the pressure readings (P1-P5) of each sensor are [50.5, 55.2, 50.1, 52.0, 51.8] kPa, respectively. At time [time], the readings are updated to [50.8, 55.3, 51.5, 51.0, 53.5] kPa. The force change in the contact area is detected and calculated. to Pressure changes of each sensor at any time ,get kPa, kPa, kPa, kPa, kPa, setting the pressure change baseline value This benchmark value The maximum noise reading of the sensor within 10 seconds under no-load conditions was used as a reference. kPa, change value Sensors P3, P4, and P5 are identified as valid force change points, and a high-amplitude change threshold is set. The threshold Based on the minimum pressure increment (1.0 kPa) required for stable gripping, i.e. The changes in P3 (1.4 kPa) and P5 (1.7 kPa) were classified as "high-amplitude" changes, P4 (-1.0 kPa) as a "medium-amplitude" change, and the changes in P1 and P2 as noise fluctuations. This determined the distribution characteristics as follows: the pressure at the lower part (P3) and the right part (P5) increased significantly, while the pressure at the left part (P4) decreased. The correspondence between each contact point and the direction of attitude change was determined. The rotation direction (clockwise) from the boundary attitude information was retrieved. Based on the robotic arm claw model, clockwise rotation would increase the pressure on P3 and P5, and decrease the pressure on P4. This expected change [P3+, P4-, P5+] was compared with the actual detected high-amplitude changes [P3+, P4-, P5+], and they were consistent. The force differences between different contact points were summarized, and the pressure difference between P3 and P1 was calculated. Calculate the pressure difference between P5 and P4 using kPa. kPa, determining the pressure change trend as the object is tilting clockwise, and recording this trend and key change values ​​( Recorded in time series In this process, contact strain characteristics are generated.

[0024] The audio coordination submodule calls the frequency and amplitude sequence of the audio signal based on the contact strain characteristics, detects the frequency fluctuation range and amplitude change range, determines the time synchronization relationship between audio changes and force distribution, integrates the dynamic correspondence trajectory of vision, force perception and audio, summarizes the coordinated distribution of each perception item in time and space, and generates an environmental perception map.

[0025] Based on the contact strain characteristics, the frequency and amplitude sequence of the audio signal are retrieved, specifically to lock the timestamp of a "high amplitude" change in pressure within the contact strain characteristics. And retrieve the microphone installed at the end of the robotic arm. An audio signal sequence was acquired within a 200ms time window (900ms to 1300ms). A Fast Fourier Transform (FFT) was performed on this sequence to obtain frequency and amplitude data. For example, in... At time (1000ms), the force perception was stable (P3=50.1kPa, P5=51.8kPa), the audio amplitude was 10.5dB, and the dominant frequency was 50Hz. At time (1100ms), the force sensation is characterized by a sudden increase in pressure (P3=51.5kPa, P5=53.5kPa), with an audio amplitude reaching 25.2dB and a dominant frequency of 3500Hz. At time (1150ms), the pressure stabilized with an amplitude of 11.0dB and a main frequency of 55Hz. The frequency fluctuation range and amplitude change range were detected, and an amplitude change threshold was set. The threshold based on The average background amplitude (10.7 dB) over the previous 1000 ms plus 5 standard deviations ( dB) setting, i.e. dB, detection The amplitude at time 25.2 dB is due to ,determination A sudden change in amplitude occurs at any time, and a reference range for frequency fluctuation is set simultaneously. This interval Based on the power frequency (50Hz) and its harmonics (100Hz) during normal equipment operation, the frequency is set to [40Hz, 110Hz] for detection. The main frequency at that moment was 3500Hz, due to Far beyond Interval, determination The time is the point of frequency fluctuation, thus determining For the segment where the audio event occurs, determine the temporal synchronization relationship between the audio change and the force distribution, and retrieve the time of the force event occurrence. Audio event occurrence time Calculate the time difference Set the synchronization time window The window Based on a setting that doubles the data acquisition cycle (10ms) of the two sensors, i.e. ,because The system determines that changes in audio and force distribution are synchronized in time, integrating the dynamic correspondence between vision, force perception, and audio to form a coherent trajectory. Boundary pose information at time ( Rotation direction = clockwise), contact strain characteristics ( ) and audio features ( dB The data is correlated with each sensing item (Hz) and the temporal and spatial collaborative distribution of each sensing item is summarized. This set of collaborative data is then used to […]. Store the data in the database to generate an environmental awareness map.

[0026] Please see Figure 3 The dynamic timing adjustment module includes: The temporal extraction submodule calls the environmental perception map to identify the pose and position changes of the target object in a continuous time period, records the correspondence between robot actions and state changes, marks the start and end nodes of each action, sorts out the sequence of events in the time series, forms the time mapping path of the action chain, and generates the action temporal structure. Calling the environment-aware map, i.e., retrieval to Extracting environmental perception map data sequences within a time window target object pose With position , Time posture With strain kPa, extracted separately Time position ,calculate to Posture changes And set the attitude change threshold. The threshold Based on the 99th percentile jitter value of the attitude sensor readings in 1000 stable grasping experiments, i.e. ,because Determine that an attitude adjustment has occurred, and simultaneously calculate... (at this time )to Z-axis position change Pixels, set Z-axis increase threshold The threshold The minimum Z-axis displacement (-45 pixels) based on 100 successful lift actions is set as follows: Pixels, due to Upon determining that a lifting action has occurred, and based on the corresponding relationship, the "adjust grip" (A1), "stabilize grip" (A2), and "lift" (A3) actions in the robot's action library are respectively matched with... Moment (sudden increase in strain) Time (posture adjustment) and The state changes at time (Z-axis elevation) are correlated, and the start and end nodes of each action are marked based on the duration of the state change. A1 start and end nodes [ A2 start and end nodes[ A3 start and end nodes[ The sequence of events in the time series is sorted out. A1, A2, and A3 are ordered according to their starting times [1050, 1150, 1350] to form an action chain A1->A2->A3. This action chain A1->A2->A3 is then integrated with the corresponding time nodes [(A1, 1050, 1150), (A2, 1150, 1250), (A3, 1350, 1450)] to generate the action time sequence structure.

[0027] The rhythm sequencing submodule reads the time sequence of each action node based on the action time sequence structure, detects action connection segments, adjusts the pause duration between actions according to the continuity of state changes, rearranges the time rhythm, revises the connection sequence of continuous actions, forms an execution chain with consistent rhythm, and generates a rhythm connection sequence. Based on action timing structure Read the time sequence of nodes A1, A2, and A3, detect the action connection segments, and calculate the pause duration between A1 and A2. Calculate the pause duration between A2 and A3. The pause duration between actions is adjusted based on the continuity of state changes. This adjustment refers to continuity rule R1: if "Steady Grip" (A2) is completed and the environmental perception map is in... The object's state is "stable" (defined as the rate of change of strain) as reported by time feedback. Then the "lifting" (A3) action must be performed within the specified time frame. Execution continues within the same period. Based on the median value (8ms) from 100 system latency tests, plus a 2ms buffer period, the setting is... Query The current state is "stable". The rule R1 was deemed violated, and adjustments were made. The timing is rearranged to 10ms. The time nodes [1050, 1150] and [1150, 1250] of A1 and A2 remain unchanged, while the start time of A3 is... Duration of A3 The new end time of A3 Revise the sequence of consecutive actions, maintaining the order of A1->A2->A3 while updating the time nodes to form a consistent execution chain. Generate rhythmic sequence.

[0028] The Action Priority submodule analyzes the overall structure of the action chain based on the rhythm connection sequence, identifies key nodes that affect the connection of actions in continuous time sequence, adjusts the priority of actions according to their occurrence position and associated frequency, improves the connection details and triggering order between adjacent actions, integrates the updated action chain, and generates an optimized action execution plan.

[0029] According to the rhythmic sequence Analyzing the overall structure of the action chain, which is a "Grasp-Lift" linear sequence, we identified key nodes affecting action continuity in a continuous time sequence. Key nodes were defined as actions that occurred more than 90% of the time in 1000 historical grasping tasks. The frequency of A1 (adjusting the grip) was statistically analyzed to be 30%. A2 (Stable Grip) frequency 95% A3 (mentioned) frequency 98% ( A2 and A3 are identified as key nodes. Based on their occurrence position and frequency of association, the priority of actions is adjusted, and priority scores are calculated. ,in Used as a key node identifier (0 or 1). The time sequence is (A1=1, A2=2, A3=3). The total number of actions is 3, and the weight is... (Crucial) and (Temporal reversal) is set based on 50 optimization experiments. and Calculate the A1 score Calculate the A2 score Calculate the A3 score ,according to Arrange the values ​​in descending order to obtain the new priority order A2(11)->A3(10)->A1(2). Improve the connection details and triggering order between adjacent actions, integrate the updated action chain, and store it in a structured manner as follows: the priority of action A1 (adjust grip) is 3, the start time is 1050ms, the end time is 1150ms, and the triggering condition is... The action is "unstable," with priority 1 for action A2 (Steady Grip), start time 1150ms, end time 1250ms, and trigger condition as follows. Action A3 (lift) has a priority of 2, a start time of 1260ms, an end time of 1360ms, and a trigger condition of... and To ensure "stability", an optimized action execution plan is generated.

[0030] Please see Figure 4 The cross-modal behavior correction module includes: The offset detection submodule extracts the robot's motion trajectory during execution based on the optimized motion execution plan, observes the changes in the running path and direction of each action segment, identifies trajectory nodes that deviate from the target position, and records their distribution in the action chain. By comparing the correspondence between the trajectory and the target position, it identifies the action segments where offsets have occurred and generates offset distribution features.

[0031] Based on the optimized action execution plan, extract the preset target path point sequence. With the corresponding time node sequence And collect data in real time from the robot motion controller. The actual motion trajectory at the corresponding node (Include actual coordinate points ), traversal to ,calculate Middle adjacent nodes and The actual running direction vector between and with The planned direction vector Compare, and simultaneously iterate. to ,calculate Time planning points With practical point Spatial offset distance Set the offset determination threshold ,Should The determination is based on historical similar tasks (e.g., 500 repeated crawls). Statistical 90th percentile And combined with the minimum permissible clearance between the end effector and the target. ,Pick and The smaller value in is used as ,Will and Comparing them one by one, when At that time, the judgment For the offset node, index its time. and offset Record to offset node list ,analyze Using time indexes, filter out the set of nodes with consecutive index values ​​(e.g., index). ), and the actual nodes corresponding to this set Identified as an offset action segment Generate offset distribution features.

[0032] The trajectory correction submodule reads the spatial orientation information of abnormal trajectories based on offset distribution characteristics, analyzes the coherence relationship between nodes, and determines the path segments caused by the offset. It then reorganizes the trajectory connection sequence according to the offset direction, adjusts the extension direction of the motion segments, and re-fits the motion path to the target trajectory, generating a path correction result.

[0033] Based on the offset distribution characteristics, the offset action segment is invoked. and its corresponding actual node set Planned node set and time node set Read and Coordinates, calculate all nodes within the segment and coordinate difference vector And by summing and averaging the difference vectors, we obtain Average offset vector of the segment This vector represents the spatial direction of the abnormal trajectory, and at the same time, analysis Adjacent time nodes (e.g.) (Time interval) Set a time continuity threshold , (For example ) is set to the motion control sampling period (e.g. )of times (e.g.) ),in The value was calibrated experimentally to make If the data transmission jitter is greater than the data transmission jitter and less than the minimum stall response time, all data within the segment will be affected. and Compare, if all All less than Then determine The nodes within form a time-coherent path segment, according to Set correction factor ( ), Value based on average offset With offset threshold ratio The ratio is determined dynamically; the larger the ratio, the better. The closer ,Will Multiply Obtain the correction vector traversal The actual nodes in ,implement The operation yields the corrected nodes. The operation was adjusted. Compared to The spatial location was used to redefine the corrected path segment. The extension direction generates path correction results.

[0034] The motion coordination submodule traces back the overall rhythm of the motion chain based on the path correction results, sorts out the connection and transition relationship between each action, adjusts the rhythm sequence and amplitude changes of the actions, so that the corrected path maintains continuity and coordination in time, integrates trajectory and motion rhythm information, and generates a behavior coordination motion sequence.

[0035] Based on the path correction results, the corrected trajectory is invoked. (Include (corrected nodes) and the original time series , back to Section entrance ( arrive (connection) and export ( arrive (transition), calculate the spatial distance at the connection point. With time interval Obtain entry running speed And calculate the velocity within the section in sequence. (For example and (between) and export speed ,analyze , , Rate of change of sequence (Acceleration), set the acceleration smoothing threshold. , The setting refers to the maximum permissible acceleration corresponding to the rated torque of the joint motor. and take (For example ), ( ) is the safety factor, and its value (e.g.) Calibration was achieved through 100 acceleration and deceleration tests under different loads to ensure that the operational impact was below the stable threshold. The calculated values ​​were then used to calibrate the system. Absolute value and In comparison, if Then determine When the rhythm of the movement (i.e., the time interval) does not match the amplitude change, maintain... Spatial position No change, adjustment The time points involved (e.g.) or ), such as extending the corresponding time interval (Right now , ), so that the adjusted acceleration and will All subsequent time points Translate sequentially This forms the corrected time series. The corrected trajectory With the corrected time series (If no adjustment is triggered, then) Integrate and generate a sequence of coordinated actions.

[0036] Please see Figure 5 The task priority allocation module includes: The task recognition submodule calls the behavior coordination action sequence, reads the robot's current task status and spatial position, sorts out the correspondence between each task node and the target object, marks the tasks being executed and those to be assigned, integrates the distribution of tasks in the scene, establishes a correspondence record between tasks and robot running status, and generates a task status distribution map. Invoke the behavior coordination action sequence to extract the current spatial position of all robots (e.g., R1, R2, R3) in the environment. and task status codes (0 indicates "idle", 1 indicates "running"), and retrieve the central task list. This list contains the target locations for all tasks (e.g., T1, T2, T3, T4). Dependencies of preceding tasks Iterate through all the robots, for The robot (e.g., R2) retrieves its current task (e.g., T1) in Lieutenant General T1's mission status Marked as 1 ("In Progress") and associated with R2, the rest Robots (R1, R3) are marked as idle. Then, the process is iterated through... All tasks in the process, check Analyze the tasks not marked as 1. ,like If not empty (e.g., T2 depends on T1), then query The state of task (T1) is due to ("In Progress") instead of 2 ("Completed"), therefore... Marked as 3 ("blocked"), if If it is empty (e.g., T3, T4), then Marked as 0 ("Pending Assignment"), integrate the above information to establish a corresponding record between tasks and robot operating status (R1 idle; R2 in progress). Execute T1; R3 is idle; T1 is executed by R2; T2 is blocked by T1; T3 and T4 are pending allocation), generate a task status distribution diagram.

[0037] The distance matching submodule extracts the positional relationship between the robot and the task target based on the task state distribution map, analyzes the adaptability between task requirements and robot functions, identifies matching objects that are close in space, judges the task affiliation of robots that meet the execution conditions, determines the order of task allocation, and generates task matching results. Based on the task state distribution map, extract the list of idle robots with a state code of 0. (Including R1 in) and R3 in (and a list of tasks to be assigned with a status code of 0) (Including T3) and T4 in ), retrieve robot capability library (For example, R1 has a {Grip, 10kg load} capacity; R3 has a {Inspect, 2kg load} capacity) and Task requirements (For example, T3 requirement {Grip, 8kg load}; T4 requirement {Inspect, 0kg load}), perform a compatibility analysis, and... and Cross-alignment, for example (R1, T3), R1's Satisfying T3 (Grip function matched and 10kg > 8kg), judgment (Adaptation), for example (R3, T3), R3's Not meeting T3 requirements (Functional mismatch and 2kg < 8kg), judgment (R1, T4) is not a match, (R3, T4) is a match, filter out all The fit Then, calculation Each pair Euclidean distance between (For example and Set spatial distance threshold ,Should The settings are based on the robot's average cruising speed. (e.g., 1.5 m / s) and maximum allowable response time The product of (e.g., 10s), that is This value of 15.0m is based on 1000 scheduling experiments, and 95% of the effective scheduling was completed within this distance. and In comparison, if If so, it is determined to be a matching object that is close in distance (assuming...). and (All are less than 15.0m). Based on the uniqueness of the function adaptation (R1 only adapts to T3, and R3 only adapts to T4), the task assignment is determined, T3 is assigned to R1, and T4 is assigned to R3, generating task matching results.

[0038] The task allocation and coordination submodule summarizes the overall task distribution of the robot based on the task matching results, checks the connection and execution order of the task chain, identifies the overlapping or gap positions between tasks, sorts out and adjusts the sequence of uncoordinated task segments, re-plans the task allocation structure, integrates the updated execution sequence, and generates a robot task execution list.

[0039] Based on the task matching results (R1->T3, R3->T4) and combined with the task state distribution diagram (R2 executes T1, T2 is blocked at T1), summarize the overall task distribution of the robot, check the task chain connection of T2, and the task state distribution of T2. ={T1}, T1 is executed by R2, and the system predicts that after R2 completes T1 (R2 will be in the...). Does it meet the task requirements of T2? By comparison and If it matches, add T2 to R2's task queue to form a task chain [T1->T2]. Then, check the intersections between tasks and analyze R1's planned path. ( -> ), R3 planning path ( -> ) and the planning path of R2 ( -> ),calculate and Do spatial intersections exist? If it exists Then calculate R3 to reach time Arrive with R2 time Set time conflict threshold (For example, 5.0s, this value is based on the robot size, speed, and braking distance safety redundancy calculation, for example) =2*( (, calibrated to 5.0s through testing), if If a conflict exists, then retrieve the necessary information. The task priorities defined in the code (e.g., T2 has a higher priority than T4) are used to reorder incompatible task segments, prioritizing tasks like R3 which have lower priority. Adjust (delay) to Then, the task allocation structure is re-planned, the updated execution sequence is integrated (R1 executes T3; R2 executes T1, then executes T2; R3 executes T4 with a delay), and a robot task execution list is generated.

[0040] Please see Figure 6 The timing conflict correction module includes: The path extraction submodule calls the robot task execution list, reads the spatial distribution and time nodes of each task path, sorts out the movement direction and stage connection of different robots in the execution process, marks the intersection of the starting point and ending point of the path, integrates the correspondence between time and space, forms the overall temporal structure of task operation, and generates a path temporal distribution map. Retrieve the robot task execution list and extract the path from R2 to execute T2. (starting point ,end The path of R3 executing T4 (starting point ,end ), and clearly stated that both R2 and R3 are in Start executing the task and set the path discretization step size. ,Will and Interpolation is performed on a series of path points. And based on the robot's average cruising speed (This value is the average speed obtained from 100 tests under different loads.) (And round down for calibration) Calculate the time nodes corresponding to each point. Through calculation ( )and ( The equation determines the two paths at... The intersections are integrated to obtain R2. arrive R3 is also in Upon reaching this point, the overall timing structure of the task execution is formed, and a path timing distribution map is generated.

[0041] The conflict identification submodule compares the running trajectories of multiple task paths based on the path time sequence distribution map, observes the temporal overlap and spatial intersection of task nodes, identifies task segments that appear in the same area at the same time, summarizes the locations and corresponding time periods of conflicts, organizes the distribution relationship of conflict nodes, and generates a task conflict list. Based on the path time series distribution map, a spatial safety threshold is set. and time safety threshold , The setting is based on the robot body radius. ( ) and maximum positioning error ( ), calculated as , The setting refers to the sampling period of the control system. ( ) and the robot's maximum braking time ( ), calculated as ,in (0.48s) is in The 95th percentile values ​​collected during 50 full-load braking tests were traversed. and Path point pairs Calculate their spacing ,like If the distance is 1.05m, then it is determined that the space interference zone has been entered. (For example (Nearby), read the corresponding time and (exist (All times are 4.73s), calculate the time difference. ,like (0.5s) (at) If the value is 0s, then a temporal overlap is determined to have occurred. When both spatial interference and temporal overlap are satisfied, the event is identified as a conflict. And calculate the transit time of the conflict zone. The conflict information is summarized as {location} Robots R2 / R3, conflict period Generate a list of task conflicts.

[0042] The sequence adjustment submodule sorts out the time sequence of task execution based on the task conflict list, locates the key nodes affected, rearranges the connection rhythm and path direction between tasks, adjusts the execution order to reduce time overlap, reorganizes the action sequence of multiple robots in the shared space, integrates the updated task time sequence relationship, and generates a collaborative execution path for robots.

[0043] According to the task conflict list (conflict) Involving R2, R3), call Priority of corresponding tasks (T2, T4) ( , (The numerical value is specified when the task is issued; the smaller the value, the higher the priority.) R2 (execute T2) is determined to have priority, and the timing of R3 (execute T4) needs to be adjusted. R2 is then positioned to clear the conflict zone. Time point And combined with time safety threshold Calculate R3 to allow entry earliest time This time is the time after R3 adjustment. At the new time point, calculate the startup latency required for R3 to execute T4. Apply this delay to the task chain of R3, adjusting its start time to... The timestamps of all subsequent path points are incremented sequentially. The finish time has been updated to It integrates the updated temporal relationships of all robots to generate a collaborative execution path for the robots.

[0044] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A somatic intelligent collaborative control system for humanoid robots based on multi-modal perception fusion, characterized in that, The system comprises: A multi-modal perception fusion module extracts force sensation pressure change characteristics, analyzes tilt angle and rotation amplitude in combination with posture information, extracts frequency wave points and amplitude changes in audio, integrates time and space distribution of multi-modal data, and generates an environment perception map by visually identifying boundaries and positions of target objects; A dynamic timing adjustment module analyzes state change trajectories of target objects, extracts execution time nodes and sequence of robot actions, optimizes time distribution and execution rhythm between actions, adjusts execution priority and revises robot action chains, and generates an optimized action execution plan based on the environment perception map; A cross-modal behavior correction module extracts offset values and action amplitudes in robot behavior trajectories, analyzes dynamic positions and trajectory differences of target objects, adjusts offset direction and amplitude, optimizes trajectory accuracy, corrects abnormal actions, and generates a behavior coordinated action sequence by calling the optimized action execution plan; A task priority allocation module extracts current task state and position distribution of robots, analyzes distance between robots and task target objects and task matching conditions, preferentially allocates key tasks to robots with the highest matching degree, adjusts task content and sequence of robots, and generates a robot task execution list based on the behavior coordinated action sequence.

2. The multi-modal perception fusion humanoid robot embodied intelligence collaborative control system according to claim 1, characterized in that: The environment perception map comprises target state characteristics, environmental dynamic parameters, and space-time distribution information; the optimized action execution plan comprises action timing parameters, rhythm coordination factors, and priority configurations; the behavior coordinated action sequence comprises trajectory correction parameters, amplitude adjustment factors, and action continuity constraints; and the robot task execution list comprises task matching indexes, division scheduling schemes, and execution path mappings. 3.The multi-modal perception fusion humanoid robot embodied intelligence collaborative control system of claim 1, wherein, The multi-modal perception fusion module comprises: A visual recognition submodule acquires target object images and boundary regions in visual input, detects continuity and shape stability of contour lines, filters regions with complete boundaries and labels target position ranges, detects posture changes and extracts tilt angle and rotation direction information for adjacent frames, establishes posture change trajectories of objects, and generates boundary posture information; A force sensation fusion submodule calls a pressure perception matrix of a contact surface based on the boundary posture information, detects force change and distribution characteristics of contact regions, determines corresponding relationships between each contact point and posture change direction, induces force differences between different contact points, determines pressure change trends and records their extension in time series, and generates contact strain characteristics; An audio coordination submodule calls frequency and amplitude sequences of audio signals according to the contact strain characteristics, detects frequency fluctuation intervals and amplitude change sections, determines time synchronization relationships between audio changes and force distribution, integrates dynamic corresponding trajectories of vision, force sensation, and audio, summarizes collaborative distribution of each perception item in time and space, and generates an environment perception map.

4. The multimodal perception fusion humanoid robot embodied intelligence collaborative control system according to claim 1, wherein, The dynamic timing adjustment module comprises: The time sequence extraction submodule calls the environment perception graph, identifies the posture and position change of the target object in the continuous time period, records the corresponding relationship between the robot action and the state change, labels the start and end nodes of each action, combs the sequence in the time sequence, forms the time mapping path of the action chain, and generates the action time sequence structure; The rhythm sequence submodule reads the time sequence of each action node based on the action time sequence structure, detects the action connection section, adjusts the pause time length between actions according to the continuity of the state change, rearranges the time rhythm, revises the connection sequence of the continuous actions, forms the execution chain with consistent rhythm, and generates the rhythm connection sequence; The action priority submodule analyzes the overall structure of the action chain according to the rhythm connection sequence, identifies the key nodes that affect the action connection in the continuous time sequence, adjusts the priority order of the actions according to their occurrence positions and correlation frequencies, perfects the connection details and trigger sequence between adjacent actions, integrates the updated action chain, and generates the optimized action execution plan.

5. The multimodal perception fusion humanoid robot embodied intelligence collaborative control system according to claim 1, wherein, The cross-modal behavior correction module includes: The offset detection submodule extracts the motion trajectory of the robot in the execution process according to the optimized action execution plan, observes the running path and direction change of each action section, identifies the trajectory node deviating from the target position, and records its distribution in the action chain. By comparing the corresponding relationship between the trajectory and the target position, the action section with deviation is identified, and the offset distribution feature is generated; The trajectory correction submodule reads the spatial direction information of the abnormal trajectory based on the offset distribution feature, analyzes the connection relationship between the nodes, and determines the path paragraph where the offset occurs. According to the offset direction, the trajectory connection order is rearranged, the extension direction of the action section is adjusted, the motion path is reattached to the target trajectory, and the path correction result is generated; The action coordination submodule retraces the overall rhythm of the action chain according to the path correction result, combs the connection and transition relationship between actions, adjusts the rhythm order and amplitude change of the actions, makes the corrected path maintain continuity and coordination in time, integrates the trajectory and action rhythm information, and generates the behavior coordination action sequence.

6. The multimodal perception fusion humanoid robot embodied intelligence collaborative control system according to claim 1, wherein, The task priority allocation module includes: The task identification submodule calls the behavior coordination action sequence, reads the current task state and spatial position of the robot, combs the corresponding relationship between each task node and the target object, labels the tasks being executed and to be allocated, integrates the distribution of tasks in the scene, establishes the corresponding record of the task and the robot running state, and generates the task state distribution map; The distance matching submodule extracts the position relationship between the robot and the task target based on the task state distribution map, analyzes the adaptability between the task demand and the robot function, identifies the matching objects close in space, makes task attribution judgment on the robots meeting the execution conditions, determines the sequence of task allocation, and generates the task matching result; The division coordination submodule aggregates the overall task distribution of the robot according to the task matching result, checks the connection and execution sequence of the task chain, identifies the cross or gap position between tasks, sequentially arranges and connects the uncoordinated task segments, re-plans the task division structure, integrates the updated execution sequence, and generates a robot task execution list.

7. The multimodal perception fusion humanoid robot embodied intelligence collaborative control system according to claim 1, wherein, Also includes: The timing conflict correction module calls the robot task execution list, extracts the spatial distribution and time node in the task path, analyzes the path intersection domain and time conflict, adjusts the task sequence and path distribution of the conflict point, optimizes the timing distribution and spatial coordination of task execution, and generates a robot collaborative execution path. The robot collaborative execution path includes a timing coordination diagram, a spatial obstacle avoidance model, and group synchronization parameters.

8. The multimodal perception fusion humanoid robot embodied intelligence collaborative control system according to claim 7, wherein, The timing conflict correction module includes: The path extraction submodule calls the robot task execution list, reads the distribution and time node of each task path in space, sorts the moving direction and stage connection of different robots in the execution process, labels the intersection section of the starting point and ending point of the path, integrates the corresponding relationship between time and space, forms the overall timing structure of task operation, and generates a path timing distribution diagram. The conflict identification submodule compares the running tracks of multiple task paths based on the path timing distribution diagram, observes the time overlap and spatial intersection of task nodes, identifies task segments that appear in the same area at the same time, summarizes the position and corresponding time period of the conflict, arranges the distribution relationship of the conflict nodes, and generates a task conflict list. The sequence adjustment submodule sorts the time sequence of task execution according to the task conflict list, locates the key nodes affected, rearranges the connection rhythm and path direction between tasks, adjusts the execution sequence to reduce time overlap, reorganizes the action sequence of multiple robots in the shared space, integrates the updated task timing relationship, and generates a robot collaborative execution path.

Images