Robot embodied intelligent control method and system based on multimodal perception

Abstract

The present invention relates to the field of robot control technology, and discloses a robot embodied intelligent control method and system based on multimodal perception. The method collects data such as environmental images, contact force perception, and sound signals through multimodal perception technology, generates a control log, performs state evaluation and action intention recognition, performs rationality verification on control instructions, extracts key elements of the behavior pattern of abnormal control logs, and establishes a mapping mechanism between state evaluation and control instructions after adjusting the behavior pattern, so as to capture the behavior pattern and generate control instructions in real-time control. The control system includes historical control analysis, control feature recognition, control instruction correction, and embodied control execution modules. The present invention improves the robot's multimodal perception capability, realizes effective verification and adjustment of control instructions, establishes an accurate mapping mechanism, improves the robot's environmental adaptability, control accuracy, and intelligence level, and is suitable for the field of robot embodied intelligent control.

Description

Technical Field

[0001] The present invention relates to the field of robot control technology, and in particular to a robot embodied intelligent control method and system based on multimodal perception. Background Art

[0002] With the rapid development of robotics technology, robots are increasingly being used in various fields. However, embodied intelligence, as the key to autonomous intelligent control of robots, faces many challenges in its development. Traditional robot control methods mostly rely on single-modal perception, such as vision or force perception alone, which limits the robot's ability to perceive complex environments. In practical applications, robots often need to process multiple environmental information simultaneously. Single-modal perception cannot fully and accurately reflect the environmental state, resulting in poor control accuracy and adaptability of robots in complex scenarios. For example, in some scenarios requiring delicate operations, vision alone cannot accurately perceive information such as the material and hardness of an object, which can easily lead to operational errors.

[0003] Existing robot control methods also have shortcomings in verifying the rationality of control instructions. The processing of abnormal control logs lacks a systematic approach, making it difficult to accurately identify abnormal behavior patterns and their key elements, and thus to effectively adjust control instructions, thus affecting the robot's control effectiveness and stability. When a robot exhibits abnormal behavior, traditional methods are unable to promptly identify the root cause and correct it, potentially leading to continued erroneous robot movements, even causing equipment damage or mission failure.

[0004] Traditional control methods lack a robust mapping mechanism between state assessment and control instructions, preventing them from flexibly and accurately generating corresponding control instructions based on different state assessment results. This prevents robots from making optimal action decisions when faced with varying environmental conditions, limiting their intelligence and ability to execute tasks.

[0005] Existing robot control systems lack interoperability between modules and lack effective analysis and utilization of historical control data. This inability to optimize control strategies through analysis of historical control logs results in weak learning and evolution capabilities for robots, making it difficult to continuously improve their control performance over the long term.

[0006] With the continuous expansion and complexity of robot application scenarios, the requirements for embodied intelligent control of robots are becoming increasingly higher. There is an urgent need for a control method and system that can comprehensively utilize multimodal perception data, effectively verify and adjust control instructions, and establish a complete mapping mechanism to improve the robot's environmental adaptability, control accuracy and intelligence level. Summary of the Invention

[0007] The purpose of the present invention is to provide a robot embodied intelligent control method and system based on multimodal perception to solve the problems raised in the above background technology.

[0008] To achieve the above objectives, the present invention provides the following technical solution: a robot embodied intelligent control method based on multimodal perception, the method comprising:

[0009] Step 1: Build a robot embodied intelligent management system to record the multimodal perception process and control instruction execution of any robot, generating a corresponding control log; based on the multimodal perception data collected and the recorded control instructions in any control log, perform state assessment and action intention recognition on the control log;

[0010] Step 2: Based on the status assessment and action judgment presented in any control log, the control instructions in the control log are checked for rationality; behavior pattern recognition is performed on abnormal control logs with unreasonable verification results, and key elements in the behavior pattern are extracted;

[0011] Step 3: By processing the key elements in any abnormal control log, the behavior pattern of the abnormal control log is adjusted, and the state evaluation and action judgment are re-performed; based on the control execution status presented after the adjustment, a mapping mechanism is established to map the corresponding control instructions to different state evaluation results;

[0012] Step 4: Whenever embodied intelligent control is performed on a robot, a state assessment is performed on the robot, relevant information is extracted for any key elements during the control execution process, and the behavior pattern of the robot is captured; the state assessment result is adjusted based on the captured behavior pattern, and corresponding control instructions are generated for situations where the robot needs to perform actions and execution feedback is provided.

[0013] Preferably, the step 1 comprises the following steps:

[0014] Step 1.1: Whenever the robot is subjected to embodied intelligent control, multimodal sensing technology is used to collect the robot's environmental image, contact force, and sound signals. This multimodal sensing data is transmitted to the robot's embodied intelligent management system, generating a corresponding control log. In the robot's embodied intelligent management system, evaluation rules are set for the multimodal sensing data in several dimensions to obtain characteristic indicators for each dimension.

[0015] Step 1.2: Preset an expected indicator range for each dimension of the multimodal perception data, set the expected indicator range of the i-th dimension as a specific interval, assign a weight value to each dimension according to its importance in the overall evaluation; and calculate the comprehensive evaluation value of the control log;

[0016] Step 1.3: Set an evaluation threshold. If the comprehensive evaluation value is lower than the evaluation threshold, it is determined that the robot needs to perform an action. The abnormal dimensions whose characteristic indicators are not within the expected indicator range are extracted from the control log, and features are extracted from each abnormal dimension to obtain the abnormal feature set of the control log; if there is a control instruction in the control log, the instruction type of the control instruction is obtained, and the abnormal feature set is matched with the instruction type.

[0017] Preferably, the step 2 comprises the following steps:

[0018] Step 2.1: Set the control log containing control instructions as an abnormal control log, obtain the comprehensive evaluation value of any abnormal control log and the instruction type of the recorded control instruction, and obtain the comprehensive evaluation value range corresponding to each instruction type; if there is overlap between the comprehensive evaluation value ranges of several instruction types, then set the several instruction types as abnormal instruction categories;

[0019] Step 2.2: Randomly select an abnormal instruction category and obtain the overlapping interval between the abnormal instruction category and the remaining abnormal instruction categories; select an abnormal control log from each abnormal control log belonging to the abnormal instruction category, where the comprehensive evaluation value of the abnormal control log is within the overlapping interval, and extract each abnormal dimension of the abnormal control log;

[0020] Step 2.3: Randomly select a reference control log from the remaining abnormal control logs belonging to the abnormal instruction category, and the comprehensive evaluation value of the reference control log is not in the overlapping interval, and extract each abnormal dimension of the reference control log; compare each abnormal dimension of the abnormal control log with each abnormal dimension of the reference control log to obtain a difference dimension set;

[0021] Step 2.4: Randomly select a difference dimension from the difference dimension set, obtain the characteristic index of the difference dimension in the two control logs, and calculate the difference amplitude based on the weight value of the dimension and the degree of deviation of the characteristic index; if the difference amplitude is less than the set threshold, set the difference dimension as the key dimension;

[0022] Step 2.5: Extract and merge all key dimensions in the anomaly control log to obtain the behavior pattern of the anomaly control log, perform feature extraction on the data presented in the anomaly control log for each key dimension, and obtain a set of key elements of the anomaly control log.

[0023] Preferably, step 3 comprises the following steps:

[0024] Step 3.1: Randomly select a key element from the key element set of any abnormal control log, obtain the value of the key element in the abnormal control log, obtain the average value of the key element from the remaining control logs, and obtain the degree of deviation of the key element in the abnormal control log;

[0025] Step 3.2: Obtain a specific value for the characteristic index of the key dimension of the key factor, and calculate a correction coefficient based on the weight value and deviation degree of the dimension; substitute the key factors contained in each control log of the key dimension into the correction rule to obtain the correction coefficient of each key factor;

[0026] Step 3.3: Correct the comprehensive evaluation value of the abnormality control log according to the correction coefficient of each key factor to obtain a corrected comprehensive evaluation value;

[0027] Step 3.4: Extract the abnormal dimensions and key dimensions from any abnormality control log, remove the key dimensions from the abnormal dimensions, and obtain the actual abnormality dimension set of the abnormality control log; compare the actual abnormality dimension sets in each abnormality control log, classify several abnormality control logs with the same actual abnormality dimension set into the same category, obtain the corrected comprehensive evaluation values ​​of each of the several abnormality control logs, obtain a comprehensive evaluation value range for the actual abnormality dimension set, and randomly assign an instruction type;

[0028] Step 3.5: After correcting the comprehensive evaluation value of each exception control log, obtain the comprehensive evaluation value range of each instruction type. If there is no overlap between the comprehensive evaluation value ranges of each instruction type, map each instruction type with the corresponding comprehensive evaluation value range. If there is overlap, continue to extract key dimensions and readjust them until there is no overlap between the instruction types.

[0029] Preferably, step 4 comprises the following steps:

[0030] Step 4.1: Collect multimodal perception data of a robot to generate a corresponding real-time control log, extract characteristic indicators of the real-time control log in various dimensions, and obtain a comprehensive evaluation value of the real-time control log;

[0031] Step 4.2: When the obtained comprehensive evaluation value is less than the set evaluation threshold, the abnormal dimension and key dimension of the real-time control log are extracted to obtain the actual abnormal dimension set of the real-time control log; a number of key elements under each key dimension are obtained to correct the real-time control log; the corrected comprehensive evaluation value is compared with the comprehensive evaluation value range of each instruction type to determine the instruction type of the real-time control log, generate the corresponding control instruction and send execution feedback to the execution module.

[0032] Preferably, the present invention further includes a robot embodied intelligent control system based on multimodal perception, which is used to execute the above-mentioned robot embodied intelligent control method based on multimodal perception, wherein the control system includes a historical control analysis module, a control feature recognition module, a control instruction correction module and an embodied control execution module;

[0033] The historical control analysis module is used to build a robot embodied intelligent management system to record the multimodal perception process and control instruction execution of any robot and generate a corresponding control log; based on the multimodal perception data collected and the recorded control instructions in any control log, the control log is evaluated and the action intention is identified;

[0034] The control feature recognition module is used to verify the rationality of the control instructions in any control log based on the status assessment and action judgment presented in the control log; identify the behavior pattern of abnormal control logs with unreasonable verification results, and extract key elements from the behavior pattern;

[0035] The control instruction correction module is used to adjust the behavior pattern of any abnormal control log by processing the key elements in the abnormal control log, and re-evaluate the status and action judgment; based on the control execution status presented after the adjustment, a mapping mechanism is established to map the corresponding control instructions to different status evaluation results;

[0036] The embodied control execution module is used to perform a state assessment on a certain robot whenever embodied intelligent control is performed on the robot, extract relevant information for any key elements during the control execution process, and capture the behavior pattern of the certain robot; adjust the state assessment result based on the captured behavior pattern, generate corresponding control instructions for situations where the certain robot needs to perform actions, and provide execution feedback.

[0037] Preferably, the historical control analysis module includes a control log construction unit and a control status evaluation unit;

[0038] The control log construction unit is used to construct a robot embodied intelligent management system to record the multimodal perception process and control instruction execution status of any robot and generate a corresponding control log; the control state evaluation unit is used to perform state evaluation and action intention recognition on the control log based on the multimodal perception data collected and the recorded control instructions in any control log.

[0039] Preferably, the control feature recognition module includes a control instruction verification unit and a key element extraction unit;

[0040] The control instruction verification unit is used to verify the rationality of the control instructions in the control log based on the status assessment and action judgment presented by any control log; the key element extraction unit is used to identify the behavior pattern of the abnormal control log with unreasonable verification results, and extract the key elements in the behavior pattern.

[0041] Preferably, the control instruction correction module includes a control state adjustment unit and an instruction mapping setting unit;

[0042] The control state adjustment unit is used to adjust the behavior pattern of any abnormal control log by processing the key elements in the abnormal control log, and re-evaluate the state and make action judgments; the instruction mapping setting unit is used to establish a mapping mechanism to map corresponding control instructions to different state evaluation results based on the control execution status presented after the adjustment.

[0043] Preferably, the embodied control execution module includes a behavior pattern capturing unit and a control abnormality feedback unit;

[0044] The behavior pattern capture unit is used to perform a state assessment on a certain robot whenever embodied intelligent control is performed on the robot, extract relevant information for any key elements during the control execution process, and capture the behavior pattern of the certain robot; the control abnormality feedback unit is used to adjust the state assessment result based on the captured behavior pattern, generate corresponding control instructions for situations where the certain robot needs to perform actions, and provide execution feedback.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] In terms of enhancing perception capabilities, multimodal perception technology collects multi-dimensional data such as the robot's environmental images, contact force, and sound signals, breaking through the limitations of traditional single-modality perception. The integration of multimodal data enables robots to perceive complex environments more comprehensively and accurately. For example, when manipulating an object, it can not only obtain the object's shape and position through vision, but also perceive the object's hardness and weight through contact force, and determine whether any abnormalities occur during the operation through sound signals. This provides a richer and more accurate information foundation for subsequent control decisions, significantly improving the robot's perception and understanding of the environment.

[0047] In terms of control instruction rationality verification and exception handling, this method can systematically verify the rationality of control instructions in the control log. By setting evaluation thresholds, analyzing comprehensive evaluation values, and processing abnormal control logs, unreasonable control instructions can be accurately identified. For abnormal control logs, in-depth behavioral pattern recognition and key elements can be extracted, providing clear direction for correcting abnormal behavior. For example, when a robot exhibits abnormal behavior, the system can quickly locate the abnormal dimension and key elements, and then adjust the control strategy in a targeted manner, effectively improving the accuracy and reliability of control instructions and reducing operational errors and equipment failures caused by unreasonable control instructions.

[0048] In terms of behavioral pattern adjustment and mapping mechanism establishment, precise adjustment of behavioral patterns is achieved by processing key elements in the abnormal control log. After re-evaluating the state and judging the action, a mapping mechanism is established between different state evaluation results and corresponding control instructions. This mapping mechanism can flexibly and accurately generate corresponding control instructions based on the different states of the robot, enabling the robot to make optimal action decisions when faced with various environmental changes. For example, in different operating scenarios, the system can automatically adjust control instructions based on real-time state evaluation results to ensure that the robot completes the task efficiently and stably, significantly improving the robot's environmental adaptability and task execution capabilities.

[0049] In terms of control system collaboration and learning evolution, the system's various modules—the historical control analysis module, control feature recognition module, control instruction correction module, and embodied control execution module—work together to form a complete control closed loop. The historical control analysis module records and analyzes control logs, providing rich historical data support for other modules. The control feature recognition module and control instruction correction module verify and adjust control instructions, optimizing the control strategy. The embodied control execution module enables real-time control and feedback of the robot. By continuously analyzing and learning from historical control data, the system can continuously optimize the control strategy and enhance the robot's learning and evolutionary capabilities, enabling the robot to accumulate experience over the long term and improve its control performance and intelligence. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This is a working principle diagram of the robot embodied intelligent control method based on multimodal perception according to the present invention;

[0051] Figure 2 Design diagram for multimodal perception data evaluation and anomaly detection;

[0052] Figure 3 Design diagram for abnormal control log analysis and key dimension identification. DETAILED DESCRIPTION

[0053] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0054] See also Figure 1-Figure 3 The present invention relates to a robot embodied intelligent control method and system based on multimodal perception, and the specific implementation steps are as follows:

[0055] A robot embodied intelligent management system is constructed to record the multimodal perception process and control instruction execution of any robot and generate a corresponding control log; based on the multimodal perception data collected and the recorded control instructions in any control log, the control log is evaluated for status and action intention recognition.

[0056] Based on the status assessment and action judgment presented in any control log, the control instructions in the control log are checked for rationality; the behavior pattern of abnormal control logs with unreasonable verification results is identified, and the key elements in the behavior pattern are extracted.

[0057] By processing the key elements in any abnormal control log, the behavior pattern of the abnormal control log is adjusted, and the status evaluation and action judgment are re-performed; based on the control execution status presented after the adjustment, a mapping mechanism is established to map the corresponding control instructions to different status evaluation results.

[0058] Whenever embodied intelligent control is performed on a robot, a state assessment is performed on the robot, relevant information is extracted for any key elements during the control execution process, and the behavior pattern of the robot is captured; the state assessment results are adjusted based on the captured behavior pattern, and corresponding control instructions are generated and execution feedback is provided for situations where the robot needs to perform actions.

[0059] Example 1: Based on step 1, the specific implementation method is as follows: When the robot is subjected to embodied intelligent control, the robot's operating status and surrounding environment information are collected through multimodal sensing technology. The multimodal sensing technology here covers the collaborative work of multiple sensing means, such as using a camera to obtain environmental images, collecting contact force data through a force sensor, and capturing sound signals with the help of a microphone. These sensing devices are reasonably deployed in different parts of the robot to ensure that the required information can be obtained comprehensively and accurately. For example, a force sensor is installed at the end of the robotic arm of an industrial robot to sense the magnitude and direction of the force when in contact with an object in real time; a high-definition camera is set on the head of the robot to capture images of the surrounding environment; and a microphone is arranged at an appropriate position on the body to collect sound signals in the working scene.

[0060] After collecting multimodal perception data, it needs to be transmitted to the robot's embodied intelligent management system. Specific communication protocols are used during data transmission to ensure data integrity and real-time performance. Transmission can be wired, such as through an Ethernet connection, or wireless, such as using Wi-Fi or Bluetooth technology, depending on the robot's operating environment and design requirements. For example, wireless transmission may be preferred for robots requiring high mobility; whereas, in industrial robots operating at fixed workstations, wired transmission may be more stable and reliable.

[0061] After data is transmitted to the robot's embodied intelligent management system, the system automatically generates a corresponding control log. This log details information such as the acquisition time and content of the multimodal perception data, as well as the corresponding control instructions. The control log format utilizes a structured data storage method to facilitate subsequent querying and analysis. For example, the control log is stored in a database table, with each field corresponding to a different information category, such as timestamp, environmental image data, contact force data, sound signal data, and control instruction type.

[0062] The robot's embodied intelligence management system establishes evaluation rules for multimodal sensory data across several dimensions. These evaluation rules are tailored to the robot's specific task requirements and the characteristics of the work scenario. For example, for environmental image data, evaluation dimensions might include clarity, contrast, and target object recognition accuracy; for contact force data, these might include force magnitude, direction, and rate of change; and for sound signals, these might include volume, frequency, and abnormal sound detection. Each dimension has corresponding evaluation criteria and calculation methods to generate characteristic indicators for each dimension.

[0063] After setting the evaluation rules, it's necessary to preset an expected indicator range for each dimension of the multimodal perception data. This expected indicator range is derived from a statistical analysis of historical data from the robot's normal operating state. The expected indicator range for the i-th dimension is set to a specific interval. For example, for the environmental image clarity dimension, the expected indicator range might be set to [80, 100] (assuming a percentage scale). At the same time, weights are assigned to each dimension based on its importance in the overall evaluation. The weightings reflect the varying degrees of influence of different dimensions on the robot's operating status assessment. For example, in certain fine manipulation tasks, contact force data may be weighted relatively higher, while in environmental navigation tasks, environmental image data may be more important. Weights can be determined through expert experience combined with a significance analysis of historical data.

[0064] After setting the expected indicator range and assigning weights, the system calculates a comprehensive evaluation value for the control log based on the characteristic indicators and corresponding weights for each dimension. The comprehensive evaluation value is calculated by multiplying the characteristic indicator for each dimension by its corresponding weight and then adding the results for all dimensions. For example, if there are three dimensions with characteristic indicators A, B, and C and weights a, b, and c, respectively, the comprehensive evaluation value is A × a + B × b + C × c.

[0065] To determine whether the robot needs to perform an action, an evaluation threshold is set. This threshold is determined based on the robot's operational requirements and safety standards. If the comprehensive evaluation value falls below the threshold, the robot's current operating state may be abnormal and requires appropriate action to correct the problem. At this point, the system extracts from the control log the individual anomaly dimensions whose characteristic indicators fall outside the expected range.

[0066] For each anomaly dimension, the system performs further feature extraction to generate an anomaly feature set for the control log. This feature set contains specific characteristic information for each anomaly dimension, such as the specific value of the characteristic indicator and the degree of deviation from the expected indicator range. If a control instruction is present in the control log, the system obtains the instruction type and matches the anomaly feature set with the instruction type. This matching allows analysis of the correlation between the control instruction and the anomaly status, providing a basis for subsequent control instruction optimization.

[0067] Throughout the implementation process, the frequency of multimodal perception data collection needs to be flexibly adjusted based on the robot's work scenario and task requirements. For example, when the robot is performing high-speed movements or handling complex tasks, the collection frequency will be set higher to capture subtle state changes in a timely manner. However, when the robot is in a stable operating state or performing simple tasks, the collection frequency can be appropriately lowered to reduce data processing volume and system resource consumption. The setting of evaluation rules is a dynamic optimization process. As the robot's operating data continues to accumulate, the system will regularly update and adjust the evaluation rules to adapt to new work scenarios and task requirements. The allocation of weight values ​​also needs to be verified and adjusted based on actual operating conditions. The weight setting can be optimized by comparing the consistency between the state evaluation results under different weight allocation schemes and the actual robot operating status.

[0068] When extracting abnormal dimensions and abnormal feature sets, it is necessary to ensure the accuracy and completeness of the data. The system preprocesses the collected multimodal perception data, including denoising, filtering, data normalization, and other operations to remove interference factors in the data and improve the accuracy of feature extraction. For example, for environmental image data, denoising can reduce noise in the image and improve the accuracy of feature extraction; for contact force data, filtering can smooth force fluctuations and more accurately reflect the changing trends of force. When matching abnormal feature sets with control instruction types, a specific matching algorithm is used. This algorithm can analyze the correlation between abnormal features and control instructions, identify which control instructions are executed under specific abnormal conditions, and the execution effect of these control instructions.

[0069] Example 2: The specific implementation method of step 2 is as follows: the system will set the control log containing control instructions as an abnormal control log. This is because under normal operating conditions, the robot may not need to execute control instructions, and when a control instruction appears in the control log, it often means that the robot's operating state has changed or an abnormal situation has occurred, which requires further analysis. For any abnormal control log, the system will obtain its comprehensive evaluation value and the instruction type of the recorded control instruction. By processing a large number of abnormal control logs, the system can statistically obtain the comprehensive evaluation value range corresponding to each instruction type. For example, the comprehensive evaluation value range corresponding to instruction type A may be [40, 60], and the comprehensive evaluation value range corresponding to instruction type B may be [50, 70], etc.

[0070] The system checks for overlap between the comprehensive evaluation value ranges of each instruction type. If there are overlapping comprehensive evaluation value ranges for several instruction types, for example, instruction type A has a range of [40, 60] and instruction type B has a range of [50, 70], and they overlap within the interval [50, 60], the system will classify these instruction types as abnormal instructions. This is because overlapping evaluation value ranges may make it impossible to accurately determine which instruction type should be executed within these intervals, thereby affecting the robot's control accuracy and reliability. Therefore, these abnormal instruction types require further analysis and processing.

[0071] After determining the abnormal instruction category, the system will arbitrarily select one of the abnormal instruction categories, and then obtain the overlapping interval between the abnormal instruction category and the remaining abnormal instruction categories. For example, abnormal instruction category A is selected, and its overlapping interval with abnormal instruction category B is [50,60]. Next, an abnormal control log with a comprehensive evaluation value within the overlapping interval is selected from the abnormal control logs belonging to the abnormal instruction category. Assuming that in the control log of abnormal instruction category A, there is a log with a comprehensive evaluation value of 55, which is within the overlapping interval of [50,60], the system will extract the various abnormal dimensions of the abnormal control log. These abnormal dimensions are determined by the evaluation rules in Example 1, that is, the dimensions whose characteristic indicators are not within the expected indicator range.

[0072] The system randomly selects a reference control log from the remaining abnormal control logs belonging to the abnormal instruction category, requiring that the reference control log's comprehensive evaluation value not fall within the overlapping interval. For example, among the other control logs for abnormal instruction category A, a log with a comprehensive evaluation value of 30 is selected as the reference control log because 30 does not fall within the overlapping interval [50, 60]. Next, the system extracts the various abnormal dimensions of this reference control log.

[0073] The system compares each exception dimension of the selected exception control log with each exception dimension of the reference control log to generate a difference dimension set. The difference dimension set contains the differences between the two control logs in terms of the exception dimensions. For example, if the exception dimensions of the exception control log include dimensions X and Y, while the exception dimensions of the reference control log include dimensions X and Z, the difference dimension set will be dimensions Y and Z.

[0074] Select any difference dimension from the set of difference dimensions, and the system will retrieve the characteristic indicators for that difference dimension in both control logs. For example, if difference dimension Y is selected, characteristic indicator A for dimension Y in the anomaly control log and characteristic indicator B for dimension Y in the reference control log are retrieved. The difference magnitude is then calculated based on the dimension's weight and the degree of deviation between the characteristic indicators. The degree of deviation can be measured as the ratio of the absolute value of the difference between the two characteristic indicators to the expected indicator range. For example, if the expected indicator range for dimension Y is [C, D], the degree of deviation is |AB| / (DC), and the difference magnitude is the degree of deviation multiplied by the dimension's weight.

[0075] If the calculated difference is less than a set threshold, the dimension is designated as a key dimension. The threshold is determined based on the required accuracy of robot control and the actual application scenario. For example, if the difference is small, the difference between the two control logs in this dimension has little impact on the overall evaluation value, but it may play a key role in distinguishing abnormal instruction categories, so it is designated as a key dimension.

[0076] The system performs the above processing on each difference dimension in the difference dimension set, extracting and merging all key dimensions from the anomaly control log to obtain the behavior pattern of the anomaly control log. The behavior pattern describes the characteristics and regularities of the robot's operating state corresponding to the anomaly control log. Feature extraction is then performed on the data presented by each key dimension in the anomaly control log to obtain the key element set of the anomaly control log. The key element set contains information such as the specific values ​​of the characteristic indicators of each key dimension and the degree of deviation from the expected indicators. This information is crucial for subsequent behavioral pattern adjustments and control instruction optimization.

[0077] Throughout the implementation process, identifying abnormal instruction categories requires processing a large amount of control log data. Therefore, the system employs efficient data storage and retrieval mechanisms. For example, database indexing technology is used to accelerate the query and retrieval of abnormal control logs, ensuring efficient processing. Furthermore, the system employs algorithms to improve efficiency and accuracy in calculating the statistical range of comprehensive evaluation values ​​and overlapping intervals.

[0078] When selecting abnormal control logs and reference control logs, the system tries to select representative logs to ensure the reliability of the analysis results. For example, when selecting abnormal control logs within an overlapping interval, logs with comprehensive evaluation values ​​in the middle of the overlapping interval are selected to better reflect the typical characteristics of the overlapping interval. When selecting reference control logs, logs with comprehensive evaluation values ​​far from the overlapping interval are selected to provide a clear contrast.

[0079] When calculating the difference magnitude, the accuracy of the weight values ​​directly impacts the determination of key dimensions. Therefore, the weight values ​​determined in this embodiment need to be regularly verified and adjusted based on actual conditions. If it is found that the determination of certain key dimensions is not accurate enough, the weight values ​​of each dimension may need to be reassessed to ensure that the difference magnitude calculation accurately reflects the impact of the differences between dimensions on the comprehensive evaluation value.

[0080] Adjusting the threshold setting is also a crucial step. When the robot's operating environment or task requirements change, the threshold setting needs to be adjusted accordingly. For example, in scenarios requiring high control accuracy, the threshold setting might be set lower to identify more key dimensions. In contrast, in scenarios requiring lower control accuracy, the threshold setting can be appropriately increased to reduce the number of key dimensions and improve processing efficiency.

[0081] When extracting key feature sets, it's important to ensure data accuracy and completeness. The system preprocesses key dimension data, including cleaning and denoising, to remove interfering factors. For example, smoothing the characteristic indicator data for key dimensions can reduce data fluctuations and more accurately reflect its characteristics.

[0082] Example 3: The specific implementation method of step 3 is as follows: for any abnormal control log, the system will arbitrarily select a key element from its key element set. The key element set is extracted by analyzing the behavioral pattern of the abnormal control log in Example 2, and includes feature data under key dimensions. For example, suppose that the key element set of a certain abnormal control log includes elements such as pressure value under contact force dimension and target recognition error under environmental image dimension. The system obtains the value of the key element presented in the current abnormal control log, and at the same time obtains the average value of the key element from the remaining control logs (i.e., other historical control logs except the current abnormal control log), and calculates the degree of deviation of the key element in the current abnormal control log by comparing the two. The remaining control logs here can be historical records of the same robot in the same or similar task scenarios, or can be related control logs of the same type of robots, which are specifically determined according to the availability and relevance of the data.

[0083] The system retrieves the characteristic indicator of the key dimension of the key element as a specific value. This specific value can be a baseline value within the expected indicator range for that dimension, such as the median or mean of the expected value. A correction factor is then calculated based on the weight of that dimension and the degree of deviation calculated previously. The correction factor calculation logic combines the importance (weight) of the dimension and the degree of deviation of the key element to determine the extent of the correction to be made to the key element. For example, if the weight of a key dimension is high and the degree of deviation of the key element is large, the correction factor will be correspondingly large, and vice versa.

[0084] After obtaining the correction coefficients for key elements, the system substitutes the key elements contained in each control log for the key dimension into the correction rules to determine the correction coefficients for each key element. Correction rules are pre-defined logical rules that guide how to adjust key elements based on the correction coefficients. For example, a correction rule might specify that the corrected value of a key element equals its original value plus (or minus) the product of the correction coefficient and a baseline value. The specific calculation method is determined based on actual needs.

[0085] The system adjusts the comprehensive evaluation value of the exception control log based on the correction coefficients for each key factor, resulting in a revised comprehensive evaluation value. The comprehensive evaluation value is adjusted by substituting the correction coefficients for each key factor into the comprehensive evaluation value calculation formula and recalculating the new comprehensive evaluation value. For example, if the original comprehensive evaluation value was obtained by multiplying the characteristic indicators of each dimension by the weight value and then adding them together, the correction will first adjust the characteristic indicators corresponding to the key factors, then recalculate the accumulated value to obtain the revised comprehensive evaluation value.

[0086] After completing the correction of the comprehensive evaluation value, the system needs to extract the abnormal dimensions and key dimensions in any abnormal control log. The abnormal dimension is the dimension whose characteristic indicator determined by the evaluation rules in Example 1 is not within the expected range, while the key dimension is the dimension that plays a key role in distinguishing the abnormal instruction categories determined by the difference analysis in Example 2. The system removes the key dimensions from the abnormal dimensions and obtains the actual abnormal dimension set of the abnormal control log. This is because although the key dimensions were previously identified as abnormal dimensions, after analysis, it was found that they have special significance in distinguishing the abnormal instruction categories and need to be processed separately. Therefore, they are no longer regarded as ordinary abnormal dimensions in the actual abnormal dimension set.

[0087] The system compares the actual abnormal dimension sets in each abnormal control log and classifies several abnormal control logs with the same actual abnormal dimension sets into the same category. For example, if the actual abnormal dimension sets of two abnormal control logs both include the two dimensions of environmental image clarity and contact force direction, then they will be classified into the same category. For each category of abnormal control logs, the system obtains the corrected comprehensive evaluation value of each log, thereby obtaining a comprehensive evaluation value range for the actual abnormal dimension set and randomly assigning an instruction type to the range. The allocation of instruction types needs to follow certain rules to ensure that the same category of abnormal control logs corresponds to a unique instruction type, so that subsequent control instructions can be generated and executed.

[0088] After correcting the comprehensive evaluation value of each exception control log, the system needs to re-acquire the comprehensive evaluation value range of each instruction type. At this time, the system will check whether there is any overlap in the comprehensive evaluation value ranges between the various instruction types. If there is no overlap, it means that through the previous correction and classification operations, the evaluation value ranges corresponding to different instruction types can be clearly distinguished. At this time, each instruction type can be mapped to the corresponding comprehensive evaluation value range to establish a clear correspondence. If there is still overlap, it means that the current correction and classification are not perfect enough, and it is necessary to continue to extract key dimensions and readjust them until there is no overlap between the various instruction types. This process may require multiple iterations until the system's requirements for instruction type discrimination are met.

[0089] Throughout the implementation process, the selection of key elements must be representative to ensure that the correction process effectively reflects the issues in the exception control log. The system can use random selection or prioritize key elements based on their degree of deviation. The specific method is determined based on actual circumstances. For example, key elements with large deviations may have a greater impact on the overall assessment value and therefore be prioritized for treatment.

[0090] The selection of remaining control logs is also crucial, ensuring their data validity and relevance. The system screens these logs, excluding those with obvious anomalies or those with significantly different task scenarios from the current abnormal control log to ensure the accuracy of the average value. For example, when calculating the average value of contact force pressure, control logs from non-standard operating conditions are excluded, retaining only log data from normal operating conditions.

[0091] The development of correction rules requires careful consideration of the robot's control characteristics and actual needs, ensuring that the corrected key elements will ensure the robot's operating state moves in the desired direction. Correction rules can be developed based on expert experience and analysis of historical data, and optimized and adjusted during system operation based on actual performance. For example, if the corrected comprehensive evaluation value still fails to accurately reflect the robot's actual state, the correction rules may need to be adjusted, either by changing the calculation method of the correction coefficient or the magnitude of the correction.

[0092] When classifying exception control logs, the actual set of exception dimensions must be accurately compared. The system uses efficient algorithms to perform set comparisons to ensure accurate and efficient classification. For control logs with multiple dimensions, hashing algorithms and other technologies may be used to speed up the comparison and reduce processing time.

[0093] The assignment of instruction types must be unique and deterministic to ensure that the type of instruction to be executed can be accurately determined based on the comprehensive evaluation value during the subsequent control process. The system will create a corresponding table of instruction types, actual abnormal dimension sets, and comprehensive evaluation value ranges to facilitate quick query and mapping.

[0094] When the comprehensive evaluation value ranges for instruction types overlap, further extraction of key dimensions requires in-depth analysis of control log characteristics within the overlapping intervals. The system may need to revisit the previous discrepancy analysis process, identify new key dimensions, or adjust the weights of existing key dimensions to further improve instruction type differentiation. This iterative process requires patience and meticulousness, ensuring that each adjustment progresses toward resolving the overlap.

[0095] Example 4: The specific implementation method of step 4 is as follows: When it is necessary to perform embodied intelligent control on a certain robot, first, the multimodal perception data such as the robot's environmental image, contact force perception, and sound signals are collected in real time through multimodal perception technology. For example, when an industrial robot performs a parts assembly task, the force sensor installed at the end of the robotic arm is used to perceive the contact force in the assembly process in real time, the position and posture of the parts are photographed by a camera fixed on the workbench, and abnormal sounds that may occur during the assembly process are collected by a microphone. The collected data will be transmitted to the robot's embodied intelligent management system in real time, and the system will generate a corresponding real-time control log based on the data collection time, type and other information. The format of the real-time control log is consistent with the historical control log to facilitate subsequent unified processing and analysis.

[0096] After generating the real-time control log, the system needs to extract the characteristic indicators of the log in various dimensions. The dimension settings here are consistent with those in Example 1, including the clarity of the environmental image, the accuracy of target object recognition, the size and direction of the contact force, the volume and frequency of the sound signal, etc. For each dimension, the system calculates the corresponding characteristic indicators according to the preset evaluation rules. For example, for the clarity dimension of the environmental image, the system determines the characteristic indicators of clarity by calculating parameters such as the pixel value distribution and edge sharpness of the image; for the size dimension of the contact force, the force value collected by the force sensor is directly read as the characteristic indicator.

[0097] After extracting the characteristic indicators of each dimension, the system calculates the comprehensive evaluation value of the real-time control log based on the weight value of each dimension. The distribution of weight values ​​is the same as in Example 1, and is pre-set according to the importance of each dimension in the overall evaluation. For example, in a parts assembly task, the weight of the contact force dimension may be set to 0.4, the weight of the environmental image dimension is set to 0.3, and the weight of the sound signal dimension is set to 0.3. The comprehensive evaluation value is calculated by multiplying the characteristic indicators of each dimension by their corresponding weight values, and then adding all the results.

[0098] After obtaining the comprehensive evaluation value, the system compares it with the set evaluation threshold. The evaluation threshold is pre-set based on the robot's task requirements and safety standards, for example, 60 points (assuming the full score for the comprehensive evaluation value is 100). If the comprehensive evaluation value is less than the evaluation threshold, it indicates that the robot's current operating state may be abnormal and requires further processing. At this point, the system extracts abnormal dimensions and key dimensions from the real-time control log. Abnormal dimensions are dimensions whose characteristic indicators are not within the expected indicator range. Key dimensions are the dimensions that play a key role in behavioral pattern recognition, as obtained by analyzing historical abnormal control logs in Example 2.

[0099] After extracting the abnormal dimensions and key dimensions, the system obtains the actual abnormal dimension set of the real-time control log. The actual abnormal dimension set is obtained after removing the key dimensions from the abnormal dimensions. For example, if the abnormal dimensions include environmental image clarity, contact force direction and key dimension contact force size, then the actual abnormal dimension set is environmental image clarity and contact force direction. Next, the system needs to obtain several key elements under each key dimension to correct the real-time control log. The key elements are extracted from the behavioral patterns of the historical abnormal control logs in Example 2, such as the pressure threshold, deviation range and other key elements under the key dimension contact force size.

[0100] When correcting the real-time control log, the system will adjust the characteristic indicators of the key dimensions based on the values ​​of the key elements and the correction rules. For example, if the key element of the key dimension of contact force size shows a normal pressure threshold of 50N±5N, and the characteristic indicator of this dimension in the real-time control log is 60N, which deviates from the normal range, the system will adjust it to within the range of 50N±5N according to the correction rules. The correction rules are obtained by processing the key elements of the historical abnormal control log in Example 3. For example, it stipulates that for every 1N of deviation, the correction amount is 0.5N until it is adjusted to the normal range.

[0101] After completing the correction of the real-time control log, the system will recalculate the corrected comprehensive evaluation value. Then, the corrected comprehensive evaluation value is compared with the comprehensive evaluation value range of each instruction type. The comprehensive evaluation value range of each instruction type is a mapping relationship established after correcting and classifying the historical abnormal control log in Example 3. For example, instruction type A corresponds to the comprehensive evaluation value range [70, 80], instruction type B corresponds to [80, 90], and so on. Through this comparison, the system can determine the instruction type of the real-time control log.

[0102] After determining the command type, the system generates the corresponding control instructions and sends execution feedback to the execution module. Upon receiving the control instructions, the execution module controls the robot to perform the corresponding action. For example, if the command type is to adjust the position of the robotic arm, the execution module will control the robotic arm to move to the specified position according to the command. During the execution feedback process, the system records the execution status of the control instructions, including execution time, execution results, and other information, for subsequent analysis and optimization.

[0103] Throughout the implementation process, real-time acquisition of multimodal perception data requires ensuring data accuracy and real-time performance. To achieve this, the system regularly calibrates and maintains sensing equipment to ensure stable performance. For example, cameras undergo regular focus adjustment and white balance calibration, while force sensors undergo zero-point calibration and accuracy testing. Furthermore, reliable communication protocols are employed during data transmission to avoid data loss or delays.

[0104] The extracted feature metrics need to be dynamically adjusted based on the characteristics of real-time data. For example, when the ambient lighting changes, the calculation parameters of the ambient image clarity feature metric are automatically adjusted to adapt to the new lighting conditions. The weighting values ​​also need to be adjusted based on changes in the actual task. For example, when the robot switches from a parts assembly task to a material handling task, the weights of the contact force and ambient image dimensions need to be reallocated to more accurately reflect the importance of each dimension in the new task.

[0105] The evaluation threshold needs to be set based on the robot's task difficulty and safety requirements. For example, when performing high-precision assembly tasks, the evaluation threshold might be set higher (e.g., 70 points) to ensure that every robot movement meets high precision requirements. However, when performing simple handling tasks, the evaluation threshold might be set lower (e.g., 50 points) to improve task execution efficiency.

[0106] When extracting abnormal and key dimensions, the system references historical data and empirical knowledge to ensure that the extracted dimensions accurately reflect the robot's actual operating status. For example, if the robot makes an unusual sound, the system will combine the abnormal dimensions corresponding to similar sounds in the historical data to quickly identify the dimension that may be the cause of the problem.

[0107] Corrections to real-time control logs require careful attention to avoid over-correction or under-correction. The system continuously optimizes correction rules based on historical correction data and actual results, improving the accuracy and effectiveness of corrections. For example, by analyzing the robot's operating status after multiple corrections, the correction calculation method is adjusted to ensure that the revised comprehensive evaluation value more accurately reflects the robot's actual status.

[0108] When determining the instruction type, the system considers the position of the comprehensive evaluation value within the corresponding range and the characteristics of the actual abnormal dimension set to more accurately determine the control instruction to be executed. For example, if the comprehensive evaluation value is close to the upper limit of the range for instruction type A, and the actual abnormal dimension set partially overlaps with the typical abnormal dimensions of instruction type B, the system will further analyze key factors to determine a more appropriate instruction type.

[0109] Recording and analyzing execution feedback is crucial for system optimization. By analyzing execution feedback data, the system can identify problems in the control command execution process, such as command execution delays and execution results that do not meet expectations. Based on this information, the system can adjust and optimize the previous steps, improving the overall performance of the robot's embodied intelligent control.

[0110] Example 5: A multimodal sensing-based embodied intelligent robot control system is used to implement the control method described above. This system is implemented using an industrial assembly robot as an example. In industrial assembly scenarios, robots must grasp, position, and install precision parts. The various modules and units in the system work together to achieve intelligent control of the robot.

[0111] The system includes a historical control analysis module, which consists of a control log construction unit and a control state evaluation unit. The control log construction unit constructs an embodied intelligent management system for the robot, recording the robot's multimodal perception process and the execution of control commands. For example, when the robot grasps a part, the control log construction unit uses a camera to capture an image of the part's position, a force sensor to record contact force data at the end of the robotic arm, and a microphone to capture acoustic signals during the grasping process. It then combines this multimodal perception data with control commands (such as "grasp force 50N" and "move to coordinates (100, 200, 300)") to generate a control log. The log details the data acquisition timestamp, the raw data from each sensor, and the command parameters. The control state evaluation unit uses the multimodal perception data and control commands in the control log to perform state evaluation and action intention recognition. For example, it analyzes whether the force data during grasping is within the expected range (for example, a normal grasping force should be 40-60N). If the actual force data collected is 70N, which exceeds the expected range, combined with the control command "grasp," the unit determines that the robot may have excessive grasping force and needs to perform an adjustment action.

[0112] The control feature recognition module includes a control instruction verification unit and a key element extraction unit. The control instruction verification unit verifies the rationality of the control instruction based on the state assessment results and action judgment. For example, if the state assessment in a control log shows that the grasping force is abnormal and the control instruction is "maintain grasping", the verification unit will determine whether the instruction is reasonable. If the grasping force continues to be abnormal but the grasping is maintained, it may cause damage to the parts. In this case, the log will be marked as an abnormal control log. The key element extraction unit performs behavioral pattern recognition and key element extraction on the abnormal control log. For example, for multiple groups of control logs with abnormal grasping force, their abnormal dimensions (such as force size, force change rate) are extracted, and by comparing them with normal grasping control logs, the difference dimensions are determined, the difference amplitude is calculated, and the dimensions with a difference amplitude less than the set threshold (such as force size) are determined as key dimensions. Then, the key elements in the key dimensions are extracted, such as the threshold range of normal grasping force, the maximum allowable force deviation amplitude, etc.

[0113] The control instruction correction module consists of a control state adjustment unit and an instruction mapping setting unit. The control state adjustment unit processes the key elements of the abnormal control log and adjusts the behavior pattern. For example, the key element in a certain abnormal control log shows that the grasping force is 70N, which deviates from the average value of this element in the historical control log by 50N. The correction coefficient is calculated based on the weight of the key dimension (force size) and the degree of deviation, and the grasping force is corrected to a reasonable range, and the state is re-evaluated. The instruction mapping setting unit establishes a mapping mechanism based on the adjusted control execution status. For example, the corrected grasping force comprehensive evaluation value range (such as 45-55N) is mapped to the "normal grasping" instruction type. If the evaluation value is lower than 45N, it is mapped to the "insufficient grasping force, increase pressure" instruction type to ensure that different state evaluation results correspond to clear control instructions.

[0114] The embodied control execution module consists of a behavior pattern capture unit and a control anomaly feedback unit. The behavior pattern capture unit performs state assessment and behavior pattern capture during real-time robot operation. For example, when the robot is assembling a new batch of parts, it collects multimodal data in real time to generate a control log. It extracts characteristic indicators and calculates a comprehensive evaluation value. If the evaluation value indicates a grasping force of 35N, which is below the evaluation threshold, it extracts abnormal dimensions (force perception) and key dimensions. The evaluation value is then corrected based on key factors (such as the normal grasping force threshold), capturing the "insufficient grasping force" behavior pattern. The control anomaly feedback unit adjusts the state assessment results based on the captured behavior pattern, generates control instructions, and provides feedback for execution. If the behavior pattern is determined to be "insufficient grasping force," it generates an "increase grasping force to 50N" instruction and sends it to the execution module. The execution module controls the robotic arm to adjust the grasping force and reports the execution result (e.g., if the actual grasping force reaches 50N) back to the system, recording it in a new control log.

[0115] During system operation, data exchange between modules is achieved through data interfaces. The historical control analysis module transmits the processed control log and evaluation results to the control feature recognition module, which passes the verified abnormality log and key elements to the control instruction correction module. The correction module sends the established mapping relationship to the embodied control execution module. The real-time data collected by the execution module and the execution feedback are then returned to the historical control analysis module, forming a closed-loop management. For example, during a certain assembly, the robot's grasping force is abnormal due to oil contamination on the part surface. The system records the abnormality log through the historical control analysis module, the control feature recognition module extracts key elements (such as force perception deviation under the influence of oil contamination), and the control instruction correction module adjusts the mapping mechanism (increasing the grasping force threshold by 10N in the oil contamination scenario). When the embodied control execution module encounters similar scenarios in the future, it automatically generates control instructions based on the new mapping relationship to achieve adaptive adjustment to special working conditions.

[0116] The system's hardware deployment includes an industrial-grade computer as the core processing unit, connected to sensing devices such as cameras, six-dimensional force sensors, and microphones, as well as execution modules such as servo motor drivers. The software layer adopts a distributed architecture, with each module running as an independent process and communicating data via message queues to ensure real-time performance and stability. For example, the control log construction unit collects sensor data at a frequency of 500Hz and transmits it to the data processing server via the UDP protocol. The control state evaluation unit parses and evaluates the data in real time. The latency of the entire process is controlled within 100ms, meeting the real-time requirements of industrial assembly.

[0117] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0118] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A robot embodied intelligent control method based on multimodal perception, characterized by: The control method comprises the following steps: Step 1: Build a robot embodied intelligent management system to record the multimodal perception process and control instruction execution of any robot, generating a corresponding control log; based on the multimodal perception data collected and the recorded control instructions in any control log, perform state assessment and action intention recognition on the control log; Step 2: Based on the status assessment and action judgment presented in any control log, the control instructions in the control log are checked for rationality; behavior pattern recognition is performed on abnormal control logs with unreasonable verification results, and key elements in the behavior pattern are extracted; Step 3: By processing the key elements in any abnormal control log, the behavior pattern of the abnormal control log is adjusted, and the state evaluation and action judgment are re-performed; based on the control execution status presented after the adjustment, a mapping mechanism is established to map the corresponding control instructions to different state evaluation results; Step 4: Whenever embodied intelligent control is performed on a robot, the robot's state is evaluated, relevant information is extracted for any key elements during the control execution process, and the robot's behavior pattern is captured; the state evaluation results are adjusted based on the captured behavior pattern, and corresponding control instructions are generated for the robot's required actions and execution feedback is provided; The step 1 comprises the following steps: Step 1.1: Whenever the robot is subjected to embodied intelligent control, multimodal sensing technology is used to collect the robot's environmental image, contact force, and sound signals. This multimodal sensing data is transmitted to the robot's embodied intelligent management system, generating a corresponding control log. In the robot's embodied intelligent management system, evaluation rules are set for the multimodal sensing data in several dimensions to obtain characteristic indicators for each dimension. Step 1.2: Preset an expected indicator range for each dimension of the multimodal perception data, set the expected indicator range of the i-th dimension as a specific interval, assign a weight value to each dimension according to its importance in the overall evaluation; and calculate the comprehensive evaluation value of the control log; Step 1.3: Set an evaluation threshold. If the comprehensive evaluation value is lower than the evaluation threshold, determine that the robot needs to perform an action. Extract each abnormal dimension whose characteristic index is not within the expected index range from the control log. Perform feature extraction on each abnormal dimension to obtain an abnormal feature set of the control log. If a control instruction exists in the control log, obtain the instruction type of the control instruction and match the abnormal feature set with the instruction type. The step 3 comprises the following steps: Step 3.1: Randomly select a key element from the key element set of any abnormal control log, obtain the value of the key element in the abnormal control log, obtain the average value of the key element from the remaining control logs, and obtain the degree of deviation of the key element in the abnormal control log; Step 3.2: Obtain a specific value for the characteristic index of the key dimension of the key factor, and calculate a correction coefficient based on the weight value and deviation degree of the dimension; substitute the key factors contained in each control log of the key dimension into the correction rule to obtain the correction coefficient of each key factor; Step 3.3: Correct the comprehensive evaluation value of the abnormality control log according to the correction coefficient of each key factor to obtain a corrected comprehensive evaluation value; Step 3.4: Extract the abnormal dimensions and key dimensions from any abnormality control log, remove the key dimensions from the abnormal dimensions, and obtain the actual abnormality dimension set of the abnormality control log; compare the actual abnormality dimension sets in each abnormality control log, classify several abnormality control logs with the same actual abnormality dimension set into the same category, obtain the corrected comprehensive evaluation values ​​of each of the several abnormality control logs, obtain a comprehensive evaluation value range for the actual abnormality dimension set, and randomly assign an instruction type; Step 3.5: After correcting the comprehensive evaluation value of each exception control log, obtain the comprehensive evaluation value range of each instruction type. If there is no overlap between the comprehensive evaluation value ranges of each instruction type, map each instruction type with the corresponding comprehensive evaluation value range. If there is overlap, continue to extract key dimensions and readjust them until there is no overlap between the instruction types.

2. The method for embodied intelligent control of a robot based on multimodal perception according to claim 1, characterized in that: The step 2 comprises the following steps: Step 2.1: Set the control log containing control instructions as an abnormal control log, obtain the comprehensive evaluation value of any abnormal control log and the instruction type of the recorded control instruction, and obtain the comprehensive evaluation value range corresponding to each instruction type; if there is overlap between the comprehensive evaluation value ranges of several instruction types, then set the several instruction types as abnormal instruction categories; Step 2.2: Randomly select an abnormal instruction category and obtain the overlapping interval between the abnormal instruction category and the remaining abnormal instruction categories; select an abnormal control log from each abnormal control log belonging to the abnormal instruction category, where the comprehensive evaluation value of the abnormal control log is within the overlapping interval, and extract each abnormal dimension of the abnormal control log; Step 2.3: Randomly select a reference control log from the remaining abnormal control logs belonging to the abnormal instruction category, and the comprehensive evaluation value of the reference control log is not in the overlapping interval, and extract each abnormal dimension of the reference control log; compare each abnormal dimension of the abnormal control log with each abnormal dimension of the reference control log to obtain a difference dimension set; Step 2.4: Randomly select a difference dimension from the difference dimension set, obtain the characteristic index of the difference dimension in the two control logs, and calculate the difference amplitude based on the weight value of the dimension and the degree of deviation of the characteristic index; if the difference amplitude is less than the set threshold, set the difference dimension as the key dimension; Step 2.5: Extract and merge all key dimensions in the anomaly control log to obtain the behavior pattern of the anomaly control log, perform feature extraction on the data presented in the anomaly control log for each key dimension, and obtain a set of key elements of the anomaly control log.

3. The method for embodied intelligent control of a robot based on multimodal perception according to claim 1, characterized in that: The step 4 comprises the following steps: Step 4.1: Collect multimodal perception data of a robot to generate a corresponding real-time control log, extract characteristic indicators of the real-time control log in various dimensions, and obtain a comprehensive evaluation value of the real-time control log; Step 4.2: When the obtained comprehensive evaluation value is less than the set evaluation threshold, the abnormal dimension and key dimension of the real-time control log are extracted to obtain the actual abnormal dimension set of the real-time control log; a number of key elements under each key dimension are obtained to correct the real-time control log; the corrected comprehensive evaluation value is compared with the comprehensive evaluation value range of each instruction type to determine the instruction type of the real-time control log, generate the corresponding control instruction and send execution feedback to the execution module.

4. A multimodal sensing-based robot embodied intelligent control system, configured to execute the multimodal sensing-based robot embodied intelligent control method according to any one of claims 1 to 3, characterized in that: The control system includes a historical control analysis module, a control feature recognition module, a control instruction correction module and an embodied control execution module; The historical control analysis module is used to build a robot embodied intelligent management system to record the multimodal perception process and control instruction execution of any robot and generate a corresponding control log; based on the multimodal perception data collected and the recorded control instructions in any control log, the control log is evaluated and the action intention is identified; The control feature recognition module is used to perform rationality verification on the control instructions in the control log based on the status evaluation and action judgment presented by any control log; Identify behavior patterns of abnormal control logs with unreasonable verification results and extract key elements from the behavior patterns; The control instruction correction module is used to adjust the behavior pattern of any abnormal control log by processing the key elements in the abnormal control log, and re-evaluate the status and determine the action; Based on the control execution status presented after adjustment, a mapping mechanism is established to map corresponding control instructions to different state evaluation results; The embodied control execution module is configured to perform a status assessment on a robot whenever embodied intelligent control is performed on the robot, extract relevant information for any key elements during the control execution process, and capture the behavior pattern of the robot; The state assessment result is adjusted based on the captured behavior pattern, and corresponding control instructions are generated and execution feedback is provided for the situation where the robot needs to perform an action.

5. The multimodal perception-based robot embodied intelligent control system according to claim 4, characterized in that: The historical control analysis module includes a control log construction unit and a control status evaluation unit; The control log construction unit is used to construct a robot embodied intelligent management system to record the multimodal perception process and control instruction execution status of any robot and generate a corresponding control log; the control state evaluation unit is used to perform state evaluation and action intention recognition on the control log based on the multimodal perception data collected and the recorded control instructions in any control log.

6. The multimodal perception-based robot embodied intelligent control system according to claim 4, characterized in that: The control feature recognition module includes a control instruction verification unit and a key element extraction unit; The control instruction verification unit is used to verify the rationality of the control instructions in the control log based on the status assessment and action judgment presented by any control log; the key element extraction unit is used to identify the behavior pattern of the abnormal control log with unreasonable verification results, and extract the key elements in the behavior pattern.

7. The multimodal perception-based robot embodied intelligent control system according to claim 4, characterized in that: The control instruction correction module includes a control state adjustment unit and an instruction mapping setting unit; The control state adjustment unit is used to adjust the behavior pattern of any abnormal control log by processing the key elements in the abnormal control log, and re-evaluate the state and make action judgments; the instruction mapping setting unit is used to establish a mapping mechanism to map corresponding control instructions to different state evaluation results based on the control execution status presented after the adjustment.

8. The multimodal perception-based robot embodied intelligent control system according to claim 4, characterized in that: The embodied control execution module includes a behavior pattern capture unit and a control abnormality feedback unit; The behavior pattern capture unit is used to perform a state assessment on a certain robot whenever embodied intelligent control is performed on the robot, extract relevant information for any key elements during the control execution process, and capture the behavior pattern of the certain robot; the control abnormality feedback unit is used to adjust the state assessment result based on the captured behavior pattern, generate corresponding control instructions for situations where the certain robot needs to perform actions, and provide execution feedback.

Images