A Multimodal Perception Fusion Method for Humanoid Robots Based on Dynamic Sparse Activation

Abstract

This invention discloses a multimodal perception fusion method for humanoid robots based on dynamic sparse activation, belonging to the field of multimodal perception fusion technology. The method includes: acquiring multimodal data streams from the robot; generating modal availability mappings based on the robot's real-time operating state; dynamically selecting activation modalities using a sparse gating strategy and constructing an activation modal set based on the selection results; performing state evaluation on the activation modal set using a Kalman filter algorithm to obtain a state evaluation set and identify modal switching states; when a modal switching is identified, constructing a state evaluation window based on task complexity and progress changes, dynamically adjusting the robot's state space dimension within the state evaluation window to obtain a multimodal state set; generating robot task execution instructions based on the multimodal state set and controlling the robot's behavior, thus solving the problem of mismatched robot state space dimensions caused by modal switching when dynamically selecting activation modalities using a sparse gating strategy.

Description

Technical Field

[0001] This invention relates to the field of multimodal perception fusion technology, and more specifically, to a multimodal perception fusion method for humanoid robots based on dynamic sparse activation. Background Technology

[0002] With the continuous development of robotics technology, especially the advancements in artificial intelligence, machine learning, and robot perception systems, robots are becoming increasingly intelligent, capable of performing various tasks in complex and dynamic environments. To achieve this, robots no longer rely on a single perception modality but need to integrate multiple perception methods to form a multimodal perception system. Common perception modalities include vision, touch, speech, hearing, and LiDAR, each providing the robot with different information and perspectives. For example, a vision system helps the robot identify objects and analyze scenes, while a tactile system enables the robot to perceive the shape, hardness, and other physical properties of objects, and a speech system allows the robot to interact with humans more naturally. These different perception modalities work together, enabling the robot to perceive its surroundings and its own state in a more comprehensive and accurate way, thus making more rational and effective decisions.

[0003] When employing a sparse gating strategy to dynamically select active perception modes, the robot dynamically chooses which perception modes to activate based on real-time perception data and operational status, thereby optimizing perception efficiency and reducing unnecessary computational overhead. However, since the input data dimensionality and processing complexity differ for each perception mode, switching between different perception modes may cause changes in the dimensionality of the state space. This dimensionality mismatch, if not adjusted and optimized in a timely manner, may lead to errors or inaccuracies in subsequent state estimation processes, thus affecting the robot's overall decision-making and behavior.

[0004] To address the above problems, this invention proposes a solution. Summary of the Invention

[0005] To overcome the aforementioned deficiencies of the prior art, embodiments of the present invention provide a multimodal perception fusion method for humanoid robots based on dynamic sparse activation. By constructing a state evaluation window and dynamically adjusting the robot's state space dimension within the state evaluation window, the method addresses the problem of mismatch in robot state space dimension caused by mode switching when dynamically selecting the active perception mode using a sparse gating strategy.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A multimodal perception fusion method for humanoid robots based on dynamic sparse activation includes the following steps: acquiring multimodal data streams of the robot and generating modal availability maps based on the robot's real-time operating state; dynamically selecting the perception modes to be activated based on the modal availability maps using a sparse gating strategy and constructing an activated modal set based on the selection results; performing state evaluation on the activated modal set using a Kalman filter algorithm to obtain a state evaluation set and identifying modal switching states based on the state evaluation set; when a modal switching is identified, constructing a state evaluation window based on task complexity and progress changes, and dynamically adjusting the robot's state space dimension within the state evaluation window to obtain a multimodal state set; generating robot task execution instructions based on the multimodal state set and controlling the robot's behavior.

[0008] In a preferred embodiment, the step of acquiring the robot's multimodal data stream and generating a modal availability map based on the robot's real-time operating status specifically involves: acquiring the robot's multimodal data stream and extracting the modal feature vector for each modal data; acquiring the robot's real-time operating status and calculating the data quality assessment value of the modal feature vector based on the real-time operating status; assigning a modal availability value to each modal feature vector according to the data quality assessment value, and generating a modal availability map based on the modal availability value.

[0009] In a preferred embodiment, the step of dynamically selecting the perceptual modalities to be activated based on the modal availability map using a sparse gating strategy and constructing an activated modal set according to the selection results specifically involves: extracting the modal availability values ​​of all modalities from the modal availability map and generating a modal availability sequence by sorting the modal availability values; using a threshold segmentation algorithm to initially segment the modal availability sequence to obtain several modal availability intervals; performing several recursive processes on each modal availability interval, and calculating the variance and distribution density of the modal availability values ​​within the modal availability interval during each recursive process; dynamically adjusting the interval boundaries according to the variance and distribution density of the modal availability values ​​within the modal availability interval to obtain the recursed modal availability intervals; dynamically selecting core activated modalities and auxiliary modalities from the recursed modal availability intervals using a sparse gating strategy; performing conflict detection on the core activated modalities and auxiliary modalities, and merging the core activated modalities and auxiliary modalities according to the conflict detection results to obtain the activated modal set.

[0010] In a preferred embodiment, the step of using the Kalman filter algorithm to evaluate the state of the active mode set, obtaining a state evaluation set, and identifying the mode switching state based on the state evaluation set specifically involves: defining the state variables and observation variables for each mode in the active mode set, and initializing the state estimation matrix and error covariance matrix of the Kalman filter; setting the time interval between the prediction step and the update step of the Kalman filter according to the data update frequency of each mode; using the Kalman filter algorithm to predict and update the state of each mode at each time interval, obtaining the state estimate and covariance matrix of each mode; calculating the error variance of the state estimate of each mode based on the covariance matrix, and performing a weighted average of the state estimates of each mode based on the error variance to obtain the state evaluation set;

[0011] The state change trend of each mode in the state assessment set is analyzed and the state difference is calculated. When the state difference exceeds the preset threshold, it is identified as a mode switching state.

[0012] In a preferred embodiment, when a mode switch is identified, constructing a state evaluation window based on task complexity and progress change specifically involves: when a mode switch is identified, obtaining the current task complexity and task progress change rate; the task complexity is calculated based on the number of task objectives and dependencies, and the task progress change rate is the ratio of completed task steps to total steps; and determining the size and position of the state evaluation window based on the task complexity and task progress change rate.

[0013] In a preferred embodiment, determining the size and position of the state evaluation window based on task complexity and task progress change rate specifically involves: determining the initial length of the state evaluation queue based on task complexity, and setting the sampling interval of the elements in the state evaluation queue based on the task progress change rate; acquiring the robot state vector and its corresponding timestamp, and constructing state evaluation points based on the robot state vector and its corresponding timestamp; inserting state evaluation points into the state evaluation queue according to the sampling interval; when the state evaluation queue reaches its initial length, activating a preset queue full handling mechanism to remove the first state evaluation point at the head of the state evaluation queue and insert a new state evaluation point; extracting the timestamps from the head and tail of the state evaluation queue, calculating the difference between the timestamps as the size of the state evaluation window, and using the time point in the middle of the state evaluation queue as the center position of the state evaluation window.

[0014] In a preferred embodiment, the step of dynamically adjusting the robot state space dimension within the state evaluation window to obtain a multimodal state set specifically involves: extracting robot state vectors from all state evaluation points in the state evaluation queue within the state evaluation window and constructing a state matrix; transforming the rows and columns of the state matrix using a matrix transpose operation to obtain a transposed state matrix; analyzing the row vector distribution of the transposed state matrix and constructing a multimodal state set based on the analysis results.

[0015] In a preferred embodiment, the step of analyzing the row vector distribution of the transposed state matrix and constructing a multimodal state set based on the analysis results specifically involves: dynamically recombining the robot state vectors according to the row vector distribution of the transposed state matrix to obtain the recombined robot state vectors; acquiring the robot's perception requirements after mode switching and analyzing the column vector change trends of the state matrix based on the robot's perception requirements to obtain the time series pattern of the robot state vectors; dynamically adjusting the dimensions of the recombined robot state vectors based on the time series pattern of the robot state vectors to obtain the adjusted robot state vectors; and integrating all the adjusted robot state vectors to construct a multimodal state set.

[0016] In a preferred embodiment, the step of generating robot task execution instructions and controlling robot behavior based on a multimodal state set specifically involves: parsing the multimodal state set, extracting the robot's current state vector, and determining the feasible action range based on the robot's current state vector; generating candidate action sequences using a preset breadth-first search algorithm based on the feasible action range and a preset task objective; acquiring robot dynamic constraints and environmental obstacle information, evaluating the feasibility of the candidate action sequences, and obtaining the optimal action sequence; generating task execution instructions based on the optimal action sequence, and sending the task execution instructions to the robot controller for real-time control of the robot's behavior.

[0017] The technical effects and advantages of this invention's humanoid robot multimodal perception fusion method based on dynamic sparse activation are as follows:

[0018] This invention effectively evaluates the data quality of each perception modality by acquiring the robot's multimodal data stream in real time and generating a modal availability map, thereby dynamically selecting and activating the most suitable perception modality. This process first uses a sparse gating strategy to dynamically adjust the set of activated modalities based on modal availability, avoiding unnecessary computational burden and improving perception efficiency. Simultaneously, a Kalman filter algorithm is used to evaluate the state of the activated modalities, accurately identifying modality switching states, and adjusting the dimensionality of the state space based on the state evaluation results to address potential state space dimensionality mismatches during modality switching. By constructing a state evaluation window and dynamically adjusting the robot's state space dimension, this method ensures that the robot maintains a consistent state space structure during modality switching, avoiding estimation errors or computational inconsistencies caused by modality switching. Finally, the task execution instructions generated based on the multimodal state set can precisely control the robot's behavior, ensuring efficient and accurate task completion. The technical effects and advantages of this method are that it can significantly improve the robot's perception ability, decision-making accuracy and execution efficiency in dynamic environments by optimizing the selection of perception modes and the dynamic adjustment of state space. In particular, it avoids calculation errors caused by mismatch of state dimensions during mode switching, thereby improving the stability and reliability of the robot in performing complex tasks. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating the multimodal perception fusion method for humanoid robots based on dynamic sparse activation according to the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0021] Example 1, Figure 1 This invention presents a multimodal perception fusion method for humanoid robots based on dynamic sparse activation, comprising the following steps:

[0022] S1, acquire the robot's multimodal data stream and generate a modal availability map based on the robot's real-time operating status;

[0023] In this example, the robot's multimodal data stream is acquired, and a modal availability mapping is generated based on the robot's real-time operating status, specifically as follows:

[0024] Acquire the robot's multimodal data stream and extract the modal feature vector for each modal data;

[0025] The robot's real-time operating status is obtained, and the data quality assessment value of the modal feature vector of the modal data is calculated based on the real-time operating status.

[0026] Each modal feature vector is assigned a modal availability value based on the data quality assessment value, and a modal availability map is generated based on the modal availability value.

[0027] It's important to note that robot multimodal data streams refer to data sets from different sensory channels, typically including vision (images or videos captured by cameras), speech (sound signals received through microphones), and tactile data (pressure, temperature, or contact perception data obtained through sensors). These data represent different sensory capabilities of the robot; for example, the visual modality provides image information of the environment, the speech modality captures voice commands or environmental noise, and the tactile modality perceives physical contact. The process of acquiring multimodal data streams usually involves sensors collecting these signals in real time. For the visual modality, the robot collects image or video data through cameras or depth sensors and extracts feature vectors from it using image processing algorithms (such as feature extraction and object recognition). These feature vectors may include the color distribution, edge information, and key point locations of the image. For the speech modality, the sound signals collected by the microphone are converted into a set of feature vectors representing the speech content through acoustic feature extraction, such as MFCC (Mel-frequency cepstral coefficients). The tactile modality collects data through tactile sensors (such as force sensors and pressure sensors), which reflect the intensity of physical contact, surface texture, or temperature changes between the robot and its environment.

[0028] Furthermore, the robot's real-time operational status refers to various dynamic information about the robot while it is performing a task. This information includes the robot's posture, speed, acceleration, task progress, etc. Acquiring the robot's operational status typically relies on internal sensors (such as accelerometers, gyroscopes, GPS, inertial measurement units, etc.) to track the robot's physical motion in real time. Data quality assessment (DQA) is an indicator used to measure the reliability and validity of the robot's modal data. It is usually calculated by analyzing the completeness, accuracy, and noise level of the modal data. The DQA calculation is based on the robot's real-time state. For example, if the robot is moving at high speed or in low ambient light, it may affect the clarity of visual data or the accuracy of tactile data; therefore, in such cases, the modal quality assessment value will be lower. Similarly, if the speech modal data is affected by noise, its quality assessment value will also decrease.

[0029] For example, suppose a robot is performing a navigation task; its real-time operating status is its current position and speed. If it moves rapidly in a complex environment, the images captured by the visual sensors may be blurry, reducing the quality assessment value. Similarly, if the tactile sensors malfunction or fail to detect the expected pressure, the data quality assessment value will also be affected. These assessment values ​​can be determined by comparing them to predetermined standards or through monitoring the sensor status (such as signal strength and error analysis).

[0030] Finally, modal availability is a quantification of the availability of each modality's data, reflecting the effectiveness of that modality in the current robot state. For example, if the quality assessment value of visual data is high, it indicates that the current visual information is relatively reliable, and therefore the availability value of the visual modality is high; conversely, if the quality assessment value of tactile data is low, then the availability value of the tactile modality will be low. When converting data quality assessment values ​​into modal availability values, a numerical value is assigned to the feature vector of each modality, representing the effectiveness of that modality in the current environment and task. This availability value is typically achieved through a mapping function, which may set a threshold based on the data quality assessment value. For example, when the quality assessment value is higher than a certain set threshold, the availability value is 1, indicating that the modality is fully usable; if the quality assessment value is lower than a certain threshold, the availability value may be close to 0, indicating that the modality is unusable.

[0031] Based on these modal availability values, a modal availability map can be generated. This map can be viewed as a list or graph representing the effectiveness of different perceptual modalities, where each modality (such as visual, speech, and tactile) is assigned an availability value. This map helps the system dynamically select the most appropriate modality to activate or use based on the current environmental state and task requirements. For example, if the visual modality has a higher availability value when performing complex gesture recognition tasks, the system may prioritize using visual perception for task execution; while in noisy environments, the speech modality has a lower availability value, and the system may choose to rely on other modalities to supplement information.

[0032] S2, based on modality availability mapping, adopts a sparse gating strategy to dynamically select the perceptual modalities to be activated and constructs an active modality set based on the selection results;

[0033] In this example, based on modality availability mapping, a sparse gating strategy is used to dynamically select the perceptual modalities to be activated and construct an activated modality set based on the selection results, specifically:

[0034] Extract modal availability values ​​for all modes from the modal availability map, and generate a modal availability sequence by sorting the modal availability values;

[0035] A threshold segmentation algorithm is used to initially segment the modal availability sequence, resulting in several modal availability intervals;

[0036] For each modal availability interval, perform several recursive processes, and calculate the variance and distribution density of the modal availability values ​​within the modal availability interval during each recursive process;

[0037] The modal availability interval is dynamically adjusted based on the variance and distribution density of the modal availability values ​​within the interval to obtain the recursive modal availability interval.

[0038] A sparse gating strategy is used to dynamically select the core active mode and auxiliary modes from the recursive modality availability interval;

[0039] Conflict detection is performed on the core activation mode and auxiliary mode, and the core activation mode and auxiliary mode are merged according to the conflict detection results to obtain the activation mode set.

[0040] It's important to note that the modality availability map includes availability values ​​for different perceptual modalities (such as vision, speech, and touch). These values ​​are calculated based on the reliability and effectiveness of each modality in the current environment and task. Specifically, the availability value for each modality is extracted from the modality availability map. These availability values ​​may be between 0 and 1, representing the modality's effectiveness. For example, the availability value for the visual modality might be 0.8, for the speech modality 0.5, and for the touch modality 0.2. Next, these modality availability values ​​are sorted by magnitude to form a modality availability sequence. This sequence is arranged in descending or ascending order, representing the relative importance and availability of each modality. For example, assuming there are three modalities: vision (0.8), speech (0.5), and touch (0.2), the sorted modality availability sequence would be [vision, speech, touch].

[0041] Furthermore, the goal of the threshold segmentation algorithm is to divide the modal availability sequence into several distinct intervals based on the modal availability values. First, the algorithm sets a threshold, typically determined by the range of availability values; for example, the threshold could be set to 0.5. Next, the algorithm segments the modal availability sequence based on the relationship between each value in the sequence and the threshold. Specifically, all modalities with availability values ​​above the threshold are grouped together, and all modalities with availability values ​​below the threshold are grouped together, forming two initial modal availability intervals. For example, assuming the modal availability sequence is [visual 0.8, speech 0.5, tactile 0.2], and the threshold is 0.5, it can be segmented into two intervals: the first interval is [visual 0.8], and the second interval is [speech 0.5, tactile 0.2].

[0042] Furthermore, for each modal availability interval, the purpose of recursive processing is to further optimize the interval division. During each recursion, the variance and distribution density of the modal availability values ​​within that interval are first calculated. Variance reflects the degree of fluctuation in modal availability values ​​within the interval; a large variance indicates significant differences in modal availability within the interval, potentially requiring further subdivision. Distribution density describes the concentration of modal availability values; higher density indicates that the availability values ​​of most modalities are similar, while lower density indicates a more dispersed distribution. For example, if a modal availability interval contains [visual 0.8, speech 0.7, tactile 0.9], the availability values ​​of these modalities are relatively concentrated, with potentially small variances and high distribution densities, and the recursive algorithm might decide not to further subdivide this interval. Conversely, if an interval contains [visual 0.8, speech 0.2, tactile 0.9], due to its large variance, it might continue to be recursively subdivided into smaller sub-intervals.

[0043] Secondly, during the recursive process, the calculation results of variance and distribution density determine whether the interval boundaries need to be adjusted. If the variance of modal availability values ​​within a modal availability interval is large, it indicates that the modal availability varies significantly within that interval, and it may be necessary to adjust the interval boundaries to allocate these modalities to different intervals. Specifically, the algorithm dynamically adjusts the boundaries based on the magnitude of the variance, making the modal availability values ​​within each interval closer together, thereby optimizing the interval division. For example, if during the recursive process it is found that an interval contains modalities with significant differences in quality (such as visual 0.9 and tactile 0.2), this interval may be further subdivided into two sub-intervals, one containing the visual modality and the other containing the tactile modality.

[0044] Furthermore, the core idea of ​​the sparse gating strategy is to dynamically select the most relevant or most important modalities for activation. After recursive processing, the resulting modality availability interval will contain several modalities. These modalities can be divided into core activation modalities and auxiliary modalities based on their availability values ​​and relevance to the current task. Core activation modalities are the most important and provide the most useful information, while auxiliary modalities are used as supplementary information sources in specific situations. For example, if the visual modality has the highest availability value when performing a fine-grained task, then the visual modality will be selected as the core activation modality. In certain specific situations, such as when vision is interfered with, the speech or tactile modalities may be selected as auxiliary modalities to help compensate for the lack of information in the visual modality.

[0045] Finally, conflict detection is used to check for inconsistencies or conflicts in information between different modalities. For example, the visual modality and the tactile modality may give different perceptual results for the same object in certain situations. The purpose of conflict detection is to determine whether such differences will affect the robot's decision-making and behavior. If the conflict is minor, the system may choose to merge the core activation modality and the auxiliary modality to make full use of the advantages of each modality; if the conflict is significant, it may adjust the priority of the modalities or select one of them for activation. For example, if the visual modality detects an obstacle while the tactile modality does not perceive it, conflict detection may analyze the accuracy of the visual information and decide whether to prioritize the visual data, or whether both need to be weighed and fused into a final perceptual result.

[0046] S3, the Kalman filter algorithm is used to evaluate the state of the active mode set, obtain the state evaluation set, and identify the mode switching state based on the state evaluation set;

[0047] In this example, the Kalman filter algorithm is used to evaluate the state of the active mode set, resulting in a state evaluation set. The mode switching state is then identified based on this state evaluation set. Specifically:

[0048] Define the state variables and observation variables for each mode in the activation mode set, and initialize the state estimation matrix and error covariance matrix of the Kalman filter;

[0049] Based on the data update frequency of each mode, set the time interval between the prediction step and the update step of the Kalman filter;

[0050] At each time interval, the state of each mode is predicted and updated using the Kalman filter algorithm to obtain the state estimate and covariance matrix of each mode;

[0051] The error variance of the state estimate for each mode is calculated based on the covariance matrix, and a weighted average of the state estimates for each mode is performed based on the error variance to obtain the state evaluation set.

[0052] The state change trend of each mode in the state assessment set is analyzed and the state difference is calculated. When the state difference exceeds the preset threshold, it is identified as a mode switching state.

[0053] It's important to note that in Kalman filtering, the state and observation variables for each mode are used to describe the dynamic characteristics and measurement data of that mode. For each active mode, the state variable can include the current state of the mode. For example, the state variable for the visual mode might be image sharpness, object recognition location, etc.; the state variable for the speech mode might include speech signal intensity, noise level, etc.; and for the tactile mode, it might represent perceived pressure or the intensity of contact. The observation variables are the measurement data directly obtained from the sensor; they are the actual measured values ​​related to the state variables. For the visual mode, the observation variable might be image data or a depth map; for the speech mode, the observation variable might be the spectral characteristics of the sound signal captured by the microphone; and for the tactile mode, the observation variable might be the pressure value output by the sensor. When initializing the Kalman filter, the state estimation matrix needs to be set. This represents a preliminary estimate of the mode state and can usually be set to zero or initialized based on initial measurements. Meanwhile, the error covariance matrix is ​​used to represent the uncertainty of the initial state estimation. When the initial error is large, a higher covariance value can be assigned. As the Kalman filtering proceeds, the covariance matrix will gradually decrease, reflecting the improvement in the estimation accuracy.

[0054] Secondly, Kalman filtering is based on time-series data for prediction and updating, and the data update frequency affects the time interval between prediction and update. The data update frequency for each modality is determined by the data acquisition frequency of the sensor for that modality. For example, the visual modality might update 30 frames of images per second, the speech modality might update speech data 5 times per second, and the tactile modality might update pressure values ​​10 times per second. Based on the update frequency of each modality, the time interval between prediction and update steps in the Kalman filter can be set. For higher-frequency modalities (such as vision), the time interval is shorter, and the filter will predict and update more frequently. For lower-frequency modalities (such as speech), the update interval may be slightly longer, and the filter's prediction step size will be correspondingly longer. For example, if visual data updates 30 frames per second, the prediction step size might be 1 / 30 of a second; while for speech data updated 5 times per second, the prediction step size would be 1 / 5 of a second.

[0055] Furthermore, within each time interval of the Kalman filter, a prediction step is first performed. Based on the state estimate from the previous step and the control input, the prediction equation of the Kalman filter is used to calculate the predicted state value at the current time. Then, based on this predicted value and the current observation data, the state is corrected using the Kalman update equation to obtain an updated state estimate. During this process, the Kalman filter also calculates the covariance matrix, representing the confidence level of the state estimate. The covariance matrix reflects the error of the current estimate. If the current data deviates significantly from the predicted value, the covariance value will be high, indicating greater uncertainty in the estimation of the modality state; if the data matches the predicted value, the covariance will be low, indicating a more accurate estimate. For example, if the predicted state of the visual modality is an image sharpness of 0.8, while the observed value is 0.85, the Kalman filter will correct the sharpness estimate based on this difference and calculate the updated sharpness estimate and covariance.

[0056] Furthermore, the diagonal elements of the covariance matrix represent the error variance of the state estimation. A larger error variance indicates higher uncertainty in the state estimation, and vice versa. During the Kalman filter update process, a weighted average of the state estimates for each modality is calculated based on the difference between the observed and predicted values ​​and the covariance matrix. The core idea of ​​weighted averaging is to adjust the weights of each modality's state based on the error variance in the covariance matrix. Modalities with smaller error variances have larger weights, meaning their state estimates will have a greater impact on the final state assessment set; conversely, modalities with larger error variances have smaller weights. For example, in the state assessment set, the error variance of the visual modality might be smaller (e.g., 0.1), while the error variance of the speech modality might be larger (e.g., 0.5). Therefore, in weighted averaging, the visual modality's estimate will have a larger weight, resulting in a more accurate overall state assessment set.

[0057] Finally, state dissimilarity is an indicator used to determine whether there are significant differences in state changes across different modalities. Specifically, after the state estimates of each modality are weighted and averaged, the state evaluation set contains state information from multiple modalities. By analyzing the trends in these state estimates, it can be determined whether a modality has undergone significant changes. If the state estimates of some modalities change rapidly and significantly within a time window, it may lead to an increase in state dissimilarity. For example, the clarity of the visual modality may suddenly drop from 0.9 to 0.5, while the signal strength of the speech modality may remain stable. This indicates that the state of the visual modality has changed significantly. When the state dissimilarity exceeds a preset threshold, it can be identified as a modality switching state, meaning that the current perceptual information may need to switch from one modality to another, or the activation strategy of the modality needs to be adjusted.

[0058] S4. When a modal switch is detected, a state evaluation window is constructed based on the task complexity and progress changes, and the robot state space dimension is dynamically adjusted within the state evaluation window to obtain a multimodal state set.

[0059] In this example, when a mode switch is detected, a state evaluation window is constructed based on task complexity and progress changes, specifically as follows:

[0060] When a mode switch is detected, obtain the current task complexity and the rate of change of task progress;

[0061] The task complexity is calculated based on the number of task objectives and their dependencies, and the task progress change rate is the ratio of completed task steps to the total number of steps.

[0062] The size and location of the status assessment window are determined based on the task complexity and the rate of change in task progress.

[0063] It's important to note that during a robot's task execution, task complexity and rate of progress are crucial parameters reflecting the current task status. Task complexity measures the difficulty of a task, typically related to the number of task objectives and their interdependencies. For example, if a robot is performing an assembly task involving assembling multiple parts in a dependent order, the task complexity depends not only on the number of objectives but also on the dependencies between them. If multiple parts are interconnected, and the assembly of each part affects the assembly of the next, the task complexity is relatively high. For instance, a simple task involving the assembly of 3 parts with no dependencies, versus a task involving 10 parts with strict dependencies on their assembly order, clearly demonstrates higher task complexity for the latter.

[0064] Secondly, the task progress rate reflects the speed at which a task is completed; it is the ratio between the currently completed task steps and the total number of task steps. If the robot is performing a task with multiple steps (such as path planning, obstacle avoidance, and target arrival in a navigation task), then the task progress rate represents the proportion of these steps that have been completed. For example, if a task has 10 steps and the robot has completed 3 steps, then the task progress rate is 30%. The progress rate can dynamically change over time, helping to determine whether the task is progressing at the expected speed. When a modal switch is detected during task execution (e.g., switching from visual to tactile mode due to poor lighting), task complexity and the progress rate become the basis for calculating adjustment strategies. These two parameters determine how to dynamically adjust the robot's perception modes during task execution to better cope with task complexity and the current progress status.

[0065] In this example, the size and position of the status evaluation window are determined based on the task complexity and the rate of change of task progress, as follows:

[0066] The initial length of the state evaluation queue is determined based on the task complexity, and the sampling interval of the elements in the state evaluation queue is set based on the task progress change rate.

[0067] Obtain the robot's state vector and corresponding timestamp, and construct state evaluation points based on the robot's state vector and corresponding timestamp;

[0068] State evaluation points are inserted into the state evaluation queue according to the sampling interval. When the state evaluation queue reaches its initial length, a preset queue full handling mechanism is activated to remove the first state evaluation point at the head of the state evaluation queue and insert a new state evaluation point.

[0069] Extract the timestamps from the head and tail of the state evaluation queue, calculate the difference between the timestamps as the size of the state evaluation window, and use the middle time point of the state evaluation queue as the center position of the state evaluation window.

[0070] It's important to note that during task execution, task complexity directly impacts the robot's ability and frequency of processing state assessment data. Higher task complexity requires the robot to process more sensory information and make more nuanced judgments, necessitating more frequent updates to the state assessment queue to adjust task execution strategies promptly. For example, in a complex assembly task, the robot might need to monitor multiple sensor data in real-time and quickly adjust its actions, while in a simple path planning task, the robot's update frequency of sensory information might be lower. With higher task complexity, the initial length of the state assessment queue may be set larger to store more historical state data for subsequent analysis and decision-making. For simple tasks, the initial queue length can be smaller. The task progress change rate affects the sampling interval of elements in the state assessment queue. A higher task progress change rate means the task is completed quickly, requiring the robot to update state assessment points more frequently; conversely, a lower task progress change rate allows for a relatively longer sampling interval, and the update frequency of state assessment points can be appropriately reduced. For example, if the task is halfway complete, the state evaluation point may be sampled every 0.5 seconds; while if the task is progressing slowly, it may be sampled every 1 second.

[0071] Suppose a robot is performing a complex multi-objective path planning task with strong dependencies between objectives and high task complexity. A large state evaluation queue (e.g., length 50) might be set. If the task progresses quickly (e.g., the arrival time of each objective is short), the sampling interval can be set to 0.2 seconds; if the task progresses slowly, a longer sampling interval (e.g., 0.5 seconds) can be set.

[0072] In each time period, the robot's state vector contains its current state information, such as position, velocity, attitude, and sensor data. Each time it samples, the robot obtains a new state vector based on its sensors and internal state updates. The timestamp indicates the sampling time of the current state vector, ensuring that all state data can be organized and used in chronological order. A state evaluation point is a data structure composed of the state vector and its corresponding timestamp. Each state evaluation point includes the robot's state information and the time point in which that state information was sampled. In this way, the robot can refer to historical state data and timestamps to perform state evaluations and adjust decisions in subsequent decision-making processes.

[0073] Furthermore, as the task progresses, the robot continuously acquires state data from various sensory modalities and adds new state evaluation points to the state evaluation queue according to the sampling interval. This queue stores state evaluation points in chronological order. If the queue reaches its initially set length, and the arrival of new state evaluation points exceeds the queue's capacity, queue full handling is required. The basic rule of the queue full handling mechanism is to remove the first state evaluation point (i.e., the earliest state point) from the queue and insert the newly sampled state evaluation point at the end of the queue. In this way, the queue always retains only the latest state data and discards older data, ensuring that the state evaluation points in the queue are always up-to-date.

[0074] Finally, once enough state evaluation points have accumulated in the queue, the next step is to determine the size and center position of the state evaluation window. The size of the state evaluation window is typically determined by calculating the difference between the timestamps at the head and tail of the queue. This difference reflects the robot's activity range over a past period. The size of the state evaluation window directly affects the range of historical states referenced in subsequent analysis and decision-making. The center position of the state evaluation window is determined by the time point in the middle of the queue. This ensures that the window covers a time range that includes both past data and reflects current state trends. The time point in the middle of the queue represents a relatively balanced state, reflecting the robot's main state changes over a past period. Assuming the queue stores 10 state evaluation points, with the timestamp at the head of the queue being "2023-11-26 14:00:00" and the timestamp at the tail being "2023-11-26 14:00:05", then the timestamp difference is 5 seconds, which is the size of the state evaluation window. The middle time point in the queue might be "2023-11-26 14:00:02", which will become the center of the state evaluation window. In this way, the robot can refer to the state changes over the past 5 seconds to make subsequent decisions and adjustments.

[0075] In this example, the robot's state space dimensions are dynamically adjusted within the state evaluation window to obtain a multimodal state set, specifically:

[0076] Within the state evaluation window, extract the robot state vectors of all state evaluation points from the state evaluation queue and construct the state matrix;

[0077] The state matrix is ​​transformed by matrix transpose to obtain the transposed state matrix;

[0078] The row vector distribution of the transposed state matrix is ​​analyzed, and a multimodal state set is constructed based on the analysis results.

[0079] It's important to note that within the state evaluation window, we need to extract all state evaluation points from the state evaluation queue. Each state evaluation point contains the robot's state vector at a specific moment, typically including multi-dimensional information such as the robot's position, velocity, attitude, and sensor data. To analyze and process this information, we organize these state vectors in chronological order, forming a state matrix. Each row represents the robot's state corresponding to a state evaluation point, while each column represents a specific attribute in the state vector, such as position, velocity, or sensor data.

[0080] Furthermore, the matrix transpose operation swaps the rows and columns of the state matrix, resulting in a new matrix. This new matrix helps us analyze the robot's state from different perspectives. The transpose operation transforms the various state attributes (such as position, velocity, and attitude) that were originally in rows into columns, thus concentrating the different attributes of each state evaluation point into a new structure. Using this transposed matrix, we can easily observe the trend of each attribute over time. For example, the first column represents the robot's x-position at each moment, the second column represents the y-position, the third column represents the velocity, and the fourth column represents the attitude.

[0081] In this example, the row vector distribution of the transposed state matrix is ​​analyzed, and a multimodal state set is constructed based on the analysis results, specifically:

[0082] The robot state vector is dynamically recombined based on the row vector distribution of the transposed state matrix to obtain the recombined robot state vector.

[0083] The robot's perception requirements after mode switching are obtained, and the trend of column vector change in the state matrix is ​​analyzed based on the robot's perception requirements to obtain the time series pattern of the robot's state vector.

[0084] The dimensions of the recombined robot state vector are dynamically adjusted based on the time series pattern of the robot state vector to obtain the adjusted robot state vector.

[0085] Integrate all adjusted robot state vectors to construct a multimodal state set.

[0086] It's important to note that the transposed state matrix provides the distribution of the robot's various state attributes at different points in time. To dynamically reorganize the robot's state vectors, we first need to analyze the changing trends of these row vectors over time. By examining these row vectors, we can identify which state attributes exhibit similar change patterns over time and which show significant differences. The core of the reorganization process is to adjust these state attributes according to their performance in the time series. For example, if we find that changes in the robot's position are closely related to changes in velocity, we can merge the position and velocity row vectors to form a new state vector, thus better reflecting the robot's motion patterns. Similarly, changes in posture may also be reorganized into a new state vector to more accurately represent the robot's posture adjustment process.

[0087] Secondly, modal switching typically leads to changes in a robot's perception requirements. For example, when switching from a visual modality to a tactile modality, the robot's perception requirements for the environment shift from image data to pressure-sensing data. Based on these new perception requirements, the robot will change its dependent perception modality and adjust its state evaluation dimensions accordingly. In this process, we analyze the column vectors in the state matrix (i.e., the temporal trends of various state attributes). Each column represents a specific state attribute of the robot at different points in time. By comparing the changing patterns of different column vectors, we can identify which perception requirements become more important and which may be ignored or have a reduced impact. For instance, when switching from a visual modality to a tactile modality, position and posture may become more important, while visually relevant features (such as image sharpness) may be ignored. Through this analysis, we can form a new time-series pattern reflecting the robot's dynamic requirements for perception data during task execution.

[0088] Furthermore, as the perception requirements change after modal switching, the dimensions of the robot's state vector need to be dynamically adjusted. This adjustment is based on the robot's current perception needs and the characteristics of the task execution. For example, in visual modality, the state vector might include multiple dimensions such as position, velocity, image sharpness, and object recognition results; however, when switching to tactile modality, only dimensions such as position, velocity, and tactile pressure may need to be considered. The process of adjusting the dimensions is based on the previously analyzed time-series patterns, identifying which perception dimensions (e.g., image sharpness, pressure, temperature) are most important to the current task and retaining these dimensions. Simultaneously, dimensions that are not very useful in the current task phase are removed.

[0089] S5 generates robot task execution instructions and controls the robot's behavior based on the multimodal state set.

[0090] In this example, robot task execution instructions are generated based on a multimodal state set, and the robot's behavior is controlled as follows:

[0091] Analyze the multimodal state set, extract the robot's current state vector, and determine the range of possible actions based on the robot's current state vector;

[0092] Based on the actionable range and the preset task objectives, a preset breadth-first search algorithm is used to generate candidate action sequences.

[0093] The robot acquires information on dynamic constraints and environmental obstacles, evaluates the feasibility of candidate action sequences, and obtains the optimal action sequence.

[0094] The task execution instructions are generated based on the optimal sequence of actions and sent to the robot controller to control the robot's actions in real time.

[0095] It's important to note that the core task of parsing a multimodal state set is to extract all the robot's state information at the current moment and determine the range of actions the robot can perform based on this information. The multimodal state set contains state data of the robot across multiple sensory modalities, such as position, velocity, posture, and sensor information. By analyzing this state data, especially the robot's current position, velocity, and posture, the robot can assess its current "state" and its feasible range of actions. For example, suppose the robot's current state vector includes position (x, y), velocity (v), posture (θ), and sensor feedback. Based on this data, the robot's current position and velocity can be determined, thus determining which actions the robot can perform in the next time step. For instance, if the robot's current velocity is slow and there are no obstacles around, it can choose a wider range of movements; if the robot is in a confined environment, its range of actions may be limited, and it may only be able to choose a path around obstacles or make small adjustments.

[0096] Secondly, after determining the robot's current range of possible actions, the next step is to generate candidate action sequences based on the preset task objective. The task objective might include reaching a designated location, avoiding obstacles, or completing a specific operation. Based on this, the Breadth-First Search (BFS) algorithm can be used to explore all possible paths from the current state to the target state. BFS is a layer-by-layer search method that starts from the current state and gradually approaches the target state, beginning with the shortest path. Using the BFS algorithm, the robot can generate a series of candidate action sequences. Each candidate sequence starts from the current state and gradually approaches the task objective by selecting a series of possible actions (such as moving forward, turning, stopping, etc.). The selection of each action considers not only the current state but also the robot's next state and the requirements of the objective. For example, if the robot's task objective is to move from the current position (5, 5) to (10, 10), and its range of actions includes moving forward, backward, turning left, and turning right, the robot can use the BFS algorithm to generate candidate action sequences. One possible sequence is: "Forward -> Turn Right -> Forward -> Turn Right -> Forward".

[0097] Furthermore, by acquiring information on robot dynamic constraints and environmental obstacles, candidate action sequences can be further filtered and optimized. Dynamic constraints include the robot's physical limitations, such as maximum speed, acceleration, and turning angle, while environmental obstacle information provides information on the position and size of obstacles around the robot, all of which may limit the robot's actions. When evaluating the feasibility of candidate action sequences, the robot needs to check whether each action sequence meets these constraints. For example, some actions may cause the robot to exceed its maximum speed limit, or some paths may cross the space of obstacles, leading to a collision risk. In this way, the robot can evaluate candidate action sequences, eliminate those that are not feasible, and select the optimal action sequence that meets all constraints. Suppose the robot's current position is (5, 5), and the task objective is to move to (10, 10). The candidate sequence generated by the breadth-first search algorithm is: "Forward -> Right Turn -> Forward -> Right Turn -> Forward". However, when performing the "Right Turn" action, the robot will approach an obstacle. If the obstacle is too close to the robot's turning radius, the robot will determine that this path is not feasible and eliminate this candidate action sequence. Ultimately, the robot chose a path that avoided the obstacles and ensured that all actions were performed within dynamic constraints.

[0098] Once the optimal sequence of actions is determined, the next step is to translate these actions into specific task execution instructions and execute them in real time through the control system. These instructions typically include specific actions and their timing, such as instructing the robot to move at a specific speed, direction, and acceleration, or to perform certain operations (such as grasping or placing). The control instructions need to be adjusted appropriately based on the robot's kinematic model and dynamic constraints to ensure that the robot can successfully complete the task.

[0099] When task execution instructions are sent to the robot's controller, the controller schedules the robot's execution process in real time based on these instructions, ensuring that each action is performed in the planned order and timing. The controller also needs to dynamically adjust its behavior based on the robot's real-time status feedback to ensure that the robot can adapt to changes in the environment and avoid collisions or failures.

[0100] For example, suppose the robot selects the action sequence: "Forward -> Right Turn -> Forward". Based on this sequence, the control system will generate corresponding task execution instructions, such as: "Forward 0.5 meters", "Right Turn 90 degrees", "Forward 0.5 meters". These instructions are sent to the robot's controller in real time. The controller adjusts the robot's trajectory according to its current position and posture, ensuring that the robot moves along the predetermined path and eventually reaches the target position.

[0101] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, in the form of a computer program product.

[0102] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0103] In addition, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0104] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0105] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multimodal perception fusion method for humanoid robots based on dynamic sparse activation, characterized in that, Includes the following steps: The process involves acquiring multimodal data streams from the robot and generating modal availability maps based on the robot's real-time operating status. Specifically, this includes: acquiring the robot's multimodal data streams and extracting modal feature vectors for each modal data; acquiring the robot's real-time operating status and calculating the data quality assessment value of the modal feature vectors based on the real-time operating status; assigning modal availability values ​​to each modal feature vector based on the data quality assessment values ​​and generating modal availability maps based on the modal availability values. Based on modal availability mapping, a sparse gating strategy is used to dynamically select the perceptual modalities to be activated and construct an activated modal set according to the selection results. Specifically, the modal availability values ​​of all modalities are extracted from the modal availability mapping and sorted according to the modal availability values ​​to generate a modal availability sequence. A threshold segmentation algorithm is used to initially segment the modal availability sequence to obtain several modal availability intervals. For each modal availability interval, perform several recursive processes, and calculate the variance and distribution density of the modal availability values ​​within the modal availability interval during each recursive process; dynamically adjust the interval boundaries based on the variance and distribution density of the modal availability values ​​within the modal availability interval to obtain the recursed modal availability interval; A sparse gating strategy is used to dynamically select the core active mode and auxiliary modes from the recursive modality availability interval; Conflict detection is performed on the core activation mode and auxiliary mode, and the core activation mode and auxiliary mode are merged according to the conflict detection results to obtain the activation mode set; The Kalman filter algorithm is used to evaluate the state of the active mode set, and the state evaluation set is obtained. The mode switching state is then identified based on the state evaluation set. When a modal switch is identified, a state evaluation window is constructed based on the task complexity and progress changes, and the robot's state space dimension is dynamically adjusted within the state evaluation window to obtain a multimodal state set. Generate robot task execution instructions and control robot behavior based on multimodal state sets.

2. The humanoid robot multimodal perception fusion method based on dynamic sparse activation according to claim 1, characterized in that, The process of using the Kalman filter algorithm to evaluate the state of the active mode set, obtaining a state evaluation set, and identifying the mode switching state based on the state evaluation set is as follows: Define the state variables and observation variables for each mode in the activation mode set, and initialize the state estimation matrix and error covariance matrix of the Kalman filter; Based on the data update frequency of each mode, set the time interval between the prediction step and the update step of the Kalman filter; At each time interval, the state of each mode is predicted and updated using the Kalman filter algorithm to obtain the state estimate and covariance matrix of each mode; The error variance of the state estimate for each mode is calculated based on the covariance matrix, and a weighted average of the state estimates for each mode is performed based on the error variance to obtain the state evaluation set. The state change trend of each mode in the state assessment set is analyzed and the state difference is calculated. When the state difference exceeds the preset threshold, it is identified as a mode switching state.

3. The humanoid robot multimodal perception fusion method based on dynamic sparse activation according to claim 2, characterized in that, When a mode switch is identified, a state evaluation window is constructed based on task complexity and progress changes, specifically as follows: When a mode switch is detected, obtain the current task complexity and the rate of change of task progress; The task complexity is calculated based on the number of task objectives and their dependencies, and the task progress change rate is the ratio of completed task steps to the total number of steps. The size and location of the status assessment window are determined based on the task complexity and the rate of change in task progress.

4. The humanoid robot multimodal perception fusion method based on dynamic sparse activation according to claim 3, characterized in that, The determination of the size and position of the status evaluation window based on task complexity and task progress change rate is as follows: The initial length of the state evaluation queue is determined based on the task complexity, and the sampling interval of the elements in the state evaluation queue is set based on the task progress change rate. Obtain the robot's state vector and corresponding timestamp, and construct state evaluation points based on the robot's state vector and corresponding timestamp; State evaluation points are inserted into the state evaluation queue according to the sampling interval. When the state evaluation queue reaches its initial length, a preset queue full handling mechanism is activated to remove the first state evaluation point at the head of the state evaluation queue and insert a new state evaluation point. Extract the timestamps from the head and tail of the state evaluation queue, calculate the difference between the timestamps as the size of the state evaluation window, and use the middle time point of the state evaluation queue as the center position of the state evaluation window.

5. The humanoid robot multimodal perception fusion method based on dynamic sparse activation according to claim 4, characterized in that, The process of dynamically adjusting the robot's state space dimension within the state evaluation window to obtain a multimodal state set specifically involves: Within the state evaluation window, extract the robot state vectors of all state evaluation points from the state evaluation queue and construct the state matrix; The state matrix is ​​transformed by matrix transpose to obtain the transposed state matrix; The row vector distribution of the transposed state matrix is ​​analyzed, and a multimodal state set is constructed based on the analysis results.

6. The humanoid robot multimodal perception fusion method based on dynamic sparse activation according to claim 5, characterized in that, The analysis of the row vector distribution of the transposed state matrix and the construction of a multimodal state set based on the analysis results are as follows: The robot state vector is dynamically recombined based on the row vector distribution of the transposed state matrix to obtain the recombined robot state vector. The robot's perception requirements after mode switching are obtained, and the trend of column vector change in the state matrix is ​​analyzed based on the robot's perception requirements to obtain the time series pattern of the robot's state vector. The dimensions of the recombined robot state vector are dynamically adjusted based on the time series pattern of the robot state vector to obtain the adjusted robot state vector. Integrate all adjusted robot state vectors to construct a multimodal state set.

7. The humanoid robot multimodal perception fusion method based on dynamic sparse activation according to claim 6, characterized in that, The process of generating robot task execution instructions and controlling robot behavior based on multimodal state sets specifically includes: Analyze the multimodal state set, extract the robot's current state vector, and determine the range of possible actions based on the robot's current state vector; Based on the actionable range and the preset task objectives, a preset breadth-first search algorithm is used to generate candidate action sequences. The robot acquires information on dynamic constraints and environmental obstacles, evaluates the feasibility of candidate action sequences, and obtains the optimal action sequence. The task execution instructions are generated based on the optimal sequence of actions and sent to the robot controller to control the robot's actions in real time.

Images