A method for processing multimodal data of a robot, an electronic device and a storage medium

Abstract

This invention relates to the field of data processing technology, and more particularly to a method, electronic device, and storage medium for processing multimodal data of a robot. This invention evaluates the quality of each individual modal data point and the overall correlation between multiple modal data points, thereby achieving a comprehensive assessment of the quality of multimodal data. Low-quality or conflicting modal data is identified before data fusion or use, enabling corresponding processing operations on modal data of different qualities to obtain high-quality modal datasets, thus improving the robustness of robot environmental perception and the success rate of task execution.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method for processing robot multimodal data, an electronic device, and a storage medium. Background Technology

[0002] Robots have become a bridge connecting the physical and digital worlds, playing a vital role in various fields such as homes, industry, healthcare, and education. Robots need to be able to understand and respond to complex natural language commands, while processing data from multiple sensors, including vision, hearing, and touch, to achieve highly realistic and natural interaction.

[0003] In related technologies, the governance of robot multimodal data still relies on independent evaluation of single modalities in similar general datasets and a simple binary governance strategy of "keeping" or "discarding" data. This approach results in significant data waste, and the lack of a unified, hierarchical quality assessment mechanism leads to a decrease in the quality of multimodal data. Consequently, subsequent fusion or use of low-quality multimodal data poses significant data security risks, easily leading to problems such as unreliable data quality, data conflicts, and decision-making errors. Summary of the Invention

[0004] In view of this, one objective of the embodiments of the present invention is to provide a method for processing robot multimodal data, an electronic device, and a storage medium to improve the low quality of robot multimodal data in the related art.

[0005] To address the aforementioned technical problems, the embodiments of the present invention provide the following technical solutions:

[0006] In a first aspect, embodiments of the present invention provide a method for processing robot multimodal data, including:

[0007] Obtain the robot's target data set, which includes data from multiple modalities;

[0008] Calculate the quality score vector of the target dataset. The quality score vector includes multiple first quality scores and a reference quality score. Each first quality score represents the quality of the data in one modality, and the reference quality score represents the overall correlation of the data across multiple modalities.

[0009] The pre-built neural network model is trained based on sample data to obtain a quality level labeling model. The quality level labeling model is then used to determine the quality level vector corresponding to the quality score vector.

[0010] Perform target processing operations corresponding to the quality level vector on data from multiple modalities.

[0011] In some embodiments, calculating a quality score vector for a target dataset includes:

[0012] Based on the candidate evaluation function corresponding to the candidate modal data, the first quality score of the candidate modal data is calculated, where the first quality score characterizes the quality of the candidate modal data, and the candidate modal data is the data of any one modality among multiple modal data;

[0013] Based on the target metric of the reference modality dataset, a second quality score is calculated for the reference modality dataset. The reference modality dataset includes data from any two modalities among multiple modalities. The target metric represents the inconsistency between the data from two modalities in the reference modality dataset, and the second quality score represents the correlation between the data from two modalities in the reference modality dataset.

[0014] Based on multiple second quality scores, a reference quality score is calculated, which represents the overall correlation of data across multiple modalities.

[0015] A quality score vector is obtained based on multiple first quality scores and reference quality scores.

[0016] In some embodiments, a reference quality score is calculated based on a plurality of second quality scores, including:

[0017] Add all the second quality scores together to obtain the candidate quality scores;

[0018] Divide the first preset value by the first value to obtain the candidate value. The first value is the product of the second value and the third value. The second value is the number of modalities in the target dataset. The third value is the difference between the second value and the natural number 1.

[0019] The reference quality score is obtained by multiplying the candidate quality score by the candidate value.

[0020] In some embodiments, a reference quality score is calculated based on a plurality of second quality scores, including:

[0021] The minimum value among multiple second quality scores is determined as the reference quality score.

[0022] In some embodiments, a reference quality score is calculated based on a plurality of second quality scores, including:

[0023] Calculate the adjacency matrix and degree matrix of the quality score graph. The quality score graph is a graph obtained by treating each modality's data as a node and using the second quality score as the weight of the edge connecting the nodes. The size of the adjacency matrix is ​​the same as the size of the degree matrix.

[0024] Subtract the adjacency matrix from the degree matrix to obtain the reference matrix;

[0025] Calculate the algebraic connectivity of the reference matrix;

[0026] Normalize the algebraic connectivity to obtain the reference quality score.

[0027] In some embodiments, the data for two modes in the reference modal dataset are first modal data and second modal data. Based on the target metric of the reference modal dataset, a second quality score for the reference modal dataset is calculated, including:

[0028] Calculate the candidate similarity between the first modality data and the second modality data;

[0029] Subtract the candidate similarity from the second preset value to obtain the target metric value;

[0030] The target metric is normalized to obtain the second quality score.

[0031] In some embodiments, a first quality score of the candidate modal data is calculated based on a candidate evaluation function corresponding to the candidate modal data, including:

[0032] Based on the modality type of the candidate modality data, the evaluation function corresponding to the modality type is determined as the candidate evaluation function, and the candidate evaluation function includes at least one evaluation index;

[0033] Calculate the value of each evaluation indicator;

[0034] The first quality score is calculated based on at least one indicator value.

[0035] In some embodiments, determining the quality grade vector corresponding to the quality score vector using a quality grade labeling model includes:

[0036] The quality score vector is input into the quality level labeling model so that the quality level labeling model outputs a quality level vector corresponding to the quality score vector. The quality level vector includes multiple first quality levels and reference quality levels. Each first quality level corresponds to a first quality score, and each reference quality level corresponds to a reference quality score.

[0037] In a second aspect, embodiments of the present invention provide an electronic device, comprising:

[0038] A processor and a memory communicatively connected to the processor;

[0039] The memory stores computer program instructions executable by the processor, which, when executed by the processor, cause the electronic device to perform any of the robot multimodal data processing methods proposed in the first aspect.

[0040] Thirdly, embodiments of the present invention provide a computer-readable storage medium storing processor-executable computer program instructions, which, when executed by a processor, cause the computer to perform any of the robot multimodal data processing methods proposed in the first aspect.

[0041] The embodiments of the present invention have the following beneficial effects: Unlike related technologies, the robot multimodal data processing method provided by the embodiments of the present invention includes: acquiring a target data set of the robot, the target data set including data of multiple modalities; calculating a quality score vector of the target data set, the quality score vector including multiple first quality scores and a reference quality score, each first quality score representing the quality of data of a modality, the reference quality score representing the overall correlation of data of multiple modalities; training a pre-constructed neural network model based on sample data to obtain a quality level labeling model; using the quality level labeling model to determine the quality level vector corresponding to the quality score vector; and performing target processing operations corresponding to the quality level vector on the data of multiple modalities.

[0042] This invention evaluates the quality of each single modal data point and the overall correlation between multiple modal data points, thereby achieving a comprehensive assessment of the quality of multimodal data. It identifies low-quality or conflicting modal data before data fusion or use, enabling corresponding processing operations on modal data of different qualities to obtain high-quality modal datasets, thus improving the robustness of robot environmental perception and the success rate of task execution. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the prior art or embodiments will be briefly introduced below. Obviously, the drawings described below only show some embodiments of the present invention and should not be considered as limiting the scope of protection. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a schematic diagram illustrating an application scenario of the robot multimodal data processing method provided in some embodiments of the present invention;

[0045] Figure 2 These are schematic diagrams of the structure of an electronic device provided in some embodiments of the present invention;

[0046] Figure 3 This is a flowchart illustrating a method for processing multimodal robot data in some embodiments of the present invention;

[0047] Figure 4 yes Figure 3A schematic diagram of a sub-process of step S32 in the robot multimodal data processing method shown in the embodiment;

[0048] Figure 5 This is a schematic diagram of a quality fraction graph provided in some embodiments of the present invention. Detailed Implementation

[0049] To make the objectives and advantages of the embodiments of the present invention more readily understood, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. The detailed description of the embodiments of the present invention in the accompanying drawings is not intended to limit the scope of protection claimed by the present invention, but only to illustrate selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] It should be noted that, unless there is a conflict, the various technical features involved in the embodiments of the present invention described below can be combined with each other, and all are within the protection scope of the present invention. Furthermore, although functional modules are divided in the device or structural schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. In addition, the terms "first," "second," "third," and other similar expressions used herein do not limit the data or execution order, but are only for illustrative purposes and to distinguish identical or similar items with substantially the same function and effect, and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features.

[0051] Unless otherwise defined, the technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. It should be understood that the term "and / or" as used herein includes any and all combinations of one or more of the listed items.

[0052] Robots have become a bridge connecting the physical and digital worlds, playing a vital role in various fields such as homes, industry, healthcare, and education. Robots need to be able to understand and respond to complex natural language commands, while processing data from multiple sensors, including vision, hearing, and touch, to achieve highly realistic and natural interaction.

[0053] In related technologies, the governance of robot multimodal data still relies on independent evaluation of single modalities in similar general datasets and a simple binary governance strategy of "keeping" or "discarding". The inventors found that this approach causes serious data waste. Related technologies lack a unified and hierarchical quality assessment mechanism to evaluate multimodal data, resulting in a decrease in the quality of multimodal data. As a result, when subsequent fusion or use of low-quality multimodal data, problems such as unreliable data quality, data conflicts, and decision-making errors are likely to occur, posing significant data security risks.

[0054] In view of this, embodiments of the present invention provide a method for processing robot multimodal data. By evaluating the quality of each single modal data and assessing the overall correlation between multiple modal data, the quality of multimodal data can be comprehensively evaluated. Low-quality or conflicting modal data can be identified before data fusion or use, thereby enabling corresponding processing operations on modal data of different qualities to obtain high-quality modal datasets, improving the robustness of robot environmental perception and the success rate of task execution.

[0055] Please see Figure 1 , Figure 1 The illustration shows an application scenario diagram of the robot multimodal data processing method provided in some embodiments of the present invention.

[0056] like Figure 1 As shown, the application scenario includes an electronic device 100 and a robot 200. The electronic device 100 communicates with the robot 200 via a network, where examples of networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0057] For example, robot 200 is equipped with various types of sensors, such as lidar, cameras, and pressure sensors. Robot 200 uses these sensors to collect environmental data (e.g., image data, video data) and its own sensory data (e.g., pressure data and joint angle data), obtaining data in multiple modalities. Robot 200 packages this data into a target data set and sends it to electronic device 100 via a network, thereby enabling electronic device 100 to acquire the data in multiple modalities (i.e., the target data set).

[0058] The target dataset includes data from multiple modalities. It should be understood that engineers can divide the data into each modality based on their engineering experience. For example, visual modal data (including image data, video data, and LiDAR data), motion modal data (including joint angle data of the robotic hand of robot 200, posture data of the dexterous hand of robot 200), and tactile modal data (including pressure data from the tactile sensors in the dexterous hand of robot 200).

[0059] In this embodiment of the invention, after acquiring data from more than 200 modalities of the robot, the electronic device 100 parses and processes the data for each modality, calculating the quality score corresponding to the data for each modality based on the corresponding data indicators. Furthermore, it calculates the correlation degree of different modal data pairs, calculates the overall correlation degree of data across multiple modalities based on the correlation degrees of all modal data pairs, and obtains a reference quality score. Finally, it combines the quality scores of data from multiple modalities and the reference quality score to obtain a quality score vector. It should be understood that a modal data pair refers to a data pair consisting of any two modalities from multiple modalities.

[0060] For example, electronic device 100 is configured with a quality level labeling model 101. Electronic device 100 inputs the quality score vector into the quality level labeling model 101 and uses the quality level labeling model 101 to predict the quality level vector corresponding to the output quality score vector. Alternatively, after training a pre-built neural network model using sample data to obtain the quality level labeling model 101, electronic device 100 inputs the quality score vector into the quality level labeling model 101 and uses the quality level labeling model 101 to predict the quality level vector corresponding to the output quality score vector. It should be understood that the quality level vector includes the quality level corresponding to the quality score of each modality's data and the quality level corresponding to the reference quality score. Finally, electronic device 100 performs corresponding target processing operations on the data from multiple modalities based on the quality level vectors, removing low-quality or conflicting modal data to obtain a high-quality modal dataset.

[0061] It should be understood that Figure 1 The application scenario shown is merely an illustrative representation of how electronic device 100 processes data from more than 200 modalities of robot 200. Electronic device 100 is a laptop computer, and this does not limit the structure, type, or quantity of electronic device 100 and robot 200 in other application scenarios or embodiments. For example, in some other application scenarios or embodiments, electronic device 100 may also be a smartphone, tablet, desktop computer, or any other suitable type of device; alternatively, electronic device 100 may be a microcontroller, FPGA chip, or any other suitable type of electronic component.

[0062] In this embodiment of the invention, robot 200 can be a mobile robot based on a SLAM system, such as a cleaning robot, delivery robot, pet robot, handling robot, care robot, and remote monitoring robot. It is understood that cleaning robots include, but are not limited to, sweeping robots, vacuuming robots, mopping robots, and floor washing robots.

[0063] To facilitate understanding of the robot multimodal data processing method provided in the embodiments of the present invention, the electronic equipment provided in the embodiments of the present invention will first be described in detail.

[0064] Please see Figure 2 , Figure 2 The schematic diagram illustrates the structure of an electronic device provided by some embodiments of the present invention.

[0065] like Figure 2 As shown, the electronic device 100 includes at least one processor 110 and a memory 120 connected in communication. Figure 2 Taking a bus system 130 and a processor 110 as an example, the various components in the electronic device 100 are coupled together through the bus system 130, which is used to realize the connection and communication between the various components. It is easy to understand that the bus system 130 may include, in addition to the data bus, a power bus, a control bus, and a status signal bus, etc. However, for the sake of clarity and brevity, in... Figure 2 The general labels all buses as Bus System 130. Understandably, Figure 2 The structures shown in the embodiments are merely illustrative and do not limit the structure of the electronic device described above. For example, the electronic device may also include components that are larger than... Figure 2 The structure shown has more or fewer components, or has the same as Figure 2 The diagram shows different configurations of the structure.

[0066] Specifically, processor 110 provides computational and control capabilities to support electronic device 100 in executing corresponding business logic and functions. For example, it supports electronic device 100 in executing any of the robot multimodal data processing methods provided in the embodiments of the present invention, or in executing the steps of any possible implementation of any of the robot multimodal data processing methods provided in the embodiments of the present invention. Those skilled in the art will understand that processor 110 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0067] The memory 120, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, instructions, and modules, such as the programs, instructions, and modules corresponding to the robot multimodal data processing method of the embodiments of the present invention. In some embodiments, the memory 120 may include a program storage area and a data storage area. The program storage area may store an operating system, an application program required for at least one function, and the data storage area may store data created according to the use of the processor 110. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the non-transitory software programs, instructions, and modules stored in the memory 120, so as to implement any robot multimodal data processing method provided in the embodiments of the present invention, or execute the steps in any possible implementation of any robot multimodal data processing method provided in the embodiments of the present invention. In some embodiments, the memory 120 may include high-speed random access memory and may also include non-transitory memory. For example, at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 120 may also include memory remotely located relative to the processor 110, and these remotely located memories may be connected to the processor 110 through a communication network. It is understood that examples of the aforementioned communication networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0068] As can be understood from the above, the implementing entity of any robot multimodal data processing method provided in the embodiments of the present invention can be any suitable type of electronic device with certain computing and control capabilities, such as the aforementioned electronic device 100. In some feasible implementations, any robot multimodal data processing method provided in the embodiments of the present invention can be implemented by a processor executing computer program instructions stored in a memory.

[0069] The following will describe in detail the robot multimodal data processing method provided by the present invention, with reference to exemplary applications and implementations of the electronic devices provided in the embodiments of the present invention.

[0070] Please see Figure 3 , Figure 3 The schematic diagram illustrates a flowchart of a robot multimodal data processing method provided in some embodiments of the present invention.

[0071] Those skilled in the art will understand that the robot multimodal data processing method provided in the embodiments of the present invention can be applied to the above-mentioned electronic device (e.g., electronic device 100). Specifically, the execution subject of the robot multimodal data processing method is one or at least two processors of the electronic device.

[0072] like Figure 3As shown, the robot multimodal data processing method provided in this embodiment of the invention includes, but is not limited to, the following steps S31-S34:

[0073] S31: Obtain the target data set for the robot.

[0074] In this embodiment, the target dataset includes data from multiple modalities.

[0075] In this step, the robot is equipped with multiple types of sensors, which are configured to collect data from multiple modalities. In this embodiment of the invention, the data collected by each sensor is obtained from the network to obtain data from multiple modalities.

[0076] S32: Calculate the quality score vector of the target dataset.

[0077] In this step, the quality score vector includes multiple first quality scores and a reference quality score. Each first quality score represents the quality of data in one modality, and the reference quality score represents the overall correlation of data across multiple modalities.

[0078] For example, for each modality of data in the target dataset, quality assessment features that reflect data quality are extracted from the original modality data, and each modality of data forms a corresponding quality feature vector. These quality assessment features include, but are not limited to, integrity features (e.g., missing data rate, sampling completeness, effective frame rate), reliability features (e.g., noise intensity, signal-to-noise ratio, outlier ratio, drift degree), resolution features (e.g., spatial resolution, temporal resolution, data density), task relevance features (e.g., the contribution of modality data to the current task objective), and temporal continuity features (e.g., cross-frame consistency and abrupt change).

[0079] In some embodiments, for each modality of data, its quality feature vector is scored using a preset scoring method to obtain a first quality score for that modality of data. Obviously, the scoring method includes, but is not limited to, weighted scoring methods based on indicators, methods based on statistical measures (such as variance, entropy, and bias), inference based on a trained quality assessment model, and segmented scoring rules based on domain experience. After scoring the quality feature vector of each modality of data, a first quality score for each modality of data is obtained.

[0080] Specifically, embodiments of the present invention measure the correlation between multiple modal data to obtain a modal correlation quality score, which reflects the degree of collaboration between different modal data. The correlation strength can be achieved through one or a combination of the following methods: based on statistical correlation (such as Pearson correlation, mutual information, covariance, etc.), based on cross-modal consistency indicators (such as synchronicity, overlap, cross-modal mutual verification rate), based on cross-inference consistency (such as the semantic consistency between visual inference results and speech recognition), and based on the fusion confidence of the trained cross-modal fusion model, ultimately obtaining a reference quality score for the multimodal data.

[0081] In some embodiments, the independent first quality scores of each modality data and the reference quality scores of the multimodal data are combined in a uniform vector form to construct a quality score vector of the target dataset, such as a quality score vector. ,in, Indicates the first Quality score of each modality data point . This represents a reference quality score for multimodal data, used to characterize the quality of collaborative performance of multimodal data.

[0082] S33: Train the pre-built neural network model based on sample data to obtain the quality level labeling model, and use the quality level labeling model to determine the quality level vector corresponding to the quality score vector.

[0083] In this embodiment, the quality grade labeling model is a model obtained by training a pre-built neural network model using sample data until it meets preset conditions. Based on the characteristics of the quality grade labeling task, a neural network model is pre-designed and trained using a sufficient amount of sample data. Specifically, the loss is calculated based on the predicted quality grade output by the model and the quality grade corresponding to the actual modality data quality score during training. The model parameters are adjusted based on the loss, and the neural network model is iteratively trained until it meets preset convergence conditions, resulting in a trained quality grade labeling model. This model is then configured to predict quality scores and obtain the corresponding quality grade. The sample data includes a sample quality score vector and a corresponding sample quality grade vector. The sample quality score vector includes multiple quality scores, and the sample quality grade vector includes multiple quality grades. Each quality score in the sample quality score vector corresponds to each quality grade in the sample quality grade vector.

[0084] Understandably, the above content only briefly illustrates how to train a pre-built neural network model using sample data to meet preset conditions and obtain a quality level labeling model. In the actual training process, it is necessary to perform multi-dimensional and comprehensive training, parameter tuning, and loss calculation on the neural network model to finally obtain the quality level labeling model.

[0085] For example, the calculated quality score vector is input into the trained quality level labeling model. The quality level labeling model determines the quality level based on the quality score vector and outputs the corresponding quality level vector, such as the quality level vector. Each quality level in the quality level vector corresponds to each quality score in the quality score vector, i.e., the first quality level. Corresponding to the first quality score The second quality level Corresponding to the second quality score ... and so on, the... Quality levels With the mass fraction Correspondingly, the first Quality levels With the mass fraction correspond.

[0086] S34: Perform target processing operations corresponding to the quality level vector on data from multiple modalities.

[0087] For example, after obtaining the quality level vector of the target dataset, target processing operations corresponding to its quality level are performed on the data of each modality to achieve quality-driven multimodal data processing. The target processing operations are determined based on the quality level corresponding to the data of each modality in the quality level vector and the overall correlation quality level between multiple modalities. In some embodiments, multiple quality levels are defined for each modality, such as A, B, C, D, and E, where A, B, C, D, and E represent excellent, good, average, poor, and unusable, respectively. Corresponding data processing strategies are preset for different quality levels, including but not limited to operations such as use, enhancement, calibration, replacement, repair, compensation, and discard. For example, for modal data with a quality level below a preset threshold, target processing operations such as noise suppression, outlier removal, interpolation completion, and data augmentation are performed to ensure improved usability without losing information.

[0088] This invention evaluates the quality of each single modal data point and the overall correlation between multiple modal data points, thereby achieving a comprehensive assessment of the quality of multimodal data. It identifies low-quality or conflicting modal data before data fusion or use, enabling corresponding processing operations on modal data of different qualities to obtain high-quality modal datasets, thus improving the robustness of robot environmental perception and the success rate of task execution.

[0089] Please see Figure 4 , Figure 4 The illustration shows a sub-process diagram of step S32 in the robot multimodal data processing method provided in some embodiments of the present invention.

[0090] like Figure 4 As shown, in some embodiments, the quality score vector of the target dataset is calculated, specifically including but not limited to the following steps S321-S324:

[0091] S321: Calculate the first quality score of the candidate modal data based on the candidate evaluation function corresponding to the candidate modal data.

[0092] The first quality score characterizes the quality of the candidate modal data, which is any one of the multiple modal data.

[0093] For example, data from any one modality among multiple modalities is selected as candidate modal data. A corresponding candidate evaluation function is determined based on the candidate modal data. The evaluation index corresponding to the candidate modal data in the candidate evaluation function is used to calculate the index value of each evaluation index. Based on the index values ​​of all evaluation indexes, the first quality score of the candidate modal data is calculated.

[0094] In some embodiments, a first quality score of the candidate modal data is calculated based on a candidate evaluation function corresponding to the candidate modal data, specifically including but not limited to the following steps S3211-S3213:

[0095] S3211: Based on the modality type of the candidate modality data, determine the evaluation function corresponding to the modality type as the candidate evaluation function.

[0096] In this embodiment, the candidate evaluation function includes at least one evaluation index. The evaluation index is used to evaluate the quality of candidate modal data. The candidate evaluation function is used to measure the quality of data of a single modality. Engineers can construct the candidate evaluation function based on evaluation indexes such as integrity, noise level, clarity, stability, and task relevance.

[0097] For example, when the modality type of the candidate modal data is video, the first evaluation function corresponding to the video modality is determined as a candidate evaluation function. When the modality type of the candidate modal data is motion, the second evaluation function corresponding to the motion modality is determined as a candidate evaluation function. When the modality type of the candidate modal data is haptic, the third evaluation function corresponding to the haptic modality is determined as a candidate evaluation function. When the modality type of the candidate modal data is task text, the fourth evaluation function corresponding to the task text modality is determined as a candidate evaluation function. It should be understood that for evaluation functions corresponding to different modality types, the evaluation metrics may be the same or different, and the number of evaluation metrics may be the same or different.

[0098] For example, when the candidate modal data is video, the evaluation metrics in the first evaluation function include sharpness, stability, and content validity. When the candidate modal data is motion, the evaluation metrics in the second evaluation function include joint angle physical feasibility and motion smoothness. When the candidate modal data is haptic, the evaluation metrics in the third evaluation function include the variance reference value of the haptic data. When the candidate modal data is task text, the evaluation metrics in the fourth evaluation function include semantic integrity, instruction clarity, and task executability.

[0099] S3212: Calculate the value of each evaluation indicator.

[0100] For example, the index value of each evaluation index is calculated based on each evaluation index in the candidate evaluation function.

[0101] In some embodiments, for the first evaluation function (i.e., the evaluation function corresponding to the video modality data), the index values ​​for sharpness, stability, and content effectiveness are calculated respectively. For example, the following formula can be used: The index value for calculating sharpness , This indicates calculating the average value over a time series. Image entropy represents the image's value. This represents the theoretical maximum entropy value of an ideally sharp image. The calculation process is as follows: First, calculate the image entropy of each frame in the video frame sequence transmitted by the camera device. Then, the average time series entropy of all frames is calculated to obtain the entropy mean. Finally, the entropy mean is divided by the theoretical maximum entropy value. To obtain the resolution index value. It should be understood that for the evaluation of sharpness, other evaluation parameters can be added according to the actual application requirements, and sharpness can be comprehensively evaluated in combination with image entropy, such as Laplacian variance, to calculate the sharpness index value. For example, the optical flow method can be used to calculate the global motion vector based on the optical flow field between consecutive frames. Based on the global motion vector Determine the motion stability of video data. Specifically, for timestamps... and Two frames of images and First, put the two images into a single frame. and Convert each image to grayscale, and then use... Dense optical flow field is calculated using the dense optical flow algorithm. ,in, and These are the height and width of the image, respectively. Indicates pixel position The horizontal and vertical displacement components at a given point are used to calculate the global motion vector at that time step by taking the median values ​​of the optical flow vector along the horizontal and vertical directions, respectively. The index value for calculating stability is ,in, as well as These are the global motion vectors. The horizontal and vertical displacement components. Represents the global motion vector of Norm, Represents the global motion vector over all time steps. of norm (i.e.) The variance is then calculated. For example, a pre-trained object detection model (e.g., YOLO) is used to process each frame, identifying frames containing the target object as target frames. The target object refers to an object related to the robot's target task. The proportion of target frames in the total number of frames is calculated; that is, the number of target frames is divided by the number of frames in the video sequence to obtain the content effectiveness index. , The number of target frame images, This represents the number of image frames in the video frame sequence.

[0102] For the second evaluation function (i.e., the evaluation function corresponding to the motion modality data), the index values ​​for the physical feasibility of joint angles and the index values ​​for motion smoothness are calculated respectively. Joint angles refer to the joint angles of the robot's manipulator. For example, it is determined whether the joint angles of the manipulator in each frame of data are within the mechanical limit angle range. Within the dataset, the number of data frames in which the joint angle is outside the mechanical limit range was counted. The index value of the physical feasibility of the joint angle was calculated. ,in, This represents the total number of data frames in a dataset. For example, it calculates the joint angular acceleration in a dataset. The sum of absolute values ​​and the wrist pose translation of the robot's robotic hand. acceleration of norm The sum of the absolute values ​​yields the motion smoothness index value: ,in, Centralized joint angular acceleration in a set of data The sum of the absolute values, For a set of datasets, the wrist pose translation of the robot's manipulator. acceleration of norm The sum of the absolute values ​​of .

[0103] For the third evaluation function (i.e., the evaluation function corresponding to the tactile modality data), the variance reference value of the tactile data is calculated respectively. Here, tactile data refers to a portion of the tactile signal extracted from the multidimensional tactile time-series signal using a sliding window. For example, multiple sets of tactile data are extracted from the multidimensional tactile time-series signal, and the variance of each set of tactile data is calculated. Then, using the following formula: Calculate the reference value of variance for the tactile data. This indicates calculating the average value. This represents the reference value for the variance of the tactile data.

[0104] For the fourth evaluation function (i.e., the evaluation function corresponding to the task text modality data), the index values ​​for semantic integrity, instruction clarity, and task executability are calculated respectively. Among them, semantic integrity is used to evaluate the task text. Does it contain all the core semantic elements necessary for successful task execution? For example, a pre-trained semantic role labeling model can be used to label the task text. The process involves parsing and extracting semantic roles such as core predicate actions (e.g., grabbing, moving), subject, and object. Then, based on a pre-defined general task slot template (e.g., a text task template (<Action Hand>, <Action>, <Target Object>, <Source Location>, <Target Location>), the fill rate of the extracted semantic roles in key slots of the general task slot module is determined. The semantic integrity index value Instruction clarity is used to evaluate task text. To address issues such as ambiguity, unclear referencing, or structural confusion, and to ensure that control commands are clearly understood, this invention measures command clarity through keyword coverage. For example, the task text is calculated based on a predefined "clear command keyword lexicon" for robot operations. The ratio of the number of content words belonging to the clear instruction keyword lexicon to the total number of content words in the clear instruction keyword lexicon. Then the index value of instruction clarity is calculated. Here, the content words refer to the core predicate action. For task text The number of content words in the clear instruction keyword lexicon. This indicates the total number of content words in the clear instruction keyword lexicon. Task executability is used to evaluate the task text. Does it match a predefined executable skill or task template in the robot system? For example, firstly, a text encoder is used to convert the names and descriptions of all preset task templates in the robot system into feature vectors, forming an executable task vector library. Subsequently, the task text was processed using the same text encoder. Convert to feature vector Calculate and The cosine similarity of each vector in the dataset is used to determine the maximum cosine similarity among all vectors. The target cosine similarity is then determined, and the executability index is calculated as follows: ,in, Let cosine similarity be the target.

[0105] It is understood that this embodiment is only an illustrative description of the evaluation indicators in the evaluation function corresponding to different modal data and the method of calculating the indicator values. Engineers can select other suitable evaluation functions and evaluation indicators in other evaluation functions, as well as the calculation method of the indicator values ​​of each evaluation indicator, according to actual needs. This embodiment of the invention does not impose any limitations on this.

[0106] S3213: Calculate the first quality score based on at least one index value.

[0107] In this embodiment of the invention, for the first evaluation function, after calculating the index values ​​of sharpness, stability, and content effectiveness, the index values ​​of sharpness, stability, and content effectiveness are weighted and summed to obtain the first quality score corresponding to the video modality data. For example, this is achieved through the following formula: The first quality score corresponding to the video modality data is calculated. , , and These are the weighting coefficients for the first video modality, the second video modality, and the third video modality, respectively. The value is an indicator of sharpness. This is an indicator value for stability. This is an indicator value for the effectiveness of the content.

[0108] For the second evaluation function, after calculating the index values ​​for joint angle physical feasibility and motion smoothness, the index values ​​for joint angle physical feasibility and motion smoothness are weighted and summed to obtain the first quality score corresponding to the motion modality data. For example, this can be achieved using the following formula: Calculate the first mass fraction corresponding to the motion mode data. , and These are the weighting coefficients for the first motion mode and the weighting coefficients for the second motion mode, respectively. This is an index value representing the physical feasibility of the joint angle. This is an index value for motion smoothness.

[0109] For the third evaluation function, the variance reference value of the tactile data is calculated. Then, the variance reference value The first quality score corresponding to the data identified as the tactile modality .

[0110] For the fourth evaluation function, after calculating the index values ​​for semantic integrity, instruction clarity, and task executability, these values ​​are weighted and summed to obtain the first quality score corresponding to the task text modality data. For example, this can be achieved using the following formula: The first quality score corresponding to the task text modality is calculated. , , and These are the text modal weighting coefficients for the first task, the second task, and the third task, respectively. This is an indicator value for semantic integrity. This is an indicator of instruction clarity. This is a metric for task feasibility.

[0111] S322: Calculate the second quality score of the reference modal data set based on the target metric value of the reference modal data set.

[0112] In this embodiment, the reference modal dataset includes data from any two modalities across multiple modalities. The target metric characterizes the inconsistency between the two modalities in the reference modal dataset and can be calculated using methods such as statistical difference indices, joint distribution bias, synchronization error, and cross-modal fusion residuals. The second quality score characterizes the correlation between the two modalities in the reference modal dataset; the smaller the inconsistency between the two modalities, the higher the correlation.

[0113] For example, based on actual task requirements, data from any two modalities are selected from multiple modalities to form a reference modality dataset. The differences and consistency between the two modalities in the reference modality dataset are analyzed and calculated to obtain a second quality score for the reference modality dataset. After calculating the second quality scores for multiple different reference modality datasets, multiple second quality scores for reference modality datasets are obtained.

[0114] In some embodiments, a second quality score of the reference modal data set is calculated based on the target metric value of the reference modal data set, specifically including but not limited to the following steps S3221-S3223:

[0115] S3221: Calculate the candidate similarity between the first modality data and the second modality data.

[0116] In this embodiment, the data of the two modes in the reference modal data set are the first modal data and the second modal data, that is, the data of the two modes in the reference modal data set are respectively used as the first modal data and the second modal data.

[0117] Specifically, first-modal data and second-modal data are extracted from the reference modality dataset. Based on a preset similarity calculation method, candidate similarity is calculated between the two modalities (i.e., the first-modal data and the second-modal data). Candidate similarity reflects the degree of consistency between the two modalities in data performance or task characteristics. In this embodiment, candidate similarity is calculated using one or a combination of the following methods: statistical correlation metrics (such as Pearson correlation coefficient, covariance, mutual information, etc.), feature space distance metrics (such as Euclidean distance, cosine similarity, Mahalanobis distance, etc.), cross-modal inference consistency (e.g., the difference between the two modalities in target identification, location estimation, trajectory analysis, etc.), and fusion model output consistency indices (such as fusion residuals, model internal consistency scores, etc.). A higher candidate similarity indicates greater consistency and higher mutual support between the two modalities.

[0118] S3222: Subtract the candidate similarity from the second preset value to obtain the target metric value.

[0119] After calculating the candidate similarity between the first modality data and the second modality data, the candidate similarity is converted into an inconsistency metric (i.e., the target metric). The target metric reflects the degree of inconsistency or mismatch between the first modality data and the second modality data. It can be understood that the higher the candidate similarity, the smaller the target metric, and vice versa.

[0120] For example, engineers set a second preset value based on actual application requirements. The second preset value represents the theoretical maximum similarity or similarity benchmark value. In this embodiment, the second preset value is 1. Subtracting the candidate similarity from the second preset value yields the target metric value.

[0121] For example, when the first modal data and the second modal data are respectively the task text modal data and the video modal data, a visual language model (such as the CLIP model) is used to extract the first embedding feature vector of the video keyframes. And converting the task text into the corresponding second embedded feature vector through a text encoder. Calculate the first embedded feature vector With the second embedded feature vector The similarity between the two modal data is used as a candidate similarity between the first modality data and the second modality data. Subtracting the candidate similarity from the second preset value 1 yields the inconsistency measure (i.e., the target measure value) between the first modality data and the second modality data. ,in, For the target metric, This represents candidate similarity. It should be understood that when the task text modality data and the video modality data form a reference modality dataset, the target metric of this reference modality dataset is used to evaluate whether the task described in the text and the scene / activity observed in the video semantically match.

[0122] For another example, when the first modal data and the second modal data are respectively the data of the task text modality and the motion modality data (specifically, the joint angles of the robotic arm), a time series model (such as LSTM or Transformer model) is used to sequence the joint angles. Encoded as action feature vector And the task text is encoded using a text encoder. Convert to intent vector Calculate action feature vector With intent vector The similarity between the two modalities is used as a candidate similarity between the first modality data and the second modality data. Subtracting the candidate similarity from the second preset value 1 yields the inconsistency measure (i.e., the target measure) between the first modality data and the second modality data. ,in, For the target metric, This refers to candidate similarity. It should be understood that when the task text modality data and motion modality data form a reference modality data set, the target metric of this reference modality data set is used to determine whether the intent of the text instruction and the executed action are consistent at the behavioral level.

[0123] Understandably, for different reference modal datasets, the calculation methods for the candidate similarity of the first modal data and the second modal data may be the same or different. Engineers can choose any suitable calculation method according to actual needs, and the embodiments of the present invention do not impose any limitations on this.

[0124] S3223: Normalize the target metric to obtain the second quality score.

[0125] In this embodiment, a suitable normalization method (e.g., maximum-minimum interval normalization, mean-variance standardization, etc.) is adopted according to actual needs. function or The target metric is normalized using a function normalization mapping or scaling linear mapping, resulting in a second quality score, which is the normalized target metric. The value of the second quality score lies within a uniform dimension or range; a larger second quality score indicates a stronger correlation between the first and second modal data.

[0126] In this embodiment of the invention, the following methods are adopted: The function normalizes the target metric to obtain the second quality score. Specifically, it is based on the following formula: The second quality score of the reference modal data set is calculated, where, for Functions used to map target metrics to The interval is used to obtain the second mass fraction. Second mass fraction The closer it is to 1, the more modal it is. Data and modal The higher the degree of correlation between the data. This is a scaling factor used to control the steepness of the mapping. The default value is 1. For the target metric, Treat as independent variable ,but The function is represented as: , It is a natural constant.

[0127] S323: Calculate a reference quality score based on multiple second quality scores.

[0128] In this embodiment, the reference quality score represents the overall correlation of data from multiple modalities, that is, it represents the consistency and fusion reliability of data from multiple modalities in the overall dimension, as well as the collaborative effectiveness of different sensors and different data sources.

[0129] For example, multiple second quality scores from different reference modal datasets are calculated and fused to obtain a reference quality score. Engineers can choose any suitable calculation method (e.g., weighted summation, averaging, model prediction, nonlinear fusion model calculation, etc.) to calculate the reference quality score according to actual needs.

[0130] In some embodiments, a reference quality score is calculated based on multiple second quality scores, including but not limited to the following steps S3231-S3233:

[0131] S3231: Sum all the second quality scores to obtain the candidate quality scores.

[0132] For example, the candidate quality scores are obtained by summing the second quality scores of all reference modal datasets.

[0133] S3232: Divide the first preset value by the first numerical value to obtain the candidate value.

[0134] Wherein, the first value is the product of the second and third values, the second value is the number of modalities in the target dataset, and the third value is the difference between the second value and the natural number 1. The first preset value is 2.

[0135] For example, the number of modalities in the target dataset is counted. For instance, if the target dataset includes visual modal data, motion modal data, tactile modal data, and task text modal data, then the number of modalities is 4, i.e., the second value is 4. Subtracting the natural number 1 from the second value 4 yields the third value 3. Multiplying the second value by the third value gives the first value. Dividing the first preset value 2 by the first value yields the candidate value.

[0136] S3233: Multiply the candidate quality score by the candidate value to obtain the reference quality score.

[0137] For example, after calculating the candidate quality score and the candidate value, the candidate quality score is multiplied by the candidate value to obtain the reference quality score.

[0138] The global average quality score (i.e., the average of the second quality scores of all reference modal datasets) is calculated using the above method as a reference quality score. This can reflect the overall consistency level among the data of all modalities, and the calculation is simple and robust.

[0139] In some embodiments, a reference quality score is calculated based on multiple second quality scores, including but not limited to the following step S3234:

[0140] S3234: Determine the minimum value among multiple second quality scores as the reference quality score.

[0141] In some embodiments, the minimum value among the second quality scores that filter all reference modal data sets is found, and the minimum second quality score is used as the reference quality score.

[0142] By using the above method, the second smallest quality score in all reference modal datasets is taken as the reference quality score. This strategy emphasizes the "barrel effect," where a serious conflict between any two modal datasets will lower the overall correlation quality score. It is suitable for scenarios with extremely high security requirements and has extremely high security.

[0143] In some embodiments, a reference quality score is calculated based on multiple second quality scores, including but not limited to the following steps S3235-S3238:

[0144] S3235: Calculate the adjacency matrix and degree matrix of the quality score graph.

[0145] In this embodiment of the invention, the quality score graph is a graph obtained by treating the data of each modality as a node and the second quality score as the weight of the edge connecting the nodes. For example, please refer to [link to relevant documentation]. Figure 5 , Figure 5 The diagram schematically illustrates mass fraction graphs in some embodiments of the present invention. Figure 5In the diagram, node A1 represents video modality data, node A2 represents motion modality data, node A3 represents haptic modality data, and node A4 represents task text modality data. The weight of the edge connecting nodes A1 and A2 is 0.92 (meaning the second quality score corresponding to the video modality data and motion modality data is 0.92), the weight of the edge connecting nodes A1 and A3 is 0.91 (meaning the second quality score corresponding to the video modality data and haptic modality data is 0.91), and the weight of the edge connecting nodes A1 and A4 is 0.94 (meaning the second quality score corresponding to the video modality data and task text modality data is 0.94). For two unconnected nodes, the weight of their edges is 0, meaning the second quality score corresponding to the two modalities they represent is 0. For example, for nodes A2 and A3, the second quality score corresponding to the motion modality data and the haptic modality data is 0.

[0146] In this embodiment, the adjacency matrix of the quality score map is calculated using a first formula, which is:

[0147]

[0148] in, The first in the adjacency matrix line, number The values ​​of the elements in the column. For the first The data of the first modality and the first The second quality score corresponding to the data of each modality It refers to the first The data of the first modality and the first The second quality score corresponding to the data of each modality Not equal to 0, It refers to the first The data of the first modality and the first The second quality score corresponding to the data of each modality It equals 0. According to the first formula, when the... The data of the first modality and the first The second quality score corresponding to the data of each modality When not equal to 0, the th in the adjacency matrix line, number The value of an element in the column is 1, when the first element... The data of the first modality and the first The second quality score corresponding to the data of each modality When equal to 0, the th in the adjacency matrix line, number The value of each element in the column is 0.

[0149] In this embodiment, the degree matrix of the quality fraction map is calculated using the second formula, which is:

[0150]

[0151] in, The degree matrix represents the first degree. line, number The values ​​of the elements in the column. This indicates that any element in the degree matrix is ​​located in a row whose number of rows is not equal to the number of columns. This indicates that any element in the degree matrix has the same row and column number. For the mass fraction graph and the first The number of nodes connected to a given node, for example, see [link to relevant documentation]. Figure 5 As shown, for the first node A1, the second node A2, the third node A3, and the fourth node A4 are all connected to the first node A1. Therefore, the value of the element in the first row and first column of the degree matrix is ​​3, i.e. According to the second formula, when the row number and column number of any element in the degree matrix are not equal (i.e., ... When ), the degree matrix of the th line, number The value of an element in a column is 0 when the row number and column number of any element in the degree matrix are equal (i.e., ...). When ), the degree matrix of the th line, number The values ​​of the elements in the column are the same as those in the quality fraction graph. The number of nodes connected to a given node.

[0152] For example, according to Figure 5 The quality score graph shown is used to calculate its adjacency matrix and degree matrix. and , and As shown below:

[0153] , .

[0154] Understandably, the adjacency matrix has the same size as the degree matrix, and both are symmetric matrices. The number of rows and columns in both the adjacency and degree matrices are the same as the number of modalities in the target dataset. For example, if the target dataset includes visual modal data, motion modal data, tactile modal data, and task text modal data, then the number of modalities is 4, meaning the adjacency and degree matrices both have 4 rows and 4 columns.

[0155] S3236: Subtract the adjacency matrix from the degree matrix to obtain the reference matrix.

[0156] S3237: Calculate the algebraic connectivity of the reference matrix.

[0157] For example, subtracting the adjacency matrix from the degree matrix yields the reference matrix. This reference matrix can also be called the Laplace matrix. For example, when the adjacency matrix and degree matrix are respectively and At that time, reference matrix for:

[0158] .

[0159] In some embodiments, a computational tool is used to calculate the reference matrix. The algebraic connectivity of , where the algebraic connectivity is the reference matrix. The second smallest eigenvalue For example, using Python's... Library calculates reference matrix The algebraic connectivity of the reference matrix Import After the library, use Library functions Calculate and obtain the reference matrix. All eigenvalues, then using Library functions Sort all eigenvalues ​​in ascending order to obtain an eigenvalue sequence. Finally, select the second smallest eigenvalue from the eigenvalue sequence. The reference matrix is ​​obtained. The algebraic connectivity. It should be understood that algebraic connectivity... The larger the value, the tighter the overall connection of the quality score plot and the better the consistency of the data across different modalities.

[0160] It is understandable that the Laplace matrix It is a symmetric positive semi-definite matrix with all non-negative real eigenvalues, and the smallest eigenvalue is... It is always 0.

[0161] S3238: Normalize the algebraic connectivity to obtain the reference quality score.

[0162] Among them, the reference quality score and algebraic connectivity Positive correlation, algebraic connectivity After normalization, a reference quality score is obtained. In some embodiments, the following normalization function is used: algebraic connectivity Mapped to reference quality score Reference quality score exist Within the interval, For the quality fraction graph, The algebraic connectivity of the quality fraction graph. This is a normalization function used to normalize the algebraic connectivity. Mapped to Interval. In some embodiments, Treat as independent variable ,but The function is represented as: , It is a natural constant. This is a scaling factor used to control... The steepness of the function, The larger, The closer the function is to a step function, With the center point, when hour, It represents the expected algebraic connectivity of the quality score graph under a well-consistent state. ,in, and It needs to be set according to actual needs and data characteristics.

[0163] S324: Obtain a quality score vector based on multiple first quality scores and reference quality scores.

[0164] Specifically, the first quality score corresponding to each modality and the reference quality score representing the overall correlation of the multimodal data are combined to form a vector with a unified structure, thereby obtaining the quality score vector of the target data set. The quality score vector represents the quality status of the data in each modality and the overall degree of coordination among the data in multiple modalities.

[0165] In some embodiments, the quality grade vector corresponding to the quality score vector is determined using a quality grade labeling model, specifically including but not limited to the following step S331:

[0166] S331: Input the quality score vector into the quality grade labeling model so that the quality grade labeling model outputs the quality grade vector corresponding to the quality score vector.

[0167] In this embodiment, the quality score vector includes multiple first quality scores and reference quality scores. The quality level vector includes multiple first quality levels and reference quality levels, where each first quality level corresponds to a first quality score, and each reference quality level corresponds to a reference quality score.

[0168] For example, the quality score vector is input into the quality grade labeling model. Based on each first quality score in the quality score vector, the quality grade labeling model predicts and outputs the first quality grade corresponding to each first quality score, resulting in multiple first quality grades. Based on a reference quality score in the quality score vector, the quality grade labeling model predicts and outputs a reference quality grade corresponding to that reference quality score. These multiple first quality grades and reference quality grades are then combined sequentially to form a vector with a unified structure, resulting in a quality grade vector corresponding to the quality score vector.

[0169] In some embodiments, target processing operations corresponding to quality level vectors are performed on data from multiple modalities, including but not limited to the following steps S341-S343:

[0170] S341: If the response reference quality level is Level 1 or Level 2, determine that the re-acquisition of data is the target processing operation.

[0171] In some embodiments, when the reference quality level in the quality level vector is the first level or the second level, it indicates that the overall correlation between the data of each modality is poor, the data has serious quality problems, and the data of each modality currently collected cannot be used. In this case, it is determined that re-collecting data is the target processing operation. In this embodiment of the invention, a re-collection instruction is generated and sent to each type of sensor, so that each type of sensor re-collects the data of each modality.

[0172] S342: If the response reference quality level is level three or level four, determine the processing operation corresponding to the first quality level of the candidate modal data as the target processing operation.

[0173] In some embodiments, when the reference quality level in the quality level vector is the third or fourth level, it indicates that the overall correlation between the data of each modality is good and the data does not have serious or unusable quality problems. The corresponding target processing operation is determined according to the first quality level of the candidate modality data, that is, the processing operation corresponding to the first quality level of the candidate modality data is determined as the target processing operation.

[0174] In some embodiments, the processing operation corresponding to the first quality level of the candidate modal data is determined as the target processing operation, which includes, but is not limited to, the following steps S3421-S3424:

[0175] S3421: The first quality level of the response candidate modal data is the fourth level, and the first operation corresponding to the fourth level is determined as the target processing operation.

[0176] S3422: The first quality level of the response candidate modal data is the third level, and the second operation corresponding to the third level is determined as the target processing operation.

[0177] S3423: The first quality level of the response candidate modal data is the second level, and the third operation corresponding to the second level is determined as the target processing operation.

[0178] S3424: The first quality level of the response candidate modal data is designated as the first level, and the fourth operation corresponding to the first level is determined as the target processing operation.

[0179] In this process, when the first quality level of the candidate modal data is level four, it indicates that the candidate modal data is of high quality and can be used directly. The first operation is designated as the target processing operation, which includes storing the data or using it directly. When the first quality level of the candidate modal data is level three, it indicates that the candidate modal data is of good quality with only minor issues. The second operation is designated as the target processing operation, which includes denoising, padding, and alignment. When the first quality level of the candidate modal data is level two, it indicates that the candidate modal data is of average quality with some moderate issues. The third operation is designated as the target processing operation, which includes marking for manual review, thus handing the candidate modal data over to experts for processing. When the first quality level of the candidate modal data is level one, it indicates that the candidate modal data is of poor quality with severe issues. The fourth operation is designated as the target processing operation, which includes moving the data to an isolation area or discarding it, isolating or discarding the candidate modal data.

[0180] S343: Perform target processing operation.

[0181] After determining the target processing operation, the target processing operation is executed to process or label the data from multiple modalities for re-acquisition. For example, a first operation, a second operation, a third operation, or a fourth operation is performed, or the data from each type of sensor is re-acquired for each modality.

[0182] In summary, the robot multimodal data processing method provided in this embodiment of the invention not only evaluates the quality of each single modal data, but also evaluates the overall correlation between multiple modal data. In this way, it achieves a comprehensive evaluation of the quality of multimodal data, identifies low-quality or conflicting modal data before data fusion or use, and enables corresponding processing operations on modal data of different quality to obtain high-quality modal datasets, thereby improving the robustness of robot environmental perception and the success rate of task execution.

[0183] This invention provides a computer-readable storage medium storing processor-executable computer program instructions. When executed by a processor, the computer program instructions cause the computer to perform any of the robot multimodal data processing methods provided in this invention, or to perform steps in any possible implementation of any of the robot multimodal data processing methods provided in this invention.

[0184] In some embodiments, the storage medium may be a flash memory, a hard disk, an optical disk, a register, a magnetic surface memory, a removable disk, a CD-ROM, a random access memory (RAM), a read-only memory (ROM), an electrically programmable ROM, and an electrically erasable programmable ROM, or any other form of storage medium known in the art, or various devices including one or any combination of the above storage media.

[0185] In some embodiments, computer program instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0186] As an example, computer program instructions may, but do not necessarily, correspond to files in a file system, and may be stored as part of a file that holds other programs or data, for example, in one or more scripts in an HTML (Hypertext Markup Language) document, or in a single file dedicated to the program in question, or in multiple collaborative files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0187] As an example, computer program instructions can be deployed to execute on a single computing device (including devices such as smart terminals and servers), or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network. It is readily understood that all or part of the steps of the methods described in the embodiments of the present invention above can be implemented directly using electronic hardware or processor-executable computer program instructions, or a combination of both.

[0188] Those skilled in the art will understand that the embodiments provided by this invention are merely illustrative. The order in which the steps in the methods of the embodiments are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The order can be adjusted, merged, and deleted according to actual needs. Modules or sub-modules, units or sub-units in the apparatus or system of the embodiments can be merged, divided, and deleted according to actual needs. For example, the division of units is only a logical functional division, and there may be other division methods in actual implementation. For another example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed.

[0189] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, it can be implemented using hardware. Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods.

[0190] It should be noted that the above embodiments are for illustrating the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be construed as limiting the scope of protection of the present invention. Those skilled in the art can understand that all or part of the processes of the above embodiments can be implemented by modifying the technical solutions described in the embodiments of the present invention, or by making equivalent substitutions for some of the technical features. It is understood that these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and should be considered as equivalent changes and modifications made based on the embodiments of the present invention, all of which should fall within the scope of the claims of the present invention.

Claims

1. A method of processing robotic multimodal data, the method comprising: The method comprises: obtaining a target data set of the robot, wherein the target data set comprises data of multiple modalities; calculating a quality score vector of the target data set, comprising: calculating a first quality score of candidate modality data based on a candidate evaluation function corresponding to the candidate modality data, wherein the first quality score represents the quality of the candidate modality data, and the candidate modality data is data of any one of the multiple modalities; calculating a second quality score of a reference modality data set based on a target metric value of the reference modality data set, wherein the reference modality data set comprises data of any two of the multiple modalities, the target metric value represents inconsistency of the two modalities in the reference modality data set, and the second quality score represents the correlation degree of the two modalities in the reference modality data set; calculating a reference quality score based on multiple second quality scores, wherein the reference quality score represents the overall correlation degree of the multiple modalities; and obtaining the quality score vector based on multiple first quality scores and the reference quality score, wherein the quality score vector comprises multiple first quality scores and one reference quality score, and each first quality score represents the quality of data of one modality; training a pre-constructed neural network model based on sample data to obtain a quality level labeling model, and determining a quality level vector corresponding to the quality score vector by using the quality level labeling model; performing a target processing operation corresponding to the quality level vector on the data of the multiple modalities.

2. The treatment method according to claim 1, characterized in that, The calculating of the reference quality score based on multiple second quality scores comprises: adding all the second quality scores to obtain a candidate quality score; dividing a first preset value by a first value to obtain a candidate value, wherein the first value is a product of a second value and a third value, the second value is the number of modalities in the target data set, and the third value is the difference between the second value and a natural number 1; multiplying the candidate quality score by the candidate value to obtain the reference quality score.

3. The treatment method of claim 1, wherein The calculating of the reference quality score based on multiple second quality scores comprises: determining the minimum value of multiple second quality scores as the reference quality score.

4. The treatment method of claim 1, wherein The calculating of the reference quality score based on multiple second quality scores comprises: calculating an adjacency matrix and a degree matrix of a quality score graph, wherein the quality score graph is a graph obtained by taking data of each modality as a node and taking the second quality score as the weight of an edge connecting the nodes, the size of the adjacency matrix is the same as that of the degree matrix; subtracting the adjacency matrix from the degree matrix to obtain a reference matrix; calculating the algebraic connectivity of the reference matrix; normalizing the algebraic connectivity to obtain the reference quality score.

5. The treatment method of claim 1, wherein The two modalities in the reference modality data set are first modality data and second modality data, and the calculating of the second quality score of the reference modality data set based on the target metric value of the reference modality data set comprises: calculating a candidate similarity between the first modality data and the second modality data; subtracting the candidate similarity from a second preset value to obtain the target metric value; normalizing the target metric value to obtain the second quality score.

6. The treatment method according to any one of claims 1 to 5, characterized in that, The calculating the first quality score of the candidate modality data based on the candidate evaluation function corresponding to the candidate modality data comprises: determining, according to a modality type of the candidate modality data, that an evaluation function corresponding to the modality type is a candidate evaluation function, the candidate evaluation function comprising at least one evaluation index; calculating an index value of each evaluation index; calculating the first quality score based on at least one index value.

7. The treatment method according to claim 6, characterized in that, The determining the quality level vector corresponding to the quality score vector by using the quality level labeling model comprises: inputting the quality score vector into the quality level labeling model, so that the quality level labeling model outputs a quality level vector corresponding to the quality score vector, wherein the quality level vector comprises a plurality of first quality levels and a reference quality level, each first quality level corresponding to a first quality score, and the reference quality level corresponding to the reference quality score.

8. An electronic device, comprising: comprise: a processor and a memory connected to the processor in communication; the memory stores computer program instructions executable by the processor, and the computer program instructions, when executed by the processor, cause the electronic device to perform the robot multi-modality data processing method according to any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer program instructions executable by the processor, and the computer program instructions, when executed by the processor, cause the computer to perform the robot multi-modality data processing method according to any one of claims 1-7.

Images