A humanoid robot multi-modal data processing method, system, device and medium

Abstract

The application discloses a humanoid robot multi-modal data processing method, system, device and medium. The method comprises the following steps: collecting a multi-modal data set by a humanoid robot, preprocessing the multi-modal data set to obtain a preprocessed data set, generating multi-layer prompt information according to an industrial scene of the humanoid robot, inputting the preprocessed data set and the multi-layer prompt information into a pre-trained multi-modal model for collaborative reasoning processing to output an industrial task processing result. The embodiment of the application can improve the data processing efficiency of the humanoid robot and can be widely applied to the technical field of intelligent robots.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent robots, and particularly relates to a humanoid robot multi-modal data processing method, system, device and medium. BACKGROUND

[0002] In the field of industrial manufacturing, a humanoid robot needs to process multi-modal information such as vision, touch, and voice to complete complex tasks such as assembly and quality inspection. In related technologies, there are methods for processing general data, but these methods are not optimized for the particularity of the industrial scene, resulting in insufficient generalization ability and execution accuracy of the humanoid robot in industrial tasks.

[0003] In summary, the technical problems existing in related technologies need to be improved. SUMMARY

[0004] The main purpose of the embodiments of the present application is to provide a humanoid robot multi-modal data processing method, system, device and medium, which can improve the data processing efficiency of the humanoid robot.

[0005] To achieve the above-mentioned purpose, one aspect of the embodiments of the present application provides a humanoid robot multi-modal data processing method, which comprises:

[0006] A multi-modal data set is collected by a humanoid robot;

[0007] The multi-modal data set is preprocessed to obtain a preprocessed data set;

[0008] Multi-layer prompt information is generated according to the industrial scene of the humanoid robot;

[0009] The preprocessed data set and the multi-layer prompt information are input into a pre-trained multi-modal model for collaborative inference processing, and an industrial task processing result is output.

[0010] In some embodiments, the multi-modal data set includes visual images, touch signals, and operation instruction texts, and the preprocessing of the multi-modal data set to obtain a preprocessed data set comprises:

[0011] The visual images are subjected to contour feature extraction processing to obtain contour features;

[0012] The visual images are subjected to defect detection and vector generation processing according to the contour features to obtain visual features;

[0013] The touch signals are subjected to time-frequency domain transformation and frequency band feature extraction processing to obtain touch features;

[0014] The operation instruction texts are subjected to semantic enhancement processing according to a pre-set industrial field vocabulary to obtain semantic enhancement texts;

[0015] performing semantic vector conversion and constraint feature addition processing on the semantic enhanced text to obtain text features;

[0016] obtaining the pre-processing data set according to the visual features, the tactile features and the text features.

[0017] In some embodiments, the generating the multi-layer prompt information according to the industrial scene of the humanoid robot comprises:

[0018] performing prompt generation processing on the role attribute of the humanoid robot according to the industrial scene to obtain base layer prompt information;

[0019] performing prompt generation processing on the industrial task of the humanoid robot according to the industrial scene to obtain task layer prompt information;

[0020] performing prompt generation processing on the task constraint of the humanoid robot according to the industrial scene to obtain constraint layer prompt information;

[0021] obtaining the multi-layer prompt information according to the base layer prompt information, the task layer prompt information and the constraint layer prompt information.

[0022] In some embodiments, the inputting the pre-processing data set and the multi-layer prompt information into a pre-trained multi-modal model for collaborative inference processing to output an industrial task processing result comprises:

[0023] performing feature extraction and knowledge graph embedding processing on the pre-processing data set to obtain visual recognition features;

[0024] performing time sequence dependency relationship capturing processing on the pre-processing data set to obtain tactile recognition features;

[0025] performing encoding processing on the multi-layer prompt information to obtain prompt feature vectors;

[0026] performing cross-modal fusion processing on the visual recognition features and the tactile recognition features according to the prompt feature vectors to obtain the industrial task processing result.

[0027] In some embodiments, the performing cross-modal fusion processing on the visual recognition features and the tactile recognition features according to the prompt feature vectors to obtain the industrial task processing result comprises:

[0028] performing attention weight calculation processing on the visual recognition features according to the prompt feature vectors to obtain visual task attention;

[0029] According to the prompt feature vector, attention weight calculation processing is performed on the haptic recognition feature to obtain haptic task attention;

[0030] According to the visual task attention and the haptic task attention, feature fusion processing is performed on the visual recognition feature and the haptic recognition feature to obtain a fusion feature;

[0031] According to the fusion feature, task output prediction processing is performed to obtain the industrial task processing result.

[0032] In some embodiments, before the pre-trained multi-modal model is input with the preprocessed data set and the multi-layer prompt information for collaborative inference processing, the method further includes training the multi-modal model, including:

[0033] obtaining a training data set and a training prompt set;

[0034] According to the training prompt set, sample construction processing is performed on the training data set to obtain a sample data set; the sample data set includes positive sample pairs and negative sample pairs;

[0035] According to a progressive training strategy, the sample data set is input into the multi-modal model to obtain a prediction result;

[0036] According to the prediction result, a model loss is calculated, and parameters of the multi-modal model are adjusted according to the model loss.

[0037] In some embodiments, according to the progressive training strategy, the sample data set is input into the multi-modal model to obtain a prediction result, including:

[0038] According to the training prompt set, single-modal prompt processing is performed on the sample data set to obtain a first training set;

[0039] According to the training prompt set, multi-modal joint prompt processing is performed on the sample data set to obtain a second training set;

[0040] According to the training prompt set, industrial scene prompt processing is performed on the sample data set to obtain a third training set;

[0041] According to different training stages, the first training set, the second training set and the third training set are respectively input into the multi-modal model for task prediction processing to obtain the prediction result.

[0042] To achieve the above-mentioned purpose, another aspect of the embodiment of the present application proposes a multi-modal data processing system of a humanoid robot, the system comprising:

[0043] A data acquisition module is configured to acquire a multi-modal data set through a humanoid robot.

[0044] The preprocessing module is used to preprocess the multimodal dataset to obtain a preprocessed dataset;

[0045] The prompt generation module is used to generate multi-layered prompt information based on the industrial scenario of the humanoid robot;

[0046] The collaborative reasoning module is used to input the preprocessed dataset and the multi-layer prompt information into a pre-trained multimodal model for collaborative reasoning processing, and output the industrial task processing result.

[0047] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0048] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described above.

[0049] To achieve the above objectives, another aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the methods described above.

[0050] The embodiments of this application include at least the following beneficial effects: This application provides a method, system, device, and medium for processing multimodal data of a humanoid robot. This solution obtains a multimodal dataset through a humanoid robot, enabling the processing of multimodal data and improving the humanoid robot's data processing capabilities. Furthermore, this solution generates multi-layered prompting information based on the industrial scenario of the humanoid robot. It can generate different prompting information according to specific industrial scenarios to guide the model's processing, and can flexibly adjust processing strategies according to the characteristics of different industrial tasks, improving task processing capabilities in complex industrial scenarios. In addition, this solution inputs the pre-processed dataset and multi-layered prompting information into a pre-trained multimodal model for collaborative reasoning processing, outputting industrial task processing results. This enables the humanoid robot to effectively integrate multimodal information such as vision and touch, adapting to the precision operation requirements in industrial scenarios and improving the accuracy of the robot in performing industrial tasks. Attached Figure Description

[0051] Figure 1 This is a flowchart of a multimodal data processing method for a humanoid robot provided in an embodiment of this application;

[0052] Figure 2 This is a schematic diagram of the structure of a multimodal model provided in an embodiment of this application;

[0053] Figure 3 FIG. 1 is a structural schematic diagram of a multi-modal data processing system of a humanoid robot according to an embodiment of the present application;

[0054] Figure 4 FIG. 4 is a hardware structural schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0055] For the purpose of making the objects, technical solutions and advantages of the present application more clear, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not intended to limit the present application. When the following description refers to the accompanying drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementation manners described in the following exemplary embodiments do not represent all the implementation manners consistent with the embodiments of the present application, but are only examples of systems and methods consistent with some aspects of the embodiments of the present application as detailed in the appended claims.

[0056] It can be understood that the terms "first", "second", and the like used in the present application can be used herein to describe various concepts, but unless specifically stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiments of the present application, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "when" or "upon" or "in response to determining".

[0057] The terms "at least one", "multiple", "each", "any", and the like used in the present application include one, two or more than two, multiple includes two or more than two, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0059] In the field of industrial manufacturing, humanoid robots need to process multi-modal information such as vision, touch, voice, etc. to complete complex tasks such as assembly, quality inspection, etc. In the related technology, general humanoid robots lack the injection of industrial field knowledge, resulting in insufficient visual recognition accuracy of industrial parts, weak cross-modal fusion ability of touch-vision, and difficulty in adapting to precision operations, as well as large semantic understanding deviation of industrial process instructions. Related humanoid robots do not optimize data processing for the special nature of industrial scenes, such as high precision requirements and strong process constraints, resulting in insufficient generalization ability and execution precision of humanoid robots in industrial tasks.

[0060] Therefore, in the embodiments of the present application, a humanoid robot multi-modal data processing method, system, device and medium are provided. The scheme can process multi-modal data by collecting multi-modal data sets by the humanoid robot, and improve the data processing capability of the humanoid robot. Moreover, the scheme generates multi-layer prompt information according to the industrial scene of the humanoid robot, can generate different prompt information to guide the processing of the model according to the characteristics of different industrial tasks, and can flexibly adjust the processing strategy according to the characteristics of different industrial tasks, thereby improving the task processing capability in complex industrial scenes. In addition, the scheme inputs the preprocessed data set and the multi-layer prompt information into the pre-trained multi-modal model for collaborative reasoning processing, and outputs the industrial task processing result, which can enable the humanoid robot to effectively fuse multi-modal information such as vision and touch, adapt to the precision operation requirements in the industrial scene, and improve the accuracy of the robot in executing industrial tasks.

[0061] The humanoid robot multi-modal data processing method provided in the embodiments of the present application relates to the technical field of intelligent robots. The humanoid robot multi-modal data processing method provided in the embodiments of the present application can be applied in a humanoid robot, can be applied in a server, and can also be software running in a humanoid robot or a server. In some embodiments, the server end can be configured as an independent physical server, can be configured as a server cluster or a distributed system composed of multiple physical servers, can be configured as a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN, and big data and artificial intelligence platform, and the server can also be a node server in a blockchain network; the software can be an application that implements the humanoid robot multi-modal data processing method, but is not limited to the above forms.

[0062] The application is operable in numerous general or specialized computer system environments or configurations. Examples of well-known computing systems, environments, and / or configurations that can be suitable for use with the application include personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like. The application can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like, that perform particular tasks or implement particular abstract data types. The application can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in local and remote computer storage media including memory storage devices.

[0063] Figure 1 is an optional flowchart of a humanoid robot multi-modal data processing method provided by an embodiment of the application, Figure 1 The method in the above embodiment can include, but is not limited to, steps S101 to S104.

[0064] Step S101, acquiring a multi-modal data set by a humanoid robot;

[0065] Step S102, preprocessing the multi-modal data set to obtain a preprocessed data set;

[0066] Step S103, generating multi-layer prompt information according to an industrial scene of the humanoid robot;

[0067] Step S104, inputting the preprocessed data set and the multi-layer prompt information into a pre-trained multi-modal model for collaborative inference processing, and outputting an industrial task processing result.

[0068] The steps S101 to S104 shown in the above embodiment are to acquire a multi-modal data set by a humanoid robot. The multi-modal data set can include a visual image of a part acquired by a camera, a pressure signal acquired by a tactile sensor, and an operation instruction text in a process document. Then, the multi-modal data set is preprocessed. Specifically, the visual image is subjected to convolution feature extraction and preprocessed in combination with an industrial defect detection algorithm, the tactile signal is subjected to time-frequency domain transformation to extract features, and the operation instruction text is subjected to semantic enhancement by an industrial domain vocabulary to obtain a preprocessed data set. The above embodiment further generates multi-layer prompt information according to an industrial scene of the humanoid robot. The prompt information can include a priority weight based on an industrial task characteristic. The preprocessed multi-modal data and the layered prompt information are input into a multi-modal model including a cross-modal fusion layer for collaborative inference processing, and an industrial task processing result is output.

[0069] The embodiment of the application preprocesses a multi-modal training set in the field of industrial manufacturing, generates multi-layer target prompt information including a basic layer, a task layer and a constraint layer according to an industrial scene, inputs the preprocessed data and the prompt information into a multi-modal encoder, and trains a target multi-modal model through contrastive learning and a progressive strategy.

[0070] In some embodiments, the multi-modal data set includes visual images, tactile signals and operation instruction texts, and the preprocessing of the multi-modal data set to obtain a preprocessed data set includes:

[0071] contour feature extraction processing is performed on the visual images to obtain contour features;

[0072] defect detection and vector generation processing are performed on the visual images according to the contour features to obtain visual features;

[0073] time-frequency domain transformation and frequency band feature extraction processing are performed on the tactile signals to obtain tactile features;

[0074] semantic enhancement processing is performed on the operation instruction texts according to a preset industrial field vocabulary to obtain semantic enhancement texts;

[0075] semantic vector conversion and constraint feature addition processing are performed on the semantic enhancement texts to obtain text features;

[0076] The preprocessed data set is obtained according to the visual features, the tactile features and the text features.

[0077] In the embodiment of the application, a part visual image is acquired by a camera, and convolution feature extraction is performed on the visual image and combined with industrial defect detection algorithm preprocessing to extract key features in the image and identify potential defects. The visual features are obtained by vector generation and contour feature fusion of the defect features. In the embodiment of the application, a pressure signal is acquired by a tactile sensor, and the pressure signal is taken as a tactile signal to extract impact features by time-frequency domain transformation of the tactile signal, so as to convert the time domain signal into time-frequency features for analysis and obtain tactile features. The operation instruction texts in the process documents are acquired, the operation instruction texts are enhanced in semantics by an industrial field vocabulary, and constraint features are added to the enhanced features to obtain text features, which can improve the understanding ability of the model for industrial professional terms and instructions.

[0078] In one possible embodiment, multi-angle image acquisition of the parts is performed by a camera, with a resolution of no less than 1280x720 pixels, ensuring that the details of the parts can be distinguished. In the preprocessing stage, contour features are first extracted by a convolutional network, which includes at least 10 convolutional layers, and batch normalization and ReLU activation functions are used to enhance the feature extraction capability. In combination with the industrial defect detection algorithm, the image is preprocessed: the part contour is identified by an edge detection operator, and morphological operations are used to optimize the contour continuity; for potential defect areas, a multi-dimensional feature vector is generated by a feature extraction network for subsequent defect classification. For example, in the motor shell assembly scene, the visual features after preprocessing can accurately distinguish between bolt hole positions and surface scratches. The pressure signals during the grabbing of the parts are collected by the multi-dimensional tactile sensor on the dexterous hand, and the sampling frequency of one embodiment is 1 kHz, ensuring that the instantaneous contact force changes are captured. The time-frequency domain transformation is performed on the tactile signals, and the time domain signals are decomposed into feature components of different frequency bands by wavelet transform, focusing on extracting impact features (such as force mutations at the moment of grabbing) and trend features (such as continuous grabbing force changes) in the high-frequency band. Taking gear grabbing as an example, the features after time-frequency domain transformation can reflect the force distribution law when the gear is engaged, providing data support for subsequent action control. The operation instruction text is extracted from the process document, such as "tighten the bolt to the specified torque" and "position the bearing mounting hole". Semantic enhancement is performed by an industrial domain vocabulary, which includes no less than 1000 industrial professional terms in one embodiment, covering part names, process parameters, operation specifications, etc. Word embedding technology is used to convert the text into a semantic vector, and process constraints (such as torque range, precision requirements, etc.) are introduced as additional feature dimensions. For example, the "tighten the cylinder head bolt" instruction is converted into a semantic vector containing keywords such as "cylinder head", "bolt", and "tighten", with the constraint feature of "torque 20-25 N·m".

[0079] In some embodiments, the generating the multi-layer prompt information according to the industrial scene of the humanoid robot comprises:

[0080] generating the base layer prompt information according to the role attribute of the humanoid robot in the industrial scene;

[0081] generating the task layer prompt information according to the industrial task of the humanoid robot in the industrial scene;

[0082] generating the constraint layer prompt information according to the task constraint of the humanoid robot in the industrial scene;

[0083] obtaining the multi-layer prompt information according to the base layer prompt information, the task layer prompt information, and the constraint layer prompt information.

[0084] In the embodiment of the present application, the role attribute of the humanoid robot is prompted according to the industrial scene, wherein the industrial scene can be obtained by scene classification judgment on a visual image, or can be determined by inputting specific scene data by a user. The embodiment of the present application defines the role attribute of the humanoid robot, generates a basic layer prompt by defining the role of the robot and the task category. For example, the role of the robot is positioned as "automobile engine assembly robot", and the task category is determined as "identifying cylinder parts". The basic layer prompt is used to clearly define the role attribute of the robot in the industrial scene, such as "as an automobile engine assembly robot" and "as an electronic component quality inspection robot". At the same time, the task category is defined, such as "identifying engine cylinder parts" and "detecting circuit board welding defects". The layer prompt adopts a template design, including scene keywords (such as "engine cylinder" and "circuit board") and role keywords (such as "assembly" and "quality inspection"), to ensure the basic domain cognition of the model. The embodiment of the present application also disassembles the specific execution steps of the industrial task to be performed by the humanoid robot, and generates a task layer prompt. For example, the task logic sequence of the industrial task is prompted, such as "positioning bolt hole → performing tightening". The embodiment of the present application also generates a constraint layer prompt according to the process standard and precision requirement, such as taking the mandatory technical parameter "torque error ≤ 1%" as the constraint layer prompt. In the part assembly scene, the basic layer prompt can be "as an automobile assembly robot", the task layer prompt can be "positioning bolt hole and performing tightening operation", and the constraint layer prompt can be "torque error ≤ 1%, tightening sequence: left up → right down → right up → left down".

[0085] In some embodiments, the pre-processed data set and the multi-layer prompt information are input into a pre-trained multi-modal model for collaborative inference processing, and an industrial task processing result is output.

[0086] The pre-processed data set is subjected to feature extraction and knowledge graph embedding processing to obtain visual recognition features;

[0087] The pre-processed data set is subjected to time sequence dependency relationship capturing processing to obtain haptic recognition features;

[0088] The multi-layer prompt information is subjected to encoding processing to obtain a prompt feature vector;

[0089] The visual recognition features and the haptic recognition features are subjected to cross-modal fusion processing according to the prompt feature vector to obtain the industrial task processing result.

[0090] In the embodiment of the present application, the pre-processed data set and the multi-layer prompt information are input into a pre-trained multi-modal model, such as Figure 2As shown, the multi-modal model includes a visual branch, a tactile branch, and a cross-modal fusion layer. Among them, the visual branch performs feature extraction and knowledge graph embedding processing on the preprocessed dataset to obtain visual recognition features. Specifically, the visual branch processes visual data in the preprocessed dataset by adopting a 3D convolutional network. In a feasible embodiment, the network depth of the visual branch is not less than 15 layers, and includes a feature extraction module and a knowledge graph embedding module. The part knowledge graph stores not less than 100,000 relationship triples, such as "bolt-nut-connection" and "bearing-shaft-support". After being encoded by a graph neural network, the visual features are injected into the visual feature space to enhance the understanding of the model on the assembly relationship of parts. The tactile branch of the embodiment captures and processes the time-dependent relationship of the preprocessed dataset to obtain tactile recognition features. The tactile branch processes the pressure signal sequence by a time sequence neural network. The network of the tactile branch includes a bidirectional LSTM layer and an attention mechanism, which can capture the time-dependent relationship of the force signal. The embodiment can also establish a tactile feature library combined with the force-displacement curve. The feature library stores more than 500 force feedback modes of industrial operations, such as "smooth grabbing", "thread engagement", "rigid collision", etc.

[0091] The embodiment encodes and processes the multi-layer prompt information to obtain a prompt feature vector. Specifically, the language branch generates an operation instruction feature through semantic enhancement, including a basic layer prompt (role), a task layer prompt (target), and a constraint layer prompt (process parameter). Finally, the dynamic weighted fusion of visual-tactile features is realized through a gating unit. The fusion weight is automatically adjusted by the type of industrial task. For example, in the assembly task, the weight proportion of visual features is 60%, and the weight proportion of tactile features is 40%. In the quality inspection task, the weight of visual features is increased to 80%. The gating unit calculates the weight coefficient through nonlinear transformation to ensure that the key modal feature dominates the fusion result.

[0092] In a feasible embodiment, the visual branch preferentially identifies the cylinder bolt hole under the basic layer prompt, the task layer prompt strengthens the hole edge feature extraction, and the constraint layer prompt increases the loss weight of the hole coordinate error by 50%, so that the final positioning error is controlled within 0.05 mm. The tactile branch matches the "tightening" force signal template according to the task layer prompt, and the loss weight of torque prediction is set to twice that of the ordinary feature according to the constraint layer prompt, so that the torque deviation is controlled within 0.7%. During cross-modal fusion, the tactile weight is increased from 0.4 to 0.6 under the constraint layer prompt, so that the tightening process is dominated by tactile feedback, and the final assembly precision meets the ISO 9283 industrial standard. The embodiment shows the guiding mechanism of multi-layer prompt information in the multi-modal model through the logic chain of "fusion input → branch interaction → cross-modal modulation → gradual optimization".

[0093] In some embodiments, the cross-modal fusion processing of the visual recognition feature and the haptic recognition feature according to the prompt feature vector obtains the industrial task processing result, including:

[0094] The attention weight calculation processing of the visual recognition feature according to the prompt feature vector obtains visual task attention;

[0095] The attention weight calculation processing of the haptic recognition feature according to the prompt feature vector obtains haptic task attention;

[0096] The feature fusion processing of the visual recognition feature and the haptic recognition feature according to the visual task attention and the haptic task attention obtains a fusion feature;

[0097] The task output prediction processing according to the fusion feature obtains the industrial task processing result.

[0098] In the embodiments of the present application, the cross-modal fusion layer calculates the fusion weight of the visual-haptic feature according to the prompt information, wherein the visual task attention is obtained by the interaction of three layers of prompt information and visual features, for example, the fusion of the basic layer prompt (such as “automobile engine assembly robot”) and the part relationship in the knowledge graph (such as the connection relationship of “bolt-cylinder”) generates the domain prior feature and injects the full connection layer of the 3D convolution network, which improves the part recognition accuracy. The task layer prompt (such as “positioning bolt hole”) is used as the visual attention mask to enhance the feature activation value of the bolt hole area in the image and suppress the background noise. The constraint layer prompt (such as “installation error ≤0.1mm”) is converted into the weight coefficient of the spatial coordinate loss function, so that the hole position coordinates output by the model are closer to the industrial standard requirements. Among them, the basic layer prompt vector is multiplied by the part knowledge graph vector of the visual branch to strengthen the domain prior knowledge; the task layer prompt vector is used as the query vector of the attention mechanism to guide the model to focus on the task-related modal features; the constraint layer prompt vector is converted into the regularization term of the loss function to increase the penalty weight for the output that does not meet the process requirements. In the embodiments of the present application, the linear transformation of the visual recognition feature and the prompt feature vector is calculated through the activation function to obtain the visual task attention. Similarly, the haptic task attention can be obtained by the attention weight calculation processing of the haptic recognition feature according to the prompt feature vector. The haptic task attention can also be directly calculated by 1-visual task attention. Finally, the visual recognition feature and the haptic recognition feature are processed by feature fusion according to the visual task attention and the haptic task attention, that is, the fusion feature is calculated by the formula: fusion feature = visual weight x visual feature + haptic weight x haptic feature. The fused feature is used to predict the task output, for example, the control instruction of the humanoid robot arm is predicted, which is used as the industrial task processing result.

[0099] In a feasible embodiment, different modal features are mapped to a unified feature space through linear transformation, taking a task-level prompt (such as "locate bolt hole and tighten") as a query vector Q, visual features V' and tactile features T' as keys K v ,K t and values V v ,V t , taking a constraint-level prompt (such as "torque error ≤ 1%") as a weight bias term b constraint , to adjust the attention weight. The relevance of visual features to the task prompt is calculated according to the visual task attention calculation formula, which is as follows:

[0100]

[0101] In the formula, Attn v represents visual task attention, and D k is the dimension of the key vector. This visual task attention reflects "the degree of attention of the task prompt to the visual features", such as focusing on the visual features of the bolt hole position in bolt assembly. Similarly, the relevance of tactile features to the task prompt is calculated to obtain tactile task attention, and the expression of the tactile task attention calculation formula is as follows:

[0102]

[0103] In the formula, Attn t represents tactile task attention. The embodiments of the present application dynamically fuse visual-tactile-linguistic features through a gating unit, where the gating signal is automatically adjusted by the type of industrial task, for example, in assembly tasks, the visual features are biased, and in quality inspection tasks, the tactile features are biased.

[0104] In some embodiments, before the pre-trained multi-modal model is input with the pre-processed data set and the multi-layer prompt information for collaborative inference processing, the method further comprises training the multi-modal model, comprising:

[0105] obtaining a training data set and a training prompt set;

[0106] performing sample construction processing on the training data set according to the training prompt set to obtain a sample data set; the sample data set includes positive sample pairs and negative sample pairs;

[0107] inputting the sample data set into the multi-modal model according to a progressive training strategy to obtain a prediction result;

[0108] calculating a model loss according to the prediction result, and adjusting parameters of the multi-modal model according to the model loss.

[0109] In the embodiments of the present application, a training data set and a training prompt set are obtained, wherein the training data set can include an action control training set, which can contain control instructions (such as "lift the robot arm to a specified position") and action execution information (such as "joint angle change amount" and "end pose coordinates"). During training, the mapping relationship between the control instructions and the action execution is learned through a time series neural network, combined with kinematic constraints (such as joint angle limits and speed upper limits). For example, in a robot arm painting task, the model learns the corresponding relationship between the "swing amplitude" instruction and the painting trajectory through training, while satisfying the process constraint of "painting speed 10-15 mm / s". The training data set in the embodiments of the present application can also include an object operation training set and a voice interaction training set, wherein the object operation training set contains operation requests (such as "grab the gear and install it to the shaft end") and operation knowledge information (such as "gear weight 5 kg" and "friction coefficient 0.3"). During training, the operation request is converted into a semantic vector, combined with the object physical property constraint, and an operation strategy is generated through a cross-modal fusion layer. For example, when grabbing a fragile component, the model will automatically reduce the grabbing force threshold and adjust the grabbing posture to reduce the impact. The voice interaction training set contains industrial scene voice requests (such as "query current station task" and "report equipment failure") and interaction response information (such as "current task: assemble motor housing" and "fault code: E001, conveyor belt jam"). During training, combined with industrial scene dialogue templates (such as fault reporting process and task query format), the model's understanding of professional terms is enhanced through an attention mechanism. For example, when receiving the voice request "detect gear box abnormal noise", the model can associate with industrial terms such as "gear box" and "abnormal noise detection", and generate the corresponding detection process response.

[0110] In a feasible embodiment, the training data set is subjected to sample construction processing through the training prompt set to obtain a sample data set, which includes positive sample pairs and negative sample pairs. Specifically, "semantic-visual" positive samples can be defined through a basic layer prompt, such as a "bolt" text and a bolt image, "action-haptic" positive samples can be defined through a task layer prompt, such as a "tighten" instruction and a torque force signal, and "parameter-feature" positive samples can be defined through a constraint layer prompt, such as "30 N·m" and a force signal peak value. According to a progressive training strategy, the sample data set is input into a multi-modal model to obtain a prediction result. The model loss can be calculated through a loss function, and the parameters of the multi-modal model are adjusted according to the model loss. According to the comparative learning loss function, the distance between samples of the same type in the feature space is forced to be smaller than that between samples of different types, and the cross-modal semantic alignment accuracy is improved.

[0111] In some embodiments, the sample data set is input into the multi-modal model according to the progressive training strategy to obtain a prediction result, including:

[0112] perform single-modal prompt processing on the sample data set according to the training prompt set to obtain a first training set;

[0113] perform multi-modal joint prompt processing on the sample data set according to the training prompt set to obtain a second training set;

[0114] perform industrial scene prompt processing on the sample data set according to the training prompt set to obtain a third training set;

[0115] input the first training set, the second training set and the third training set into the multi-modal model respectively for task prediction processing according to different training stages to obtain the prediction result.

[0116] In the embodiments of the present application, the sample data set is processed by single-modal prompt processing according to the training prompt set, that is, only the basic layer prompt is used to establish the field basic representation to obtain the first training set. The sample data set is processed by multi-modal joint prompt processing according to the training prompt set to obtain the second training set, that is, the task layer prompt is superimposed to guide multi-modal collaborative reasoning. The sample data set is processed by industrial scene prompt processing according to the training prompt set to obtain the third training set, that is, the three-layer prompt is fully enabled, and the model output is dynamically adjusted in combination with the real-time constraints of the production line. By inputting the first training set, the second training set and the third training set into the multi-modal model respectively for task prediction processing in different training stages, the prediction result of different training stages can be obtained.

[0117] In the following, the scheme of the embodiments of the present application will be described and explained in detail in combination with specific application examples:

[0118] The embodiments of the present application are applied to the technical field of intelligent robots and are suitable for industrial tasks such as humanoid robot assembly and quality inspection. Specifically, a multi-modal training set in the field of industrial manufacturing is obtained, which contains visual images, tactile signals and operation instruction texts. The multi-modal data is preprocessed, which can include convolution feature extraction, wavelet transform, semantic enhancement, etc. At the same time, hierarchical target prompt information containing a basic layer, a task layer and a constraint layer is generated. The preprocessed data and the prompt information are input into a multi-modal encoder, and a target multi-modal model is trained through contrastive learning and progressive strategy to obtain an industrial task processing result. In actual industrial production lines, the model is fine-tuned online through reinforcement learning. The reward function is designed based on task execution accuracy (such as assembly in-place rate) and efficiency (such as operation time consumption), so that the model adapts to the constraint conditions of a specific industrial environment.

[0119] Please refer to Figure 3 The embodiments of the present application also provide a humanoid robot multi-modal data processing system, which can implement the above-mentioned method. The system comprises:

[0120] The data acquisition module 301 is configured to acquire a multi-modal data set by the humanoid robot.

[0121] The preprocessing module 302 is configured to preprocess the multi-modal data set to obtain a preprocessed data set.

[0122] The prompt generation module 303 is configured to generate multi-layer prompt information according to an industrial scene of the humanoid robot.

[0123] The collaborative reasoning module 304 is configured to input the preprocessed data set and the multi-layer prompt information into a pre-trained multi-modal model for collaborative reasoning processing, and output an industrial task processing result.

[0124] It can be understood that the content in the above method embodiments is applicable to the present system embodiments, the present system embodiments specifically implement the functions of the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0125] The present application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor implements the above method when executing the computer program. The electronic device can be any intelligent terminal, such as a tablet computer or a vehicle-mounted computer.

[0126] It can be understood that the content in the above method embodiments is applicable to the present device embodiments, the present device embodiments specifically implement the functions of the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0127] Please refer to Figure 4 , Figure 4 The hardware structure of the electronic device of another embodiment is illustrated, and the electronic device includes:

[0128] The processor 401 can be implemented in a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, and is configured to execute related programs to implement the technical solutions provided by the present application.

[0129] The memory 402 can be implemented in the form of a Read Only Memory (ROM), a static storage device, a dynamic storage device, or a Random Access Memory (RAM), etc. The memory 402 can store an operating system and other application programs, and when the technical solutions provided by the embodiments of the present specification are implemented by software or firmware, the related program codes are stored in the memory 402 and are called and executed by the processor 401 to perform the above-mentioned method of the embodiments of the present application;

[0130] The input / output interface 403 is configured to realize information input and output.

[0131] The communication interface 404 is configured to realize the communication interaction between the device and other devices, and the communication can be realized by a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.).

[0132] The bus 405 is configured to transmit information between various components (for example, the processor 401, the memory 402, the input / output interface 403, and the communication interface 404) of the device.

[0133] The processor 401, the memory 402, the input / output interface 403, and the communication interface 404 are connected to each other through the bus 405 to realize the communication connection between the device.

[0134] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to realize the above-mentioned method.

[0135] It can be understood that the contents in the above-mentioned method embodiments are all applicable to the present storage medium embodiments, the functions specifically realized by the present storage medium embodiments are the same as those of the above-mentioned method embodiments, and the beneficial effects achieved by the present storage medium embodiments are also the same as those achieved by the above-mentioned method embodiments.

[0136] The embodiments of the present application also provide a computer program product, which includes a computer program. The computer program is executed by a processor to realize the above-mentioned method.

[0137] It can be understood that the contents in the above-mentioned method embodiments are all applicable to the present program product embodiments, the functions specifically realized by the present program product embodiments are the same as those of the above-mentioned method embodiments, and the beneficial effects achieved by the present program product embodiments are also the same as those achieved by the above-mentioned method embodiments.

[0138] The memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory that is remotely arranged relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0139] The embodiment of the present application provides a kind of humanoid robot multi-modal data processing method, system, equipment and medium, the scheme is obtained by humanoid robot acquisition Multi-modal data set, multi-modal data can be processed, improve the processing capacity of data of humanoid robot.And, the scheme generates multi-layer prompt information according to the industrial scene of humanoid robot, can generate different prompt information according to specific industrial scene to guide generation to the processing of model, can flexibly adjust processing strategy according to the characteristics of different industrial tasks, improve the task processing capability under complex industrial scene.In addition, the scheme is by inputting pre-processing data set and multi-layer prompt information into pre-trained multi-modal model to carry out collaborative inference processing, output obtains industrial task processing result, can make humanoid robot effectively fuse multi-modal information such as vision and touch, adapt to the precision operation demand in industrial scene, improve the accuracy of robot execution industrial task.

[0140] The embodiment described in the present application is to more clearly illustrate the technical scheme of the present application, and does not constitute a limitation on the technical scheme provided by the present application, and those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical scheme provided by the present application is also applicable to similar technical problems.

[0141] Those skilled in the art can understand that the technical scheme shown in the figure does not constitute a limitation on the present application, and can include more or fewer steps than the figure, or combine certain steps, or different steps.

[0142] The system embodiments described above are only schematic, and the units described as separate components can be or can not be physically separated, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment.

[0143] Those skilled in the art can understand that all or some steps in the method disclosed above, the function modules / units in the system and the equipment can be implemented as software, firmware, hardware and appropriate combinations thereof.

[0144] The terms "first", "second", "third", "fourth", and the like in the description of this application and in the claims hereof, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so termed herein is solely for descriptive purposes and not for pronouncing the limitations of the application described except as described in the claims. Moreover, the terms "comprise", "have", "contain" and "include" and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, product or apparatus that comprises, has, contains or includes a list of steps or elements, but not those not expressly listed or inherent to such process, method, system, product or apparatus, is not excluded from the scope of this application.

[0145] It should be understood that, in this application, "at least one" means one or more, "multiple" means two or more. "And / or" is used to describe the relationship between associated objects, which means that there can be three relationships, for example, "A and / or B" can mean: only A, only B, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or similar expressions means any combination of these items, including single or multiple combinations. For example, at least one of a, b or c, can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0146] In several embodiments provided in the present application, it should be understood that the disclosed system and method can be implemented in other ways. For example, the above-described system embodiments are only illustrative, for example, the division of the above-mentioned units is only a logical functional division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be through some interface, indirect coupling or communication connection between systems or units, which can be electrical, mechanical or other forms.

[0147] The units described above as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0148] In addition, each of the functional units in the embodiments of the present application can be integrated in one processing unit, or each unit can exist alone physically, or two or more units can be integrated in one unit. The integrated unit can be implemented in the form of hardware, or in the form of a software functional unit.

[0149] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such an understanding, the technical solutions of the present application, essentially or partially, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions used to cause a computer device (such as a personal computer, a server, or a network device) to perform all or part of the steps of the methods in the embodiments of the present application. The foregoing storage medium includes: various memories (such as a read-only memory, a random access memory, a flash memory, or the like) and a magnetic disk or an optical disk and the like.

[0150] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not intended to limit the scope of the embodiments of the present application. Any modification, equivalent replacement, and improvement made by those skilled in the art without departing from the scope and spirit of the embodiments of the present application shall fall within the scope of the embodiments of the present application.

Claims

1. A humanoid robot multi-modal data processing method, characterized by, The method comprises the following steps: A multi-modal data set is collected by a humanoid robot; The multi-modal data set is preprocessed to obtain a preprocessed data set; Multi-layer prompt information is generated according to the industrial scene of the humanoid robot; The preprocessed data set and the multi-layer prompt information are input into a pre-trained multi-modal model for collaborative reasoning processing, and an industrial task processing result is output. The multi-modal data set comprises visual images, tactile signals and operation instruction texts, and the preprocessing of the multi-modal data set comprises: Contour feature extraction processing is performed on the visual images, edge detection operators are used to identify part contours, and morphological operations are used to optimize contour continuity to obtain contour features; Defect detection and vector generation processing are performed on the visual images according to the contour features, multi-dimensional feature vectors are generated for potential defect areas by a feature extraction network, and visual features are obtained; Time-frequency domain transformation and frequency band feature extraction processing are performed on the tactile signals, time domain signals are decomposed into feature components of different frequency bands by wavelet transformation, impact features of high frequency bands and trend features of low frequency bands are extracted, and tactile features are obtained; The operation instruction texts are subjected to semantic enhancement processing according to a preset industrial field vocabulary library to obtain semantic enhancement texts; The semantic enhancement texts are subjected to semantic vector conversion and constraint feature addition processing to obtain text features; The preprocessed data set is obtained according to the visual features, the tactile features and the text features; The multi-layer prompt information is generated according to the industrial scene of the humanoid robot, comprising: Role attribute prompt generation processing is performed on the humanoid robot according to the industrial scene to obtain basic layer prompt information; Industrial task prompt generation processing is performed on the humanoid robot according to the industrial scene to obtain task layer prompt information; Task constraint prompt generation processing is performed on the humanoid robot according to the industrial scene to obtain constraint layer prompt information; The multi-layer prompt information is obtained according to the basic layer prompt information, the task layer prompt information and the constraint layer prompt information.

2. The method of claim 1, wherein, The preprocessed data set and the multi-layer prompt information are input into a pre-trained multi-modal model for collaborative reasoning processing, and an industrial task processing result is output. Feature extraction and knowledge graph embedding processing are performed on the preprocessed data set to obtain visual recognition features; Time sequence dependency capturing processing is performed on the preprocessed data set to obtain tactile recognition features; The multi-layer prompt information is encoded to obtain a prompt feature vector; The visual recognition features and the tactile recognition features are subjected to cross-modal fusion processing according to the prompt feature vector to obtain the industrial task processing result.

3. The method of claim 2, wherein, The visual recognition features are subjected to attention weight calculation processing according to the prompt feature vector to obtain visual task attention. ​ According to the prompt feature vector, attention weight calculation processing is performed on the haptic recognition feature to obtain a haptic task attention; According to the visual task attention and the haptic task attention, feature fusion processing is performed on the visual recognition feature and the haptic recognition feature to obtain a fusion feature; According to the fusion feature, task output prediction processing is performed to obtain the industrial task processing result.

4. The method of claim 1, wherein, Before the preprocessed data set and the multi-layer prompt information are input into the pre-trained multi-modal model for collaborative inference processing, the method further comprises training the multi-modal model, comprising: obtaining a training data set and a training prompt set; According to the training prompt set, sample construction processing is performed on the training data set to obtain a sample data set; the sample data set includes positive sample pairs and negative sample pairs; According to the progressive training strategy, the sample data set is input into the multi-modal model to obtain a prediction result; According to the prediction result, a model loss is calculated, and the parameters of the multi-modal model are adjusted according to the model loss.

5. The method of claim 4, wherein, According to the progressive training strategy, the sample data set is input into the multi-modal model to obtain a prediction result, comprising: According to the training prompt set, single-modal prompt processing is performed on the sample data set to obtain a first training set; According to the training prompt set, multi-modal joint prompt processing is performed on the sample data set to obtain a second training set; According to the training prompt set, industrial scene prompt processing is performed on the sample data set to obtain a third training set; According to different training stages, the first training set, the second training set and the third training set are respectively input into the multi-modal model for task prediction processing to obtain the prediction result.

6. A humanoid robot multi-modal data processing system, characterized by, The system is applied to the humanoid robot multi-modal data processing method of claim 1, and the system comprises: a data acquisition module for acquiring a multi-modal data set through a humanoid robot; a preprocessing module for preprocessing the multi-modal data set to obtain a preprocessed data set; a prompt generation module for generating multi-layer prompt information according to the industrial scene of the humanoid robot; a collaborative inference module for inputting the preprocessed data set and the multi-layer prompt information into a pre-trained multi-modal model for collaborative inference processing to output an industrial task processing result.

7. An electronic device, comprising: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method of any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 7. The computer program is executed by the processor to implement the method of any one of claims 1 to 5.

Images