A multimodal collaborative decision-making method and device for humanoid robots in industrial scenarios

Abstract

This application discloses a multimodal collaborative decision-making method and device for a humanoid robot in an industrial scenario, relating to the field of robot control technology. The method includes: obtaining multi-frame monitoring images, force and tactile sensor data, and equipment operation sensor data from the humanoid robot's visual camera in the industrial scenario; extracting feature vectors of the multi-frame monitoring images, force and tactile sensor data, and equipment operation sensor data as multimodal feature vectors; using the multimodal feature vectors as training samples to train a cross-modal model, so that the cross-modal model learns the correlation between vision, force and tactile, and equipment data; obtaining dynamic industrial scene data, and then extracting feature vectors and inputting them into the trained cross-modal model to generate scene description information; outputting operation decision instructions based on the scene description information to control the humanoid robot. Through cross-modal feature fusion and dynamic knowledge reasoning, this application can improve the task execution accuracy and environmental adaptability of humanoid robots in complex industrial environments.

Description

Technical Field

[0001] The present application relates to the field of robot control technology, and in particular to a multimodal collaborative decision-making method and device for a humanoid robot in an industrial scenario. Background Art

[0002] In industrial scenarios, the application of humanoid robots faces the following technical problems: 1. Isolated processing of multimodal data: Visual images, force and tactile signals and equipment sensor data are analyzed independently, lacking cross-modal semantic associations. For example, when a robot grasps a part, it only recognizes the shape through vision and does not adjust the grasping force in combination with force and tactile feedback, which can easily lead to damage to the part. 2. Lagging industrial knowledge updates: Traditional decision-making systems rely on pre-programmed rules and cannot incorporate emerging equipment failure modes or assembly processes in real time. For example, when the assembly parameters of new model parts change, it is difficult for robots to learn autonomously. 3. Insufficient adaptability to complex tasks: In industrial scenarios, robots need to handle visual positioning, force control operations and equipment status monitoring at the same time. Existing methods make it difficult to achieve collaborative optimization of multiple tasks, resulting in low production efficiency. Summary of the Invention

[0003] The main purpose of the embodiments of the present application is to propose a multimodal collaborative decision-making method and device for humanoid robots in industrial scenarios, so as to improve the task execution accuracy and environmental adaptability of humanoid robots in complex industrial environments.

[0004] To achieve the above objectives, one aspect of an embodiment of the present application provides a multimodal collaborative decision-making method for a humanoid robot in an industrial scenario, the method comprising the following steps:

[0005] Obtain multi-frame monitoring images from the humanoid robot's visual camera, force and tactile sensor data, and equipment operation sensor data in industrial scenarios;

[0006] Extracting feature vectors of multiple frames of the monitoring image, feature vectors of the force and tactile sensor data, and time series feature vectors of the device operation sensor data as multimodal feature vectors;

[0007] Using the multimodal feature vector as a training sample to train a cross-modal model, so that the cross-modal model learns the correlation between vision, force and touch, and device data;

[0008] Acquiring dynamic industrial scene data, extracting feature vectors of the dynamic industrial scene data and inputting the feature vectors into the trained cross-modal model to generate scene description information;

[0009] Outputting operation decision instructions according to the scene description information to control the humanoid robot.

[0010] In some embodiments, extracting feature vectors of multiple frames of surveillance images includes the following steps:

[0011] Extracting key frames from multiple frames of monitoring images;

[0012] Extracting a feature vector from the key frame using a pre-trained general visual feature extraction model as a feature vector of the monitoring image;

[0013] The forward propagation function of the general visual feature extraction model is:

[0014] =M ;

[0015] Wherein, M represents the general visual feature extraction model, represents the feature vector extracted from the key frame, represents the key frame.

[0016] In some embodiments, extracting key frames from multiple frames of surveillance images includes the following steps:

[0017] Extracting the key frames from the multiple monitoring images using an inter-frame difference algorithm;

[0018] The expression of the inter-frame difference algorithm is:

[0019] ;

[0020] in, are the width and height of the monitoring image respectively; Respectively Pixel values ​​of the monitoring image of the frame; is the coordinate of the pixel value; when When a preset threshold is reached, the monitoring image is extracted as a key frame.

[0021] In some embodiments, the step of extracting the feature vector of the force tactile sensor data includes the following steps:

[0022] Performing sliding window processing on the force tactile sensor data to obtain window data;

[0023] The expression of the sliding window is:

[0024] ;

[0025] in, For the The force tactile sensor data at time n is the window coefficient; t is the duration of the window; is the window data;

[0026] The tactile texture features of the window data are extracted using a general tactile feature extraction model of a convolutional neural network structure as feature vectors of the force tactile sensor data.

[0027] In some embodiments, the method of training a cross-modal model using the multimodal feature vector as a training sample so that the cross-modal model learns the association between vision, force and touch, and device data includes the following steps:

[0028] Mapping the multimodal feature vectors to the same feature space to obtain sample features;

[0029] Training the cross-modal model using a contrastive loss function based on the sample features, so that the cross-modal model learns the correlation between vision, force and touch, and device data;

[0030] The contrast loss function is:

[0031] ;

[0032] in, is the contrast loss function, is the feature vector of the monitoring image, is the cosine similarity, is the coefficient of the contrast loss function; is a negative sample feature; is the number of negative samples, is the time series feature vector.

[0033] In some embodiments, outputting an operation decision instruction based on the scene description information to control the humanoid robot includes the following steps:

[0034] The retrieval enhancement generation system outputs the operation decision instruction according to the scene description information and the industrial equipment knowledge base to control the humanoid robot.

[0035] In some embodiments, the step of building the industrial equipment knowledge base includes the following steps:

[0036] A general knowledge graph construction tool based on a graph database structure constructs an equipment knowledge graph as the industrial equipment knowledge base; wherein the nodes of the equipment knowledge graph represent components and failure modes, and the edges of the equipment knowledge graph represent assembly relationships and failure associations;

[0037] The expression for constructing the device knowledge graph is:

[0038] ;

[0039] in, is the device knowledge graph, is a node set, is the edge set, is the identifier of the total number of nodes, Represents an edge between nodes.

[0040] To achieve the above objectives, another aspect of the present application provides a multimodal collaborative decision-making device for a humanoid robot in an industrial scenario, the device comprising:

[0041] A data acquisition unit, used to acquire multi-frame monitoring images, force and tactile sensor data, and equipment operation sensor data from the humanoid robot's visual camera in the industrial scene;

[0042] a feature extraction unit, configured to extract feature vectors of multiple frames of the monitoring image, feature vectors of the force and tactile sensor data, and time series feature vectors of the device operation sensor data as multimodal feature vectors;

[0043] a model training unit, configured to train a cross-modal model using the multimodal feature vector as a training sample, so that the cross-modal model learns the correlation between vision, force and touch, and device data;

[0044] An information acquisition unit, configured to acquire dynamic industrial scene data, extract feature vectors of the dynamic industrial scene data, and input the feature vectors into the trained cross-modal model to generate scene description information;

[0045] A decision control unit is used to output an operation decision instruction according to the scene description information to control the humanoid robot.

[0046] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned method when executing the computer program.

[0047] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements the above-mentioned method when executed by a processor.

[0048] The embodiments of the present application include at least the following beneficial effects:

[0049] This application provides a scheme that obtains multi-frame monitoring images, force and tactile sensor data, and equipment operation sensor data from a humanoid robot's visual camera in an industrial scene; extracts feature vectors of the multi-frame monitoring images, feature vectors of the force and tactile sensor data, and time-series feature vectors of the equipment operation sensor data as multimodal feature vectors; uses the multimodal feature vectors as training samples to train a cross-modal model, so that the cross-modal model learns the correlation between vision, force and tactile, and equipment data; obtains dynamic industrial scene data, extracts feature vectors of the dynamic industrial scene data, and inputs them into the trained cross-modal model to generate scene description information; outputs operation decision instructions based on the scene description information to control the humanoid robot. This application can improve the task execution accuracy and environmental adaptability of humanoid robots in complex industrial environments through cross-modal feature fusion and dynamic knowledge reasoning. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0051] Figure 1 A flowchart of a multimodal collaborative decision-making method for a humanoid robot in an industrial scenario provided in an embodiment of the present application;

[0052] Figure 2 An example flow chart of a multimodal collaborative decision-making method for a humanoid robot in an industrial scenario provided in an embodiment of the present application;

[0053] Figure 3 An example flowchart of training a cross-modal model provided in an embodiment of the present application;

[0054] Figure 4 This is an example diagram of the industrial equipment knowledge graph provided in the embodiments of this application;

[0055] Figure 5 A schematic diagram of the structure of a multimodal collaborative decision-making device for a humanoid robot in an industrial scenario provided in an embodiment of the present application;

[0056] Figure 6 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are merely examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the appended claims.

[0058] It will be understood that the terms "first", "second", etc. used in this application may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0059] The terms "at least one", "plurality", "each", "any", etc. used in this application include "at least one", "two" or more, "plurality" or "each", "any" or "any one", "each" or "any one" as used herein.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0061] The embodiment of the present application provides a method and device for multimodal collaborative decision-making of a humanoid robot in an industrial scene, which relates to the field of robot control technology. The method and device for multimodal collaborative decision-making of a humanoid robot in an industrial scene provided by the embodiment of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements a multimodal collaborative decision-making method for a humanoid robot in an industrial scene, etc., but is not limited to the above forms.

[0062] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0063] Reference Figure 1 The embodiment of the present application provides a multimodal collaborative decision-making method for a humanoid robot in an industrial scenario. The method may include but is not limited to S100 to S140, as follows:

[0064] S100: Acquires multi-frame monitoring images from the humanoid robot's visual camera, force and tactile sensor data, and equipment operation sensor data in industrial scenarios;

[0065] S110: Extracting feature vectors of multiple frames of the monitoring image, feature vectors of the force and tactile sensor data, and time series feature vectors of the device operation sensor data as multimodal feature vectors;

[0066] S120: Using the multimodal feature vector as a training sample to train a cross-modal model, so that the cross-modal model learns the correlation between vision, force and touch, and device data;

[0067] S130: Acquire dynamic industrial scene data, extract feature vectors of the dynamic industrial scene data, and input the feature vectors into the trained cross-modal model to generate scene description information;

[0068] S140: Outputting an operation decision instruction according to the scene description information to control the humanoid robot.

[0069] Optionally, extracting feature vectors of multiple frames of surveillance images includes the following steps:

[0070] Extracting key frames from multiple frames of monitoring images;

[0071] Extracting a feature vector from the key frame using a pre-trained general visual feature extraction model as a feature vector of the monitoring image;

[0072] The forward propagation function of the general visual feature extraction model is:

[0073] =M ;

[0074] Wherein, M represents the general visual feature extraction model, represents the feature vector extracted from the key frame, represents the key frame.

[0075] Optionally, extracting key frames from multiple frames of surveillance images includes the following steps:

[0076] Extracting the key frames from the multiple monitoring images using an inter-frame difference algorithm;

[0077] The expression of the inter-frame difference algorithm is:

[0078] ;

[0079] in, are the width and height of the monitoring image respectively; Respectively Pixel values ​​of the monitoring image of the frame; is the coordinate of the pixel value; when When a preset threshold is reached, the monitoring image is extracted as a key frame.

[0080] Optionally, the step of extracting the feature vector of the force tactile sensor data includes the following steps:

[0081] Performing sliding window processing on the force tactile sensor data to obtain window data;

[0082] The expression of the sliding window is:

[0083] ;

[0084] in, For the The force tactile sensor data at time n is the window coefficient; t is the duration of the window; is the window data;

[0085] The tactile texture features of the window data are extracted using a general tactile feature extraction model of a convolutional neural network structure as feature vectors of the force tactile sensor data.

[0086] Optionally, the training of a cross-modal model using the multimodal feature vector as a training sample so that the cross-modal model learns the association between vision, force and touch, and device data comprises the following steps:

[0087] Mapping the multimodal feature vectors to the same feature space to obtain sample features;

[0088] Training the cross-modal model using a contrastive loss function based on the sample features, so that the cross-modal model learns the correlation between vision, force and touch, and device data;

[0089] The contrast loss function is:

[0090] ;

[0091] in, is the contrast loss function, is the feature vector of the monitoring image, is the cosine similarity, is the coefficient of the contrast loss function; is a negative sample feature; is the number of negative samples, is the time series feature vector.

[0092] Optionally, outputting an operation decision instruction according to the scene description information and then controlling the humanoid robot comprises the following steps:

[0093] The retrieval enhancement generation system outputs the operation decision instruction according to the scene description information and the industrial equipment knowledge base to control the humanoid robot.

[0094] Optionally, the step of constructing the industrial equipment knowledge base includes the following steps:

[0095] A general knowledge graph construction tool based on a graph database structure constructs an equipment knowledge graph as the industrial equipment knowledge base; wherein the nodes of the equipment knowledge graph represent components and failure modes, and the edges of the equipment knowledge graph represent assembly relationships and failure associations;

[0096] The expression for constructing the device knowledge graph is:

[0097] ;

[0098] in, is the device knowledge graph, is a node set, is the edge set, is the identifier of the total number of nodes, Represents an edge between nodes.

[0099] Next, the solution of the embodiment of the present application will be introduced and explained in detail with reference to specific application examples.

[0100] This embodiment is applicable to industrial scenarios such as parts assembly, fault diagnosis, and material handling.

[0101] The technical solutions of this embodiment include:

[0102] 1. Collaborative extraction of modal features: This approach integrates visual, force, tactile, and device sensor data in industrial scenarios. High-dimensional features are extracted through a general visual feature extraction model and a general tactile feature extraction model based on a convolutional neural network to construct a cross-modal semantic representation.

[0103] 2. Cross-modal model training: Using contrastive learning and attention mechanisms, multimodal features are mapped to the same space to learn the association patterns of visual, tactile, and equipment data in industrial operations, such as the correspondence between part appearance and gripping force.

[0104] 3. Dynamic construction of industrial knowledge base: Integrate real-time scenario descriptions with knowledge such as equipment manuals and fault cases, and use general knowledge graph construction tools based on graph database structures to build a graph-structured knowledge base that can be updated in real time to support reasoning about assembly processes, fault diagnosis, and other issues.

[0105] 4. Retrieval-enhanced generation system decision generation: Relevant industrial knowledge in the knowledge base is obtained through the retrieval component, and the generation component combines multimodal features to output operation instructions, such as part assembly angles, equipment maintenance processes, etc.

[0106] Specifically, refer to Figure 2 , this embodiment may include the following steps:

[0107] Obtain multi-frame monitoring images from the humanoid robot's visual camera, force and tactile sensor data, and equipment operation sensor data in industrial scenarios, and extract image feature vectors, force and tactile feature vectors, and time series feature vectors;

[0108] Use multimodal feature vectors as training samples to train a cross-modal model and learn the correlation between vision, force, touch, and device data;

[0109] Acquire real-time industrial scene data, extract features, and input them into a trained cross-modal model to generate scene description information;

[0110] Based on the scene description information and the industrial equipment knowledge base, the robot operation decision instructions are output by the retrieval enhancement generation system.

[0111] Exemplarily, the step of extracting the image feature vector includes:

[0112] The inter-frame difference algorithm is used to extract key frames from the industrial scene video sequence. The difference calculation formula is:

[0113] ;

[0114] in, is the image width and height, For the Frame pixel value, when Extract as keyframes.

[0115] For example, the image feature vector is extracted by a pre-trained general visual feature extraction model, whose forward propagation function is:

[0116] =M ;

[0117] The output dimension is 1024-dimensional image semantic features.

[0118] Exemplarily, the step of extracting the force tactile feature vector includes:

[0119] Perform sliding window processing on the force sensor data, and the window function is:

[0120] ;

[0121] in, For the moment force value, n=50;

[0122] The tactile texture features are extracted through the general tactile feature extraction model of the convolutional neural network structure, and the output dimension is a 256-dimensional force tactile feature vector. .

[0123] Exemplarily, the steps of training the cross-modal model include:

[0124] Map multimodal features to the same feature space and use the contrast loss function to optimize the model:

[0125] ;

[0126] in, is the contrast loss function, is the feature vector of the monitoring image, is the cosine similarity, is the coefficient of the contrast loss function; is a negative sample feature; is the number of negative samples, A time series feature vector of sensor data for device operation.

[0127] In one embodiment, Is the temperature parameter as the coefficient of the contrast loss function. It is 0.5 (medium temperature), and the application scenario is such as parts assembly work in normal temperature industrial workshops. It is 1.0 (high temperature), and its application scenario is such as removing heat-treated parts from a heat treatment furnace.

[0128] Exemplarily, the steps of building an industrial equipment knowledge base include:

[0129] A general knowledge graph construction tool based on the graph database structure constructs a device knowledge graph. The nodes are "components" and "failure modes", and the edges are "assembly relationships" and "failure associations". The graph construction function is:

[0130] ;

[0131] in, is the identifier of the total number of nodes, Represents the relationship between nodes.

[0132] Reference Figure 3 , a more specific implementation method is described below.

[0133] 1. Modal data acquisition and feature extraction.

[0134] 1.1 Industrial visual data processing:

[0135] The humanoid robot's head camera collects production line videos at 30fps and calculates the inter-frame difference formula. Extract key frames in real time to reduce data redundancy.

[0136] A general visual feature extraction model pre-trained with keyframe input. This model has a multi-layer convolution and pooling structure and can extract a 1024-dimensional visual feature vector containing part contours and surface textures. , which can capture the global and local semantics of parts.

[0137] 1.2 Force tactile data processing:

[0138] The six-dimensional force sensor of the robot hand samples the force data at 1kHz and uses a sliding window function (window size n=50) smoothing to eliminate noise interference.

[0139] The smoothed data is input into a general tactile feature extraction model based on a convolutional neural network structure (such as a residual connection structure similar to ResNet), which extracts tactile texture features (such as rough / smooth surface) and outputs a 256-dimensional force tactile feature vector. , and simultaneously records the sensor coordinate information to generate a three-dimensional tactile representation containing the spatial position.

[0140] 1.3 Equipment sensor data processing:

[0141] The sensor data of machine tool vibration, temperature, etc. is transmitted to the robot controller through the industrial bus. The time series features are extracted using a general time series feature extraction model with a recurrent neural network structure (one embodiment is LSTM). The hidden layer dimension is 128 and the output is a 512-dimensional time series feature vector. , capturing the dynamic changes of the equipment's operating status.

[0142] 2. Cross-modal model training.

[0143] 2.1 Feature Space Mapping:

[0144] Design a cross-modal encoder, including visual branch, force tactile branch and timing branch, through the fully connected layer Mapped to a 1024-dimensional common feature space, the mapping function is:

[0145] ;

[0146] in, The high-dimensional feature vector (force tactile feature vector) extracted after processing the tactile sensor data is used to characterize the force value change, texture characteristics and spatial position information when the object contacts. represents feature splicing, is the weight matrix, is the bias vector.

[0147] 2.2 Contrastive Learning Optimization:

[0148] Construct positive sample pairs (such as visual-tactile features of the "bolt tightening" operation) and negative sample pairs (different operation features), using the contrast loss function Optimize the model to reduce the distance between related features in the feature space and increase the distance between irrelevant features, thereby improving the accuracy of cross-modal association.

[0149] 3. Industrial scene description generation and knowledge base construction.

[0150] 3.1 Real-time scene processing:

[0151] The robot collects multimodal data in real time, inputs the trained cross-modal model, outputs fused features, and generates natural language descriptions through a decoder. One example is: "In the assembly station, vision detects that the M5 bolt is missing, force tactile feedback shows that the gripper is currently subjected to a force of 0.8N, and the machine tool vibration sensor reading exceeds the threshold by 15%."

[0152] 3.2 Construction of industrial knowledge base:

[0153] Reference Figure 4 Utilizing a general knowledge graph construction tool based on a graph database structure (capable of creating nodes and edges and reasoning about relationships), we construct a device knowledge graph. Nodes include components like "bolts" and "motor," as well as failure modes like "insufficient tightening torque," and edges include relationships like "assembly in" and "failure cause." Entities in new scenarios (such as new sensor models) are automatically created with nodes, and graph embedding algorithms (such as TransE) are used to infer their associations with existing nodes, such as the connection between the "new sensor" and "temperature monitoring" nodes.

[0154] 4. Retrieval-enhanced generation system decision generation.

[0155] 4.1 Retrieval component:

[0156] Receive key information from the current scenario description (such as "missing bolts") and use vector retrieval tools (such as the Faiss-based similarity search algorithm) to search for similar cases in the knowledge base, such as "assembly process adjustment plan when a certain model of bolt is missing," and return relevant knowledge fragments (such as tightening sequence and torque parameters).

[0157] 4.2 Generate components:

[0158] A general generative model with sequence-to-sequence generation capabilities is used as the generator. It takes as input a "scene description + search results" and outputs robot operation instructions. For example, "Move to the bolt hopper, grasp an M5 bolt with a force of 1.2N, and assemble it to the motor flange in a clockwise direction, with a tightening torque of 8±0.5N·m." These instructions are converted into joint angle control signals using an inverse kinematics algorithm to drive the robot to perform the assembly task.

[0159] Beneficial effects:

[0160] Precision industrial operation: Multimodal feature fusion enables robots to combine visual part recognition with force and tactile feedback to achieve precision assembly. For example, force-vision collaborative control during chip welding can reduce the error to within 0.1mm.

[0161] Real-time fault warning: By analyzing equipment images and vibration sensor data through cross-modal models, faults such as bearing wear can be identified in advance. For example, when visual detection of heat sink anomalies and vibration spectrum peak shifts is detected, an early warning is automatically triggered.

[0162] Flexible production adaptation: A dynamic knowledge base supports the rapid learning of new parts and new processes. For example, after a robot is trained with visual, force, and tactile data on multiple sets of new parts, it can autonomously generate assembly strategies to adapt to production line changes.

[0163] Reference Figure 5 The present application also provides a multimodal collaborative decision-making device for a humanoid robot in an industrial scenario, which can implement the above-mentioned multimodal collaborative decision-making method for a humanoid robot in an industrial scenario. The device includes:

[0164] A data acquisition unit, used to acquire multi-frame monitoring images, force and tactile sensor data, and equipment operation sensor data from the humanoid robot's visual camera in the industrial scene;

[0165] a feature extraction unit, configured to extract feature vectors of multiple frames of the monitoring image, feature vectors of the force and tactile sensor data, and time series feature vectors of the device operation sensor data as multimodal feature vectors;

[0166] a model training unit, configured to train a cross-modal model using the multimodal feature vector as a training sample, so that the cross-modal model learns the correlation between vision, force and touch, and device data;

[0167] An information acquisition unit, configured to acquire dynamic industrial scene data, extract feature vectors of the dynamic industrial scene data, and input the feature vectors into the trained cross-modal model to generate scene description information;

[0168] A decision control unit is used to output an operation decision instruction according to the scene description information to control the humanoid robot.

[0169] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0170] The present application also provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method of the present application. The electronic device can be any smart terminal, such as a tablet computer or an in-vehicle computer.

[0171] It can be understood that the contents of the above method embodiments are all applicable to the embodiments of the present device, the functions specifically implemented by the embodiments of the present device are the same as those of the method of the present application, and the beneficial effects achieved are also the same as those achieved by the method of the present application.

[0172] See also Figure 6 , Figure 6 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0173] The processor 601 may be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0174] The memory 602 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 602 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 602 and is called by the processor 601 to execute the methods of the embodiments of this application.

[0175] Input / output interface 603, used to implement information input and output;

[0176] Communication interface 604, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0177] Bus 605 , which transmits information between various components of the device (e.g., processor 601 , memory 602 , input / output interface 603 , and communication interface 604 );

[0178] The processor 601 , the memory 602 , the input / output interface 603 and the communication interface 604 are connected to each other in communication within the device via a bus 605 .

[0179] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the method of the present application is implemented.

[0180] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiment, the functions specifically implemented by the present storage medium embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0181] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0182] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0183] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0184] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0185] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0186] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0187] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0188] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0189] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0190] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0191] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0192] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A multimodal collaborative decision-making method for humanoid robots in industrial scenarios, characterized by: The method comprises the following steps: Obtain multi-frame monitoring images from the humanoid robot's visual camera, force and tactile sensor data, and equipment operation sensor data in industrial scenarios; Extracting feature vectors of multiple frames of the monitoring image, feature vectors of the force and tactile sensor data, and time series feature vectors of the device operation sensor data as multimodal feature vectors; Using the multimodal feature vector as a training sample to train a cross-modal model, so that the cross-modal model learns the correlation between vision, force and touch, and device data; Acquiring dynamic industrial scene data, extracting feature vectors of the dynamic industrial scene data and inputting the feature vectors into the trained cross-modal model to generate scene description information; Outputting an operation decision instruction according to the scene description information to control the humanoid robot; The step of extracting feature vectors of multiple frames of surveillance images comprises the following steps: Extracting key frames from multiple frames of monitoring images; Extracting a feature vector from the key frame using a pre-trained general visual feature extraction model as a feature vector of the monitoring image; The forward propagation function of the general visual feature extraction model is: =M ； Wherein, M represents the general visual feature extraction model, represents the feature vector extracted from the key frame, represents the key frame; Extracting key frames from multiple frames of monitoring images includes the following steps: Extracting the key frames from the multiple monitoring images using an inter-frame difference algorithm; The expression of the inter-frame difference algorithm is: ； in, are the width and height of the monitoring image respectively; Respectively Pixel values ​​of the monitoring image of the frame; is the coordinate of the pixel value; when When a preset threshold is reached, the monitoring image is extracted as a key frame.

2. The multimodal collaborative decision-making method for humanoid robots in industrial scenarios according to claim 1 is characterized in that: The step of extracting the feature vector of the force tactile sensor data comprises the following steps: Performing sliding window processing on the force tactile sensor data to obtain window data; The expression of the sliding window is: ； in, For the The force tactile sensor data at time n is the window coefficient; t is the duration of the window; is the window data; The tactile texture features of the window data are extracted using a general tactile feature extraction model of a convolutional neural network structure as feature vectors of the force tactile sensor data.

3. The multimodal collaborative decision-making method for humanoid robots in industrial scenarios according to claim 1 is characterized in that: The method of training a cross-modal model using the multimodal feature vector as a training sample so that the cross-modal model learns the association between vision, force and touch, and device data includes the following steps: Mapping the multimodal feature vectors to the same feature space to obtain sample features; Training the cross-modal model using a contrastive loss function based on the sample features, so that the cross-modal model learns the correlation between vision, force and touch, and device data; The contrast loss function is: ； in, is the contrast loss function, is the feature vector of the monitoring image, is the cosine similarity, is the coefficient of the contrast loss function; is a negative sample feature; is the number of negative samples, is the time series feature vector.

4. The multimodal collaborative decision-making method for a humanoid robot in an industrial scenario according to any one of claims 1 to 3, characterized in that: Outputting an operation decision instruction according to the scene description information and then controlling the humanoid robot comprises the following steps: The retrieval enhancement generation system outputs the operation decision instruction according to the scene description information and the industrial equipment knowledge base to control the humanoid robot.

5. The multimodal collaborative decision-making method for humanoid robots in industrial scenarios according to claim 4, characterized in that: The steps of constructing the industrial equipment knowledge base include the following steps: A general knowledge graph construction tool based on a graph database structure constructs an equipment knowledge graph as the industrial equipment knowledge base; wherein the nodes of the equipment knowledge graph represent components and failure modes, and the edges of the equipment knowledge graph represent assembly relationships and failure associations; The expression for constructing the device knowledge graph is: ； in, is the device knowledge graph, is a node set, is the edge set, is the identifier of the total number of nodes, Represents an edge between nodes.

6. A multimodal collaborative decision-making device for humanoid robots in industrial scenarios, characterized by: The device comprises: A data acquisition unit, used to acquire multi-frame monitoring images, force and tactile sensor data, and equipment operation sensor data from the humanoid robot's visual camera in the industrial scene; a feature extraction unit, configured to extract feature vectors of multiple frames of the monitoring image, feature vectors of the force and tactile sensor data, and time series feature vectors of the device operation sensor data as multimodal feature vectors; a model training unit, configured to train a cross-modal model using the multimodal feature vector as a training sample, so that the cross-modal model learns the correlation between vision, force and touch, and device data; An information acquisition unit, configured to acquire dynamic industrial scene data, extract feature vectors of the dynamic industrial scene data, and input the feature vectors into the trained cross-modal model to generate scene description information; A decision control unit, configured to output an operation decision instruction according to the scene description information and thereby control the humanoid robot; The step of extracting feature vectors of multiple frames of surveillance images comprises the following steps: Extracting key frames from multiple frames of monitoring images; Extracting a feature vector from the key frame using a pre-trained general visual feature extraction model as a feature vector of the monitoring image; The forward propagation function of the general visual feature extraction model is: =M ； Wherein, M represents the general visual feature extraction model, represents the feature vector extracted from the key frame, represents the key frame; Extracting key frames from multiple frames of monitoring images includes the following steps: Extracting the key frames from the multiple monitoring images using an inter-frame difference algorithm; The expression of the inter-frame difference algorithm is: ； in, are the width and height of the monitoring image respectively; Respectively Pixel values ​​of the monitoring image of the frame; is the coordinate of the pixel value; when When a preset threshold is reached, the monitoring image is extracted as a key frame.

7. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images