Multimodal large model training method, robot motion prediction method and processing device
Through the multimodal large model training method, combined with event cameras and RGB camera data, the problem of multimodal input processing in the existing technology of robots in complex environments is solved, and more accurate action prediction and perception capabilities are achieved, reducing data and Cost requirements.
Patent Information
- Application Number
- CN202410948190.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-16
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-07-16
AI Technical Summary
When used in robots, existing large models are difficult to achieve satisfactory multimodal input processing levels in complex environments, and have high data requirements and costs.
The multimodal large model training method is adopted, including event encoding module, image encoding module, text instruction encoding module and feature fusion encoding module. The feature fusion is carried out through cross attention and self-attention structure, and combined with event camera and RGB camera data to achieve Training and prediction of multimodal large models.
In complex environments, the accuracy and perception ability of robot motion prediction are improved, data requirements and costs are reduced, multi-modal input is supported, and the robot's performance in high-speed motion and high-low lighting environments is enhanced.
Smart Images

Figure CN118504645B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a multimodal large model training method, a robot motion prediction method and a processing device. Background Art
[0002] With the rapid development of artificial intelligence technology, big models continue to heat up as a research hotspot, showing strong development momentum and wide social influence. Big models use the Transformer model, a deep neural network model based on the self-attention mechanism, as the basic architecture. The number of parameters usually reaches billions or even trillions, and supports multimodal learning, that is, processing multiple types of data such as text, images, and audio at the same time. Through cross-modal interactive learning, the model's understanding and expression capabilities of complex real-world scenarios are enhanced.
[0003] With the maturity of large model technology and its outstanding performance in natural language processing tasks, researchers have begun to actively explore its deep integration with robotics technology to improve the intelligence level of robots and their adaptability to application scenarios.
[0004] At present, when large models are applied to robots, the supported input modes are relatively simple, and it is difficult to achieve a satisfactory level in complex environments. Summary of the invention
[0005] The purpose of this application is to provide a multi-modal large model training method, a robot motion prediction method and a processing device to at least solve the above technical problems. The various technical effects that can be produced by the optional technical solutions provided by the present invention are described in detail below.
[0006] To achieve the above-mentioned purpose, in a first aspect, the present application provides a multimodal large model training method for robot motion prediction, wherein the multimodal large model comprises an event coding module, an image coding module, a text instruction coding module and a feature fusion coding module, wherein the event coding module, the image coding module and the text instruction coding module are all connected to the feature fusion coding module; the multimodal large model training method for robot motion prediction comprises:
[0007] Acquire first event data of a first event camera, first image data of a first RGB camera, and a first text instruction;
[0008] The first event data obtained is encoded by the event encoding module to obtain a first event frame feature vector of a preset length; the first image data obtained is encoded by the image encoding module to obtain a first image feature vector of the preset length; the first text instruction obtained is encoded by the text instruction encoding module to obtain a text feature vector of the preset length;
[0009] splicing the first event frame feature vector and the first image feature vector to obtain a visual feature vector;
[0010] In the cross-attention structure and self-attention structure of the feature fusion encoding module, the visual feature vector and the text feature vector are feature fused and iteratively trained until the first difference between the fused feature vector and the robot's action sample is less than a first threshold, so as to obtain a multimodal large model.
[0011] In some embodiments, the visual feature vector and the text feature vector are subjected to feature fusion and iterative training until a first difference between the fused feature vector and the robot's action sample is less than a first threshold value to obtain a multimodal large model, including:
[0012] Calculating the difference between the fused feature vector and the action sample of the robot using a first loss function to obtain a first loss function value;
[0013] Based on the first loss function value, the weights of the event encoding module and the cross-attention structure are adjusted, and the weights of the image encoding module, the text instruction encoding module and the self-attention structure are frozen for iterative training until the first difference is less than the first threshold, so as to obtain the multimodal large model.
[0014] In some embodiments, the multimodal large model training method for robot motion prediction further includes obtaining the event encoding module;
[0015] The obtaining of the event coding module comprises:
[0016] compressing the acquired second event data into a format of an event frame;
[0017] Using a contrast learning mechanism, encoding the acquired second image data to obtain a second image feature vector, encoding the event frame to obtain a second event feature vector;
[0018] The event encoding module is obtained by training using the second image feature vector and the second event feature vector.
[0019] In some embodiments, compressing the acquired second event data into an event frame format includes:
[0020] Acquire a plurality of second event data, wherein the second event data includes an address (x, y), a polarity p, and a timestamp t, wherein the address identifies a pixel position where an event occurs; the polarity indicates a direction of brightness change; and the timestamp records a time node of brightness change;
[0021] Determine each of the second event data with a same timestamp;
[0022] A format converter is used to convert the format of the second event data with the same timestamp into the format of an event frame.
[0023] In some embodiments, the determining of each of the second event data having the same timestamp includes:
[0024] Sort each of the second event data according to timestamps;
[0025] The sorted second event data are put into a first-in-first-out queue, and the number of second event data at different timestamps and the number of second event data at the same timestamp are counted through a counter and a timer.
[0026] In some embodiments, the step of obtaining the event encoding module by training the second image feature vector and the second event feature vector includes:
[0027] Normalizing the second image feature vector and the second event feature vector respectively to obtain a normalized image vector and a normalized event vector;
[0028] Performing matrix multiplication on the normalized image vector and the normalized event vector to obtain a cosine similarity matrix;
[0029] In the horizontal direction of the cosine similarity matrix, a first softmax result of the normalized image vector relative to the normalized event vector is calculated, and in the vertical direction of the cosine similarity matrix, a second softmax result of the normalized event vector relative to the normalized image vector is calculated, and a probability similarity matrix is obtained based on the first softmax result and the second softmax result;
[0030] The horizontal loss value and the vertical loss value are calculated in the probability similarity matrix to obtain a second loss function value, until a second difference between the second loss function value and the image sample is less than a second threshold, thereby obtaining the event encoding module.
[0031] In some embodiments, calculating the lateral loss value and the longitudinal loss value in the probability similarity matrix to obtain a second loss function value includes:
[0032] Obtaining a first softmax result and a second softmax result located on a main diagonal of the probability similarity matrix;
[0033] Substituting the first softmax result and the second softmax result located on the main diagonal into the predicted probability q in the cross entropy loss function formula respectively, and setting the true probability p in the cross entropy loss function formula to 1, to obtain a horizontal loss value and a vertical loss value;
[0034] The formula of the cross entropy loss function is:
[0035] ;
[0036] The formula of the cross entropy loss function is:
[0037] in, represents the cross entropy loss function, p represents the true probability that the first softmax result and the second softmax result are equal, q represents the predicted probability that the first softmax result and the second softmax result are equal; N represents the number of the first softmax result or the second softmax result;
[0038] The lateral loss value and the longitudinal loss value are added to obtain the second loss function value.
[0039] In some embodiments, obtaining the second loss function value until a second difference between the second loss function value and the image sample is less than a second threshold, and obtaining the event encoding module, comprises:
[0040] Based on the second loss function value, the weight for encoding the event frame is adjusted, and the weight for encoding the second image data is frozen for iterative training until the second difference is less than the second threshold, thereby obtaining the event encoding module.
[0041] In a second aspect, the present application provides a robot motion prediction method, wherein the robot includes a second event camera and a second RGB camera, and the robot motion prediction method includes:
[0042] Inputting the third event data acquired by the second event camera at the current moment, the third image data acquired by the second RGB camera at the current moment, and the second text instruction into the trained multimodal large model to obtain the currently predicted motion information of the robot; the multimodal large model is trained by the multimodal large model training method for robot motion prediction according to any one of the first aspects;
[0043] The currently predicted action information is input into a long short-term memory network, and the long short-term memory network outputs the final action of the robot at the current moment based on the currently predicted action information and the historical state information of the robot.
[0044] In a third aspect, the present application provides a processing device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors are used to execute the one or more computer programs stored in the memory, so that the one or more processors execute a multimodal large model training method for robot motion prediction as described in any one of the first aspect and the robot motion prediction method as described in the second aspect.
[0045] In a fourth aspect, the present application provides an electronic device, wherein the electronic device comprises the processing device described in the third aspect.
[0046] Implementing one of the above technical solutions of this application has the following advantages or beneficial effects:
[0047] The multimodal large model training method, robot motion prediction method and processing device of the present application, when training the multimodal large model, the event encoding module enables the first event data to be converted into input that can be understood by the multimodal large model, and encodes and processes the first event data of the first event camera, the first image data of the first RGB camera and the first text instruction respectively, and extracts the first event frame feature vector, the first image feature vector and the text feature vector of the same preset length, so that the first event data, the first image data and the first text instruction can be mapped to the same high-dimensional space, providing a basis for subsequent multimodal fusion, supporting multimodal input, making the robot work more accurately in a complex environment, achieving satisfactory work, and reducing data requirements and costs.
[0048] When the robot predicts actions, the second event camera has the characteristics of high dynamic range and low latency, so it can respond well to the robot's high-speed movement and the high and low light environments it is in. The multimodal large model supports multimodal learning, that is, it simultaneously processes the second text instructions, the third event data of the third event camera, and the third image data of the second RGB camera. Through cross-modal interactive learning, the robot has stronger perception capabilities in complex scenes (high-speed movement, poor lighting conditions). At the same time, by utilizing the powerful background knowledge of the multimodal large model, the robot can have more flexible and reasonable decision-making capabilities. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. It is obvious that the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:
[0050] Figure 1is a schematic diagram of a multimodal large model for robot motion prediction according to an embodiment of the present application;
[0051] Figure 2 It is a flowchart of a multi-modal large model training method for robot motion prediction according to an embodiment of the present application;
[0052] Figure 3 It is a schematic diagram of a detailed process of step S100 in the embodiment of the present application;
[0053] Figure 4 is a schematic block diagram of event compression according to an embodiment of the present application;
[0054] Figure 5 is a schematic block diagram of an event encoding module according to an embodiment of the present application;
[0055] Figure 6 It is a flow chart of the robot motion prediction method according to an embodiment of the present application. DETAILED DESCRIPTION
[0056] In order to make the purpose, technical solutions and advantages of the present application clearer, the various exemplary embodiments to be described below will refer to the corresponding drawings, which constitute a part of the exemplary embodiments, wherein various exemplary embodiments that may be used to implement the present application are described. Unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation methods described in the following exemplary embodiments do not represent all implementation methods consistent with the present disclosure. It should be understood that they are only examples of processes, methods, and devices that are consistent with some aspects disclosed in the present application as detailed in the attached claims, and other embodiments may also be used, or the embodiments listed herein may be modified in structure and function without departing from the scope and essence of the present application.
[0057] In the description of the present application, it should be understood that the terms "first", "second", etc. are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. The term "plurality" means two or more. The terms "connected" and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, an integral connection, a mechanical connection, an electrical connection, a communication connection, a direct connection, an indirect connection through an intermediate medium, and can be the internal connection of two elements or the interaction relationship between two elements. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to the specific circumstances.
[0058] In order to illustrate the technical solution described in the present application, a specific embodiment is provided below, and only the parts related to the embodiment of the present application are shown.
[0059] Embodiment 1:
[0060] like Figure 1 As shown, the present application provides a multimodal large model for robot motion prediction. The multimodal large model includes an event encoding module 10, an image encoding module 20, a text instruction encoding module 30 and a feature fusion encoding module 40.
[0061] The event encoding module 10 , the image encoding module 20 , and the text instruction encoding module 30 are all connected to the feature fusion encoding module 40 .
[0062] The event encoding module 10 is used to encode the first event data into a first event frame feature vector of a preset length N, namely E1, E2, . . . EN, and transmit the first event frame feature vector to the feature fusion encoding module 40.
[0063] The image encoding module 20 is used to encode the first image data into a first image feature vector of a preset length N, which is I1, I2, . . . IN, and transmit the first image feature vector to the feature fusion encoding module 40.
[0064] The text instruction encoding module 30 is used to encode the first text instruction into a text feature vector of a preset length N, namely T1, T2, ... TN, and transmit the text feature vector to the feature fusion encoding module 40.
[0065] The event encoding module 10, the image encoding module 20, and the text instruction encoding module 30 are all trained modules.
[0066] The feature fusion encoding module 40 is used to receive the first event frame feature vector, the first image feature vector and the text feature vector, and the feature fusion encoding module 40 includes a plurality of repeated cross attention structures Cross Attention and self attention structures Self Attention, so that the first event frame feature vector, the first image feature vector and the text feature vector are effectively fused together in the repeated cross attention structures Cross Attention and self attention structures Self Attention. In addition, the first difference between the fused feature vector and the robot's action sample during each iterative training is calculated, and when the first difference is less than the first threshold, the multimodal large model converges.
[0067] This application also provides a multi-modal large model training method for robot motion prediction, such as Figure 2 As shown, the training method of the multimodal large model includes the following steps S101 to S104.
[0068] S101: Acquire first event data of a first event camera, first image data of a first RGB camera, and a first text instruction.
[0069] Usually, the robot is equipped with an event camera and an RGB camera. RGB represents three color channels, namely the colors of the red, green, and blue channels. The RGB camera continuously captures images at a fixed frame rate with a minimum time interval. However, in high-speed motion scenes, the frame rate will cause motion blur and cannot accurately capture the changing details. It is easily troubled by high contrast. In scenes with strong changes in light intensity and high contrast, it may face overexposure (loss of brightness details) or underexposure (loss of dark details). The first event camera has the characteristics of high dynamic range and low latency, and can respond well to high-speed motion, high and low light. Therefore, the characteristics of event data and image data are well complementary.
[0070] When training the multimodal large model, in order to enable the multimodal large model to support multimodal learning, the first event data of the first event camera, the first image data of the first RGB camera, and the first text instruction can be obtained. The first event data and the first image data are used as visual inputs of the multimodal large model. Combined with the first text instruction, through cross-modal interactive learning and model training, the multimodal large model's ability to understand and express real-life scenes is enhanced. The obtained multimodal large model can enable the robot to perform better in complex environments.
[0071] It is understandable that the first event camera and the first RGB camera can be cameras of other devices, as long as they can simultaneously capture the event camera and RGB camera in the current scene, and are not limited to the event camera and RGB camera of the robot.
[0072] For the current multimodal fusion of text and images, the language-image data (Language-Image) model of the CLIP (Contrastive Language-Image Pre-Training) model is used. CLIP adopts the contrastive learning framework. Its core idea is to train on a large number of image-text pairs so that the model learns how to map images and texts that match their descriptions tightly into a shared multimodal representation space, while keeping mismatched image-text pairs far apart in the space. Therefore, it is easy to obtain a large number of image-text pair datasets on the Internet, but event camera data-text pair data is difficult to obtain, and different event camera models and data processing methods are difficult to unify. That is, how to map event camera data to a shared multimodal representation space that is unified with image and text is a difficult problem.
[0073] Based on the above problems, this application proposes to train the event encoding module to obtain the event encoding module, which is the CEIP (Contrastive Event-Image Pre-Training) model. By training the event encoding module, the model of language-image pair data Language-Image is conveniently generalized into a model with input as event-image pair data (Event-Image).
[0074] Therefore, in some of the embodiments, before executing step S101, the multimodal large model training method for robot motion prediction further includes:
[0075] S100: Obtain the event encoding module.
[0076] Specifically, when training the multimodal large model, the event encoding module of the multimodal large model is trained first, that is, the CEIP model is trained. The event encoding module can extract the first event data of the first event camera into a first event frame feature vector, and the first event frame feature vector matches the first image feature vector of the first RGB camera, and is better spliced with the first image feature vector.
[0077] In some embodiments, such as Figure 3 As shown, Figure 3 is a schematic diagram of a detailed process of step S100. Step S100 may include:
[0078] S11: compressing the acquired second event data into an event frame format.
[0079] Event cameras simulate the working mode of biological retinas, especially the response of retinal ganglion cells to changes in light intensity, which can be changes in brightness. Compared with RGB cameras that capture complete image frames of the entire scene at fixed time intervals, event cameras detect brightness changes at each pixel position in real time and immediately generate an "event" when a significant change occurs. These events are not output in the form of continuous image frames, but in the form of asynchronous, discrete second event data. Therefore, each second event data includes the following information: address (x, y), polarity p and timestamp t, where the address identifies the pixel position where the event occurs; the polarity indicates the direction of brightness change; and the timestamp records the time node of brightness change.
[0080] It can be understood that the address (x, y) is the two-dimensional coordinate where the brightness change occurs in the image. Polarity includes "ON" events and "OFF" events. "ON" events indicate that the pixel brightness increases, and "OFF" events indicate that the pixel brightness decreases. The timestamp t records the precise event of the event, which can usually achieve microsecond accuracy, reflecting the instantaneity of the brightness change.
[0081] The format of the original second event data is (x, y, p, t), where x and y represent the horizontal and vertical coordinates of the pixel respectively, and generally require a bit width of 10 to 16 bits depending on the resolution; p represents polarity, which can be represented by 1 bit; t is a timestamp with microsecond accuracy, which requires a bit width of 32 to 64 bits. Therefore, the bit width required for a second event data is as high as (10+10+1+32) = 53 bits or more. Obviously, the bit width required for the timestamp t accounts for more than 60%.
[0082] Therefore, when the robot moves quickly or there are large changes in brightness in the environment where the robot is located, the number of events will be very large, for example, exceeding 100MEvent / s, which will place great demands on the robot's bandwidth. In order to alleviate this problem, this application compresses the second event data of the event camera.
[0083] It can be understood that, similar to the first event camera, the event camera can be an event camera on other devices and is not limited to the event camera of a robot.
[0084] In order to compress the second event data, in some implementations, step S11 may include:
[0085] Acquire a plurality of second event data, wherein the second event data includes an address (x, y), a polarity p, and a timestamp t, wherein the address identifies a pixel position where an event occurs; the polarity indicates a direction of brightness change; and the timestamp records a time node of brightness change;
[0086] Determine each of the second event data with a same timestamp;
[0087] A format converter is used to convert the format of the second event data with the same timestamp into the format of an event frame.
[0088] Specifically, after acquiring several second event data and determining each second event data with the same timestamp t, the format of the second event data with the same timestamp is converted to realize compression processing of the second event data, thereby reducing the transmission bandwidth of the robot occupied by multiple asynchronous and discrete second event data, and improving data transmission efficiency.
[0089] Furthermore, since features need to be extracted from subsequent event frames and correspond to the length of the image feature vector extracted by the image encoding module, converting the format of the second event data with the same timestamp can not only compress the bandwidth occupied by the second event data, but also convert the format of the second event data into a format that matches the image feature vector.
[0090] For example, the format of N second event data (x, y, p, t) is converted into the format of event frame by the format converter: (t, (x1, y1, p1), (x2, y2, p2), ..., (xN, yN, pN)), that is, all N events share the same timestamp t. This can significantly reduce bandwidth requirements.
[0091] In some implementations, determining each of the second event data with the same timestamp may include:
[0092] Sort each of the second event data according to timestamps;
[0093] The sorted second event data are put into a first-in-first-out queue, and the number of second event data at different timestamps and the number of second event data at the same timestamp are counted through a counter and a timer.
[0094] Specifically, Figure 4 As shown, a first-in-first-out queue FIFO buffer, a counter and a timer can be set to sort N second event data (x, y, p, t) according to the timestamp t, and the sorted second event data are sequentially put into the first-in-first-out queue, and the sorted second event data are counted in the counter according to the first-in-first-out queue. At the same time, the timer determines whether the current event has timed out at the current timestamp. If it has timed out, the timer sends a reset signal to the counter to let the counter count again, and the number of times the counter is reset is the number of different timestamps. The count of the second event data in each timestamp by the counter is the number of second event data at the same timestamp, the number of second event data at different timestamps, and the number of second event data at the same timestamp.
[0095] For example, within a time period, if the RGB camera outputs at 50FPS, the timer reset interval will be set to 20ms. Then, N=5000 events are generated cumulatively within a timer reset interval. The original events are encoded using 53 bits, and the amount of data to be transmitted will drop from the original 265000 to 105032.
[0096] Since the input of the subsequent training event encoding module is in event frame format, N events will be used to generate the same event frame, so N events sharing the same timestamp will not affect the generation of subsequent event frames.
[0097] By compressing and formatting the second event data of each timestamp through a counter, a timer and a format converter, the bandwidth requirement of the robot can be effectively reduced, and the format of the obtained event frame can match the format of the image data of the RGB camera.
[0098] S12: using a contrast learning mechanism, extracting and encoding the acquired second image data to obtain a second image feature vector, and encoding the event frame to obtain a second event feature vector.
[0099] Before training the event coding module, an event coding module is first created, and the event coding module is an event coding module that has not yet been trained.
[0100] like Figure 5 As shown, Figure 5 is a schematic block diagram of the event encoding module 10 of the present application. Figure 5 In the figure, the event encoding module 11 is an untrained event encoding module 10. After the training is completed, it is the event encoding module 10 of the multimodal large model. The image encoding module 20 can be the trained image encoding module 20 in the multimodal large model. The second image data obtained by the RGB camera is encoded by the image encoding module 20 to generate N second image feature vectors. The length of each image feature vector is d. The second image feature vector is represented by I_f. Then, the second image feature vector I_f can be stored using a matrix of size [N, d]. Correspondingly, the event frame also generates N second event feature vectors after the event encoding module 11. The length of the second event feature vector is also d. The second event feature is represented by E_f and stored using a matrix of size [N, d].
[0101] S13: Using the second image feature vector and the second event feature vector to train and obtain the event encoding module.
[0102] In some implementations, step S13 may include steps S131 to S134.
[0103] S131: Normalize the second image feature vector and the second event feature vector respectively to obtain a normalized image vector and a normalized event vector.
[0104] The second image feature I_f and the second event feature E_f can be processed using the L2 normalization method respectively. For each vector X = [x1, x2, ..., xd], the L2 normalized vector y is calculated as follows:
[0105] Formula 1;
[0106] Wherein, y represents a normalized image vector or a normalized event vector, and x represents a vector of second image features or a vector of second event features.
[0107] S132: Perform matrix multiplication on the normalized image vector and the normalized event vector to obtain a cosine similarity matrix.
[0108] like Figure 5 As shown, the normalized image vectors are I1, I2, I3, ..., IN, and the normalized event vectors are E1, E2, E3, ..., EN. In the cosine similarity matrix S, the matrix size is [N, N]. For each value in the cosine similarity matrix S, the higher the value, the more similar the second image feature and the second event feature are.
[0109] S133: Calculate a first softmax result of the normalized image vector relative to the normalized event vector in the horizontal direction of the cosine similarity matrix, and calculate a second softmax result of the normalized event vector relative to the normalized image vector in the vertical direction of the cosine similarity matrix, and obtain a probability similarity matrix based on the first softmax result and the second softmax result.
[0110] The first softmax result and the second softmax result can be calculated using Formula 2:
[0111] Formula 2;
[0112] Among them, xi represents the normalized image vector or the normalized event vector respectively.
[0113] There are N vectors in the horizontal and vertical directions of the cosine similarity matrix S, respectively. The first softmax result represents the sum of the probabilities of the normalized image vector in the horizontal direction of the cosine similarity matrix S relative to the normalized event vector, and the second softmax result represents the sum of the probabilities of the normalized event vector in the vertical direction of the cosine similarity matrix S relative to the normalized image vector. Softmax represents a normalization function.
[0114] S134: Calculate the horizontal loss value and the vertical loss value in the probability similarity matrix to obtain a second loss function value, until a second difference between the second loss function value and the image sample is less than a second threshold, and obtain the event encoding module 10.
[0115] Specifically, the matrix size of the probability similarity matrix is still [N, N], representing the probability of the second image feature relative to the second event feature and the probability of the second event feature relative to the second image feature.
[0116] In some implementations, step S134 may include:
[0117] Obtaining a first softmax result and a second softmax result located on a main diagonal of the probability similarity matrix;
[0118] Substituting the first softmax result and the second softmax result located on the main diagonal into the predicted probability q in the cross entropy loss function formula respectively, and setting the true probability p in the cross entropy loss function formula to 1, to obtain a horizontal loss value and a vertical loss value;
[0119] The formula of the cross entropy loss function is:
[0120] Formula 3;
[0121] in, represents the cross entropy loss function, p represents the true probability that the first softmax result and the second softmax result are equal, q represents the predicted probability that the first softmax result and the second softmax result are equal; N represents the number of the first softmax result or the second softmax result;
[0122] The lateral loss value and the longitudinal loss value are added to obtain the second loss function value.
[0123] Specifically, since the values of the main diagonal are larger than those of other positions in the probability similarity matrix, in order to allow the model to learn this optimization goal, a cross entropy loss function can be introduced. The cross entropy loss function is shown in Formula 3. Figure 5 , the data on the main diagonal of the cosine similarity matrix S is I1·E1, I2·E2, I3·E3, ..., IN·EN. Then, the location of the data on the main diagonal of the probability similarity matrix can be referred to Figure 5 The data on the main diagonal of is located.
[0124] Since the input of each multimodal large model is image-event pair data, the true probability is that the main diagonal of the probability similarity matrix is 1, and the other positions are 0. That is, substitute the first softmax result S_i on the main diagonal of the probability similarity matrix into the predicted probability q in formula 3, and set the true probability p to 1 to calculate the horizontal loss value L_i; substitute the second softmax result S_e on the main diagonal of the probability similarity matrix into the longitudinal loss value L_e. Finally, add the horizontal loss value L_i and the longitudinal loss value L_e to obtain the total loss function L_total, which is the second loss function value.
[0125] The image sample is a real sample label for training the event encoding module. When the second difference between the second loss function value and the real sample label is less than the second threshold, it is an acceptable error. At this time, the CEIP model converges, thereby obtaining the event encoding module 10.
[0126] In some implementations, obtaining the second loss function value until a second difference between the second loss function value and the image sample is less than a second threshold, and obtaining the event encoding module may include:
[0127] Based on the second loss function value, the weight for encoding the event frame is adjusted, and the weight for encoding the second image data is frozen for iterative training until the second difference is less than the second threshold, thereby obtaining the event encoding module 10.
[0128] Specifically, the trained image encoding module 20 can be derived from a trained CLIP model. The trained CLIP model of the present application can use ViT-L / 14, ViT (Vision Transformer) feature extraction network, divide the 224x224 resolution image into 14x14 pixel blocks, and then use the self-attention mechanism for processing. It is a relatively mature image encoding module and will not be described in detail here.
[0129] Iterative training usually includes the following steps: data preprocessing, forward propagation, loss calculation, back propagation, weight update, performance evaluation, iteration termination and model deployment. Usually, during iterative training, weights are involved in the steps of forward propagation, loss calculation, back propagation, weight update and iterative optimization.
[0130] Since the image coding module of the current CLIP model is usually trained more completely and with high accuracy, it is possible to obtain a trained image coding module and freeze the weight of the trained image coding module 20. When the trained event coding module 10 is obtained, the weight of the event frame coding is adjusted based on the second loss function value, that is, the weight of the event coding module 11 during training is adjusted, and the weight of the second image data coding is frozen, that is, the weight of the image coding module 20 during training is kept unchanged, thereby performing back propagation, weight updating, and iterative optimization steps of iterative training, which can reduce the amount of data during training of the event coding module 20.
[0131] The training of the event encoding module 20 of the present application, namely the training of the CEIP model, can conveniently generalize the Language-Image model in the current CLIP model into an Event-Image model of event-image data, so that the first event data of the first event camera can be converted into input that can be understood by the large model.
[0132] S102: The first event data obtained is encoded by the event encoding module to obtain a first event frame feature vector of a preset length; the first image data obtained is encoded by the image encoding module to obtain a first image feature vector of the preset length; the first text instruction obtained is encoded by the text instruction encoding module to obtain a text feature vector of the preset length.
[0133] The preset length may be N.
[0134] Since the event coding module has been trained, when the first event data is input into the event coding module, the event coding module encodes the first event data into a first event frame feature vector of length N, which is E1, E2, ...EN, which can match the length N of the first image feature vector (I1, I2, ...IN) and text feature vector (T1, T2, ...TN) extracted by the current CLIP model.
[0135] S103: Concatenate the first event frame feature vector and the first image feature vector to obtain a visual feature vector.
[0136] After obtaining the first event frame feature vector and the first image feature vector, the concat function can be used to concatenate the first event frame feature vector and the first image feature vector to obtain a visual feature vector V, and the visual feature vector will not change the first event frame feature vector and the first image feature vector.
[0137] S104: In the cross-attention structure and self-attention structure of the feature fusion encoding module, the visual feature vector and the text feature vector are subjected to feature fusion and iterative training, and when the difference between the fused feature vector and the robot's action sample is less than a first threshold, a multimodal large model is obtained.
[0138] The feature fusion decoding module is a structure used in artificial intelligence models to improve information processing and utilization efficiency. The feature fusion decoding module effectively fuses the first event frame feature vector, the first image feature vector and the text feature vector, so that the multimodal large model can better understand and analyze the input first event data, the first image data and the first text instruction.
[0139] In the present application, the feature fusion coding module may include 32 cross-attention structures CrossAttention and self-attention structures Self Attention of the same structure. The feature fusion coding module enhances the feature representation of the first event frame feature vector, the first image feature vector and the text feature vector through a series of repeated attention mechanisms Self Attention. It can be understood that 32 is the number of repetitions of the cross-attention structure Cross Attention and the self-attention structure Self Attention in the multimodal large model.
[0140] Cross Attention and Self Attention are two forms of attention mechanisms, and they have their own advantages when processing sequence data or feature maps. Cross Attention allows the model to establish associations between different visual feature vectors and text feature vectors; Self Attention focuses on the contextual information within a single visual feature vector, as well as the contextual information within a single text feature vector.
[0141] By combining the cross attention structure Cross Attention and the self-attention structure Self Attention, we can make full use of the feature information of the first event data, the first image data and the first text instruction from different sources, and improve the multimodal large model's ability to parse complex data.
[0142] Iterative training refers to the process of continuously iterating the multimodal large model during the training process to optimize the parameters of the model and improve the performance of the model in multimodal data processing. The number of iterations can be determined according to the amount of data of the first event data, the first image data and the first text instruction. For example, when the iteration reaches more than one million times, the difference between the fused feature vector and the robot's action sample is less than the first threshold, so that the model converges and the multimodal large model is obtained.
[0143] Among them, the robot's action sample is the robot's action sample label input when training the model, which is used to predict the robot's action.
[0144] Therefore, the trained multimodal large model can be used for robot motion prediction.
[0145] When a large multimodal model is iteratively trained, weights are involved in the steps of forward propagation, loss calculation, back propagation, weight update, and iterative optimization. If the weights are updated in each step involving weights, the amount of data will be large.
[0146] Based on the above problem, in some implementations, the feature fusion and iterative training of the visual feature vector and the text feature vector until a first difference between the fused feature vector and the robot's action sample is less than a first threshold value to obtain a multimodal large model may include:
[0147] Calculating a first difference between the fused feature vector and the action sample of the robot using a first loss function to obtain a first loss function value;
[0148] Based on the first loss function value, the weights of the event encoding module and the cross-attention structure are adjusted, and the weights of the image encoding module, the text instruction encoding module and the self-attention structure are frozen for iterative training until the first difference is less than the first threshold, so as to obtain the multimodal large model.
[0149] During the training process, the weights of each encoding module and feature fusion encoding module are updated through optimization methods such as backpropagation algorithm and gradient descent. These weights represent the model's understanding and interpretation of the input data. Therefore, they are fixed after the training is completed and used when the model makes predictions or processes new data.
[0150] In the present application, since the self-attention structures in the image coding module, the text instruction coding module and the feature fusion coding module are usually trained relatively well in the previous multimodal large model and do not need to update the weights again, during the training process of the multimodal large model of the present application, during the iterative training, based on the first loss function value, the weights of the self-attention structures in the image coding module, the text instruction coding module and the feature fusion coding module are frozen, and only the weights of the event coding module and the cross-attention structure are adjusted. This can reduce the amount of calculation in the training process, reduce memory usage and computing resource requirements, and allow the training process to proceed faster. In addition, it can prevent the model from over-relying on the self-attention structures in the image coding module, the text instruction coding module and the feature fusion coding module on the features of the training data, and reduce the risk of overfitting.
[0151] Freezing the weights of the self-attention structures in the image encoding module, text instruction encoding module, and feature fusion encoding module can simplify the structure of the model and make it easier to explain and understand.
[0152] In an embodiment of the present application, when training a multimodal large model, an event encoding module enables the first event data to be converted into an input that can be understood by the multimodal large model, and encodes and processes the first event data of the first event camera, the first image data of the first RGB camera, and the first text instruction respectively, and extracts the first event frame feature vector, the first image feature vector, and the text feature vector of the same preset length, so that the first event data, the first image data, and the first text instruction can be mapped to the same high-dimensional space, providing a basis for subsequent multimodal fusion, supporting multimodal input, making the robot work more accurately in a complex environment, achieving satisfactory work, and reducing data requirements and costs.
[0153] Embodiment 2:
[0154] like Figure 6 As shown, the present application also provides a robot motion prediction method, the robot includes a second event camera and a second RGB camera, and the robot motion prediction method includes steps S201 to S202.
[0155] S201: inputting the third event data acquired by the second event camera at the current moment, the third image data acquired by the second RGB camera at the current moment, and the second text instruction into the trained multimodal large model to obtain the currently predicted action information of the robot; the multimodal large model is trained by the multimodal large model training method for robot action prediction described in any one of Embodiment 1;
[0156] S202: Input the currently predicted action information into a long short-term memory network, and the long short-term memory network outputs the final action of the robot at the current moment based on the currently predicted action information and the historical state information of the robot.
[0157] Specifically, after the training of the multimodal large model is completed, the trained multimodal large model can be used to predict the robot's movements.
[0158] The multimodal large model predicts the robot's current action based on the third event data acquired by the robot's second event camera at the current moment, the third image data acquired by the second RGB camera at the current moment, and the second text instruction, and inputs the robot's currently predicted action information into the long short-term memory network LSTM to integrate the robot's currently predicted action information and the robot's historical state information, and outputs the robot's final action at the current moment.
[0159] In the embodiments of the present application, since the second event camera has the characteristics of high dynamic range and low latency, it can respond well to the high-speed movement of the robot and the high and low light environments in which it is located. The trained multimodal large model supports multimodal learning, that is, it simultaneously processes the second text instructions, the third event data of the second event camera, and the third image data of the second RGB camera. Through cross-modal interactive learning, it effectively combines the advantages of the second event camera and the multimodal large model, so that the robot has stronger perception capabilities in complex scenes (high-speed movement, poor lighting conditions), and at the same time utilizes the powerful background knowledge of the multimodal large model to enable the robot to have more flexible and reasonable decision-making capabilities.
[0160] Those skilled in the art can understand that all or part of the features / steps of the above-mentioned method embodiments can be implemented by methods, data processing systems or computer programs, and these features can be implemented without hardware, entirely by software, or by a combination of hardware and software. The aforementioned computer program can be stored in one or more computer-readable storage media, and the storage medium stores a computer program. When the computer program is executed (such as a processor), the steps of the above-mentioned multimodal large model training method for robot motion prediction and robot motion prediction method embodiments are executed.
[0161] The aforementioned storage media that can store program codes include: static hard disks, solid-state hard disks, random access memories (SRAM), electrically erasable programmable read-only memories (EEPROM), erasable programmable read-only memories (EPROM), programmable read-only memories (PROM), read-only memories (ROM), optical storage devices, magnetic storage devices, flash memories, magnetic disks or optical disks and / or combinations of the above devices, that is, they can be implemented by any type of volatile or non-volatile storage devices or combinations thereof.
[0162] Embodiment three:
[0163] The present application also provides a processing device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors are used to execute the one or more computer programs stored in the memory, so that the one or more processors execute a multimodal large model training method for robot motion prediction as described in any one of Example 1 and the robot motion prediction method of Example 2.
[0164] The processing device of the present invention may be a chip, and the processing device provided by the present invention may be, for example, a large model processing device. When the processing device is a chip, each module in the chip may be implemented in whole or in part by software, hardware, and a combination thereof. The above modules may be embedded in or independent of the processor in the computing device in hardware form, or may be stored in the memory in the computing device in software form, so that the processor can call and execute the operations corresponding to the above modules. The invention can be applied to fields such as artificial intelligence and metaverse that require high precision, high reliability, low power consumption, and low area cost computing power, artificial intelligence training or reasoning chip products, autonomous driving chips, VR chips, robot built-in chips, and deep learning applications of parallel computing. It can be applied to fields such as artificial intelligence and metaverse that require high precision, high reliability, low power consumption, and low area cost computing power, such as artificial intelligence training or reasoning chips, autonomous driving chips, VR chips, robot built-in chips, and other fields.
[0165] The present application also provides an electronic device, the electronic device including a processing device. The electronic device can be a series of electronic devices such as a humanoid robot, a smart phone, a tablet computer, a wearable electronic device, a smart home electronic product, and an industrial device.
[0166] Since the implementation of the electronic device is described in detail in the first embodiment, it will not be repeated here.
[0167] The above description is only the preferred embodiment of the present invention. It is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application belong to the protection scope of the present invention.
Claims
1. A multimodal large model training method for robot motion prediction, characterized in that: The multimodal large model includes an event coding module, an image coding module, a text instruction coding module and a feature fusion coding module, and the event coding module, the image coding module and the text instruction coding module are all connected to the feature fusion coding module; the multimodal large model training method for robot motion prediction includes: Acquire first event data of a first event camera, first image data of a first RGB camera, and a first text instruction; The first event data obtained is encoded by the event encoding module to obtain a first event frame feature vector of a preset length; the first image data obtained is encoded by the image encoding module to obtain a first image feature vector of the preset length; the first text instruction obtained is encoded by the text instruction encoding module to obtain a text feature vector of the preset length; splicing the first event frame feature vector and the first image feature vector to obtain a visual feature vector; In the cross-attention structure and the self-attention structure of the feature fusion encoding module, the visual feature vector and the text feature vector are subjected to feature fusion and iterative training until a first difference between the fused feature vector and the action sample of the robot is less than a first threshold, so as to obtain a multimodal large model; The multimodal large model training method for robot motion prediction also includes obtaining the event encoding module, and the obtaining event encoding module includes: compressing the acquired second event data into a format of an event frame; Using a contrast learning mechanism, encoding the acquired second image data to obtain a second image feature vector, encoding the event frame to obtain a second event feature vector; The event encoding module is obtained by training using the second image feature vector and the second event feature vector; The step of obtaining the event encoding module by training the second image feature vector and the second event feature vector includes: Normalizing the second image feature vector and the second event feature vector respectively to obtain a normalized image vector and a normalized event vector; Performing matrix multiplication on the normalized image vector and the normalized event vector to obtain a cosine similarity matrix; In the horizontal direction of the cosine similarity matrix, a first softmax result of the normalized image vector relative to the normalized event vector is calculated, and in the vertical direction of the cosine similarity matrix, a second softmax result of the normalized event vector relative to the normalized image vector is calculated, and a probability similarity matrix is obtained based on the first softmax result and the second softmax result; The horizontal loss value and the vertical loss value are calculated in the probability similarity matrix to obtain a second loss function value, until a second difference between the second loss function value and the image sample is less than a second threshold, thereby obtaining the event encoding module.
2. The multimodal large model training method for robot motion prediction according to claim 1, characterized in that: The visual feature vector and the text feature vector are subjected to feature fusion and iterative training until a first difference between the fused feature vector and the robot's action sample is less than a first threshold value to obtain a multimodal large model, including: Calculating a first difference between the fused feature vector and the action sample of the robot using a first loss function to obtain a first loss function value; Based on the first loss function value, the weights of the event encoding module and the cross-attention structure are adjusted, and the weights of the image encoding module, the text instruction encoding module and the self-attention structure are frozen for iterative training until the first difference is less than the first threshold to obtain the multimodal large model.
3. The multimodal large model training method for robot motion prediction according to claim 1, characterized in that: The step of compressing the acquired second event data into a format of an event frame includes: Acquire a plurality of second event data, wherein the second event data includes an address (x, y), a polarity p, and a timestamp t, wherein the address identifies a pixel position where an event occurs; the polarity indicates a direction of brightness change; and the timestamp records a time node of brightness change; Determine each of the second event data with a same timestamp; The format of the second event data with the same timestamp is converted into the format of the event frame.
4. The multimodal large model training method for robot motion prediction according to claim 1, characterized in that: The calculating the lateral loss value and the longitudinal loss value in the probability similarity matrix to obtain a second loss function value includes: Obtaining a first softmax result and a second softmax result located on a main diagonal of the probability similarity matrix; Substituting the first softmax result and the second softmax result located on the main diagonal into the predicted probability q in the cross entropy loss function formula respectively, and setting the true probability p in the cross entropy loss function formula to 1, to obtain a horizontal loss value and a vertical loss value; The formula of the cross entropy loss function is: ; in, represents the cross entropy loss function, p represents the true probability that the first softmax result and the second softmax result are equal, q represents the predicted probability that the first softmax result and the second softmax result are equal; N represents the number of the first softmax result or the second softmax result; The lateral loss value and the longitudinal loss value are added to obtain the second loss function value.
5. The multimodal large model training method for robot motion prediction according to claim 1, characterized in that: The step of obtaining the second loss function value until a second difference between the second loss function value and the image sample is less than a second threshold value, and obtaining the event encoding module, comprises: Based on the second loss function value, the weight for encoding the event frame is adjusted, and the weight for encoding the second image data is frozen for iterative training until the second difference is less than the second threshold, thereby obtaining the event encoding module.
6. A robot motion prediction method, characterized in that: The robot includes a second event camera and a second RGB camera, and the robot motion prediction method includes: Input the third event data acquired by the second event camera at the current moment, the third image data acquired by the second RGB camera at the current moment, and the second text instruction into the trained multimodal large model to obtain the currently predicted action information of the robot; the multimodal large model is obtained by training the multimodal large model training method for robot action prediction according to any one of claims 1 to 5; The currently predicted action information is input into a long short-term memory network, and the long short-term memory network outputs the final action of the robot at the current moment based on the currently predicted action information and the historical state information of the robot.
7. A processing device, characterized in that: include: one or more processors; A memory for storing one or more computer programs, and one or more processors for executing one or more computer programs stored in the memory, so that one or more processors execute the multimodal large model training method for robot motion prediction as described in any one of claims 1 to 5 and the robot motion prediction method as described in claim 6.
8. An electronic device, characterized in that: The electronic device comprises the processing device according to claim 7.
Citation Information
Patent Citations
Map-free robot path navigation method and system based on DDPG and LSTM
CN112629542A
Multi-modal event detection method based on complementary content awareness
CN114154587A
Extraction method of training sample image and training method of sample image extraction model
CN116977761A
Visual entity linking method based on multi-modal pre-training model
CN117689963A
Robot planning method and system based on multi-mode large model predictive control
CN118061186A
