Data processing method and device based on multi-modal large model and electronic equipment

Abstract

The invention provides a data processing method and device based on a multi-modal large model and electronic equipment, and relates to the technical field of data processing. The method is applied to a service robot with a built-in multi-mode large model, and the multi-mode large model comprises a multi-mode encoder, a language model and a multi-mode decoder. The method specifically comprises the step of triggering a visual data acquisition operation based on a voice instruction signal and / or a text instruction signal sent by a target user. Then coding the voice command signal and / or the text command signal and the collected visual data based on a multi-mode encoder to obtain a high-dimensional feature vector; and further processing the high-dimensional feature vector based on a user-defined attention mechanism in a language model to obtain a target feature sequence. And finally, performing autoregression decoding processing on the target feature sequence based on a multi-mode decoder to obtain an action instruction signal. According to the method, the data processing efficiency of the multi-modal large model can be improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data processing method, apparatus and electronic device based on a multimodal large model. Background Technology

[0002] Multimodal large models are deep learning models that are jointly trained by integrating multiple types of data, such as text, images, videos, and audio. With their powerful comprehensive understanding and generative capabilities, they are widely used in various technical fields.

[0003] Existing multimodal large-scale models are centered around a language model. They use encoders of different modalities to encode input data from various modalities, such as text, images, video, and speech, into high-dimensional feature vectors. These high-dimensional feature vectors are then merged and fed into the language model for processing, resulting in a feature sequence. At the output end, the feature sequences generated by the language model are fed into decoders of different modalities for decoding, ultimately yielding the outputs for each modality.

[0004] However, existing technologies suffer from low data processing efficiency. Summary of the Invention

[0005] This application provides a data processing method, apparatus, and electronic device based on a multimodal large model to solve the technical problem of low data processing efficiency in the prior art.

[0006] In a first aspect, this application provides a data processing method based on a multimodal large model, the method being applied to a service robot, the service robot having a built-in multimodal large model, the multimodal large model including a multimodal encoder, a language model, and a multimodal decoder, the method comprising:

[0007] Based on the voice command signal and / or text command signal sent by the target user, a visual data acquisition operation is triggered to acquire visual data; wherein, the visual data represents image data or video data used in response to the voice command signal and / or text command signal;

[0008] Based on the multimodal encoder, the visual data, the voice command signal, and / or the text command signal are encoded to obtain a high-dimensional feature vector; based on the preset custom attention mechanism in the language model, the high-dimensional feature vector is processed to obtain the target feature sequence;

[0009] Based on the multimodal decoder, the target feature sequence is subjected to autoregressive decoding processing to obtain an action command signal; the physical action indicated by the action command signal is executed.

[0010] Optionally, the preset custom attention mechanism includes a cross-attention mechanism and a linear attention mechanism;

[0011] Based on the pre-defined custom attention mechanism in the language model, the high-dimensional feature vector is processed to obtain the target feature sequence, including:

[0012] Based on the cross-attention mechanism, the high-dimensional feature vector is aligned to map it to the word embedding space of the language model, thereby obtaining a multimodal sequence.

[0013] Based on the linear attention mechanism, attention calculation is performed on the multimodal sequences in the word embedding space to obtain the target feature sequence.

[0014] Optionally, based on the linear attention mechanism, attention calculation is performed on the multimodal sequences in the word embedding space to obtain the target feature sequence, including:

[0015] Based on the high-dimensional feature vectors of the multimodal sequences in the word embedding space, the query matrix, key matrix, and value matrix are calculated;

[0016] Based on the key matrix and the value matrix, the target state matrix is ​​calculated; matrix multiplication is performed on the target state matrix and the query matrix to obtain the target feature sequence.

[0017] Optionally, based on the key matrix and the value matrix, a target state matrix is ​​calculated; the target state matrix and the query matrix are then multiplied to obtain a target feature sequence, including:

[0018] Based on the preset proxy matrix, the key matrix is ​​processed to obtain the processed key matrix, and based on the preset proxy matrix, the query matrix is ​​processed to obtain the processed query matrix;

[0019] Based on the processed key matrix and the value matrix, the target state matrix is ​​calculated; matrix multiplication is performed on the target state matrix and the processed query matrix to obtain the target feature sequence.

[0020] Optionally, the multimodal decoder includes an audio decoder, a text decoder, and a visual decoder;

[0021] Based on the multimodal decoder, the target feature sequence is subjected to autoregressive decoding processing to obtain an action command signal, including:

[0022] Based on the audio decoder, autoregressive decoding processing is performed on the audio features in the target feature sequence to generate an audio response, and / or, based on the text decoder, autoregressive decoding processing is performed on the text features in the target feature sequence to generate a text response; wherein, the audio response represents the audio information used to complete the voice command signal and / or text command signal, and the text response represents the text information used to complete the voice command signal and / or text command signal.

[0023] Based on the visual decoder, the visual features in the target feature sequence are subjected to autoregressive decoding processing to generate a target visual image; wherein, the target visual image represents the visual information used to complete the voice command signal and / or text command signal;

[0024] The target visual image, the text response, and / or the audio response are integrated to obtain an action command signal; wherein the action command signal represents the physical action used to complete the voice command signal and / or text command signal.

[0025] Optionally, based on the visual decoder, autoregressive decoding processing is performed on the visual features in the target feature sequence to generate a target visual image, including:

[0026] Based on the visual features in the target feature sequence, and using the autoregressive visual generation algorithm in the visual decoder, a visual image at a preset resolution is generated.

[0027] Based on the visual image at the preset resolution, and using the autoregressive visual generation algorithm, a visual image at the next resolution adjacent to the preset resolution is generated; wherein, the next resolution is greater than the preset resolution.

[0028] If the current resolution is the same as the resolution corresponding to the visual data, then the visual image at the current resolution is determined as the target visual image.

[0029] Optionally, the multimodal encoder includes an audio encoder, a text encoder, and a visual encoder;

[0030] Based on the multimodal encoder, the visual data, the speech command signal, and / or the text command signal are encoded to obtain a high-dimensional feature vector, including:

[0031] Based on the audio encoder, the voice command signal is encoded to obtain an audio feature vector, and / or, based on the text encoder, the text command signal is encoded to obtain a text feature vector;

[0032] Based on the visual encoder, the visual data is encoded to obtain a visual feature vector;

[0033] The visual feature vector, the text feature vector, and / or the audio feature vector are determined to be the high-dimensional feature vector.

[0034] Optionally, the visual encoder includes a understanding sub-encoder and a generating sub-encoder;

[0035] Based on the visual encoder, the visual data is encoded to obtain a visual feature vector, including:

[0036] Based on the understanding sub-encoder, the visual data is processed to obtain visual features;

[0037] Based on the generator sub-encoder, the visual features are processed to obtain a visual feature vector.

[0038] Optionally, the visual encoder is a trained composite encoder, and the training process of the visual encoder includes:

[0039] Input the pre-collected historical visual data samples into the pre-created initial visual encoder;

[0040] Based on the initial visual encoder, the historical visual data samples are divided into blocks to obtain multiple data sample blocks;

[0041] The data sample block is preprocessed, and the preprocessed data sample block is input into the first data processing model, the second data processing model and the third data processing model included in the initial visual encoder for processing;

[0042] Determine the first data feature output by the first data processing model, the second data feature output by the second data processing model, and the third data feature output by the third data processing model;

[0043] The first data feature, the second data feature, and the third data feature are concatenated to obtain a data feature sequence.

[0044] Based on a preset segmentation rule, the data feature sequence is segmented to obtain a first data feature subsequence and a second data feature subsequence; the first data feature subsequence is determined as the understanding sub-encoder, and the second data feature subsequence is determined as the generating sub-encoder; based on the understanding sub-encoder and the generating sub-encoder, the visual encoder is generated.

[0045] Secondly, this application provides a data processing device based on a multimodal large model, the device being applied to a service robot, the service robot having a built-in multimodal large model, the multimodal large model including a multimodal encoder, a language model, and a multimodal decoder, the device comprising:

[0046] The processing module is used to trigger a visual data acquisition operation based on the voice command signal and / or text command signal sent by the target user, and acquire visual data; wherein, the visual data represents image data or video data used in response to the voice command signal and / or text command signal;

[0047] The encoding module is used to encode the visual data, the voice command signal and / or the text command signal based on the multimodal encoder to obtain a high-dimensional feature vector;

[0048] The processing module is also used to process the high-dimensional feature vector based on the preset custom attention mechanism in the language model to obtain the target feature sequence;

[0049] The decoding module is used to perform autoregressive decoding processing on the target feature sequence based on the multimodal decoder to obtain the action command signal;

[0050] An execution module is used to execute the physical action indicated by the action command signal.

[0051] Optionally, the preset custom attention mechanism includes a cross-attention mechanism and a linear attention mechanism;

[0052] The processing module is further configured to perform alignment processing on the high-dimensional feature vector based on the cross-attention mechanism, so as to map the high-dimensional feature vector to the word embedding space of the language model to obtain a multimodal sequence;

[0053] The processing module further includes a calculation module, used to perform attention calculation on the multimodal sequences in the word embedding space based on the linear attention mechanism to obtain the target feature sequence.

[0054] Optionally, the computing module is further configured to:

[0055] Based on the high-dimensional feature vectors of the multimodal sequences in the word embedding space, the query matrix, key matrix, and value matrix are calculated;

[0056] Based on the key matrix and the value matrix, the target state matrix is ​​calculated; matrix multiplication is performed on the target state matrix and the query matrix to obtain the target feature sequence.

[0057] Optionally, the processing module is further configured to process the key matrix based on a preset proxy matrix to obtain a processed key matrix, and to process the query matrix based on the preset proxy matrix to obtain a processed query matrix;

[0058] The calculation module is further configured to calculate the target state matrix based on the processed key matrix and the value matrix; and to perform matrix multiplication on the target state matrix and the processed query matrix to obtain the target feature sequence.

[0059] Optionally, the multimodal decoder includes an audio decoder, a text decoder, and a visual decoder;

[0060] The decoding module is also used for:

[0061] Based on the audio decoder, autoregressive decoding processing is performed on the audio features in the target feature sequence to generate an audio response, and / or, based on the text decoder, autoregressive decoding processing is performed on the text features in the target feature sequence to generate a text response; wherein, the audio response represents the audio information used to complete the voice command signal and / or text command signal, and the text response represents the text information used to complete the voice command signal and / or text command signal.

[0062] Based on the visual decoder, the visual features in the target feature sequence are subjected to autoregressive decoding processing to generate a target visual image; wherein, the target visual image represents the visual information used to complete the voice command signal and / or text command signal;

[0063] The processing module is further configured to integrate the target visual image, the text response, and / or the audio response to obtain an action command signal; wherein the action command signal represents the physical action used to complete the voice command signal and / or text command signal.

[0064] Optionally, the decoding module further includes: a generation module, used for:

[0065] Based on the visual features in the target feature sequence, and using the autoregressive visual generation algorithm in the visual decoder, a visual image at a preset resolution is generated.

[0066] Based on the visual image at the preset resolution, and using the autoregressive visual generation algorithm, a visual image at the next resolution adjacent to the preset resolution is generated; wherein, the next resolution is greater than the preset resolution.

[0067] The decoding module further includes a determining module, used to determine the visual image at the current resolution as the target visual image if the current resolution is the same as the resolution corresponding to the visual data.

[0068] Optionally, the multimodal encoder includes an audio encoder, a text encoder, and a visual encoder;

[0069] The encoding module is also used for:

[0070] Based on the audio encoder, the voice command signal is encoded to obtain an audio feature vector, and / or, based on the text encoder, the text command signal is encoded to obtain a text feature vector;

[0071] Based on the visual encoder, the visual data is encoded to obtain a visual feature vector;

[0072] The determining module is further configured to determine the visual feature vector, the text feature vector, and / or the audio feature vector as the high-dimensional feature vector.

[0073] Optionally, the visual encoder includes a understanding sub-encoder and a generating sub-encoder;

[0074] The processing module is further configured to:

[0075] Based on the understanding sub-encoder, the visual data is processed to obtain visual features;

[0076] Based on the generator sub-encoder, the visual features are processed to obtain a visual feature vector.

[0077] Optionally, the visual encoder is a trained composite encoder, and the data processing device based on a multimodal large model further includes: a training module, used for:

[0078] Input the pre-collected historical visual data samples into the pre-created initial visual encoder;

[0079] Based on the initial visual encoder, the historical visual data samples are divided into blocks to obtain multiple data sample blocks;

[0080] The data sample block is preprocessed, and the preprocessed data sample block is input into the first data processing model, the second data processing model and the third data processing model included in the initial visual encoder for processing;

[0081] Determine the first data feature output by the first data processing model, the second data feature output by the second data processing model, and the third data feature output by the third data processing model;

[0082] The first data feature, the second data feature, and the third data feature are concatenated to obtain a data feature sequence.

[0083] Based on a preset segmentation rule, the data feature sequence is segmented to obtain a first data feature subsequence and a second data feature subsequence; the first data feature subsequence is determined as the understanding sub-encoder, and the second data feature subsequence is determined as the generating sub-encoder; based on the understanding sub-encoder and the generating sub-encoder, the visual encoder is generated.

[0084] Thirdly, this application provides an electronic device, comprising:

[0085] Memory and processor;

[0086] The memory stores computer-executed instructions;

[0087] The processor executes computer execution instructions stored in the memory to implement the data processing method based on a multimodal large model as described in the first aspect above and in various possible implementations.

[0088] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the data processing method based on a multimodal large model as described in the first aspect and various possible implementations above.

[0089] Fifthly, this application provides a computer program product comprising a computer program that, when executed by a processor, implements the data processing method based on a multimodal large model as described in the first aspect and various possible implementations above.

[0090] This application provides a data processing method, apparatus, and electronic device based on a multimodal large model, applicable to service robots, which have a built-in multimodal large model. First, based on voice and / or text command signals sent by the target user, a visual data acquisition operation is triggered to acquire visual data. The visual data represents image or video data used in response to the voice and / or text command signals. Then, based on the multimodal encoder included in the multimodal large model, the visual data, as well as the voice and / or text command signals, are encoded to obtain a high-dimensional feature vector. Further, based on a custom attention mechanism in the language model included in the multimodal large model, the high-dimensional feature vector is processed to obtain a target feature sequence. Finally, based on the multimodal decoder included in the multimodal large model, the target feature sequence is subjected to autoregressive decoding processing to obtain an action command signal, and the physical action indicated by the action command signal is executed. This application uses a custom attention mechanism with lower computational complexity to replace the original self-attention mechanism in the language model, thereby accelerating the data processing speed of the language model. Simultaneously, by performing autoregressive decoding on the target feature sequence through a multimodal decoder, high-quality decoding results can be directly generated without additional data filtering to ensure data quality. Therefore, the method in this application can improve the data processing efficiency of large multimodal models. Attached Figure Description

[0091] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0092] Figure 1 A schematic diagram illustrating a scenario for data processing methods based on multimodal large models provided by existing technologies;

[0093] Figure 2 This is a schematic diagram of a multimodal large model for the data processing method based on a multimodal large model applicable to embodiments of this application;

[0094] Figure 3 A flowchart illustrating the data processing method based on a multimodal large model provided in this application embodiment;

[0095] Figure 4 A schematic diagram of the visual encoder training process for a data processing method based on a multimodal large model provided in this application embodiment;

[0096] Figure 5 A schematic diagram illustrating the attention calculation process of a data processing method based on a multimodal large model provided in this application embodiment;

[0097] Figure 6A schematic diagram of the visual decoder decoding process of the data processing method based on a multimodal large model provided in the embodiments of this application;

[0098] Figure 7 This is a schematic diagram of the structure of a data processing device based on a multimodal large model provided in an embodiment of this application;

[0099] Figure 8 A hardware structure diagram of the electronic device provided in this application.

[0100] The accompanying drawings have illustrated specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0101] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0102] The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented, for example, in orders other than those illustrated or described herein.

[0103] In this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0104] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0105] Figure 1 A schematic diagram illustrating a scenario for data processing methods based on multimodal large models provided by existing technologies. (Example) Figure 1 As shown, existing multimodal large-scale models are centered around a language model. They encode text, images, video, and speech into high-dimensional feature vectors using encoders of different modalities (text encoder, image encoder, video encoder, and audio encoder). These high-dimensional feature vectors are then merged and fed into the language model for attention calculation, resulting in a feature sequence. At the output end, before being fed into the decoders of different modalities, the feature sequence is filtered. The filtered feature sequences are then fed into the text decoder, image decoder, video decoder, and audio decoder respectively for decoding, ultimately yielding the outputs for each modality.

[0106] However, language models typically use their built-in self-attention mechanism to process the normalized vectors generated by encoders of different modalities. The computational complexity of the self-attention mechanism is the square of the length of the normalized vector, resulting in a huge computational burden for language models.

[0107] In addition, to improve the quality of the feature sequences output by the language model, the feature sequences will be filtered again before being sent to the encoder for encoding.

[0108] This shows that the self-attention mechanism inherent in language models, as well as the additional filtering processing, leads to low data processing efficiency in large multimodal models.

[0109] To address the aforementioned technical problems, considering the excessive computational complexity of the self-attention mechanism, the inventors decided to design a custom attention mechanism with lower computational complexity to replace the self-attention mechanism for data processing, thereby improving the data processing speed of the language model. Furthermore, while additional data filtering ensures the quality of the feature sequences output by the language model, it also impacts data processing speed. Therefore, the inventors conceived of using a decoding method that directly generates high-quality data, eliminating the filtering step. This approach simultaneously accelerates data processing efficiency and generates high-quality data.

[0110] The technical solution of this application and how it solves the above-mentioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0111] This application provides a data processing method based on a multimodal large model, which is applied to a service robot, and the service robot has a built-in multimodal large model. Figure 2This is a schematic diagram of a multimodal large model for the data processing method based on a multimodal large model applicable to embodiments of this application, as shown below. Figure 2 As shown, this large multimodal model includes a multimodal encoder, a speech model, and a multimodal decoder. The multimodal encoder includes an audio encoder, a text encoder, and a visual encoder, while the multimodal decoder includes an audio decoder, a text decoder, and a visual decoder.

[0112] Obviously, compared with the multimodal large model in the prior art, the multimodal large model of this application uses a visual encoder to replace the image encoder and video encoder in the multimodal large model of the prior art, and uses a visual decoder to replace the image decoder and video decoder in the multimodal large model of the prior art.

[0113] exist Figure 2 Based on the multimodal large model shown, Figure 3 A flowchart illustrating the data processing method based on a multimodal large model provided in this application embodiment is shown below. Figure 3 As shown, this data processing method based on a multimodal large model includes:

[0114] S301. Based on the voice command signal and / or text command signal sent by the target user, trigger the visual data acquisition operation to acquire visual data.

[0115] Visual data refers to image or video data used in response to voice and / or text command signals.

[0116] Interpretive, the target user sends voice and / or text commands to the service robot. Upon receiving these commands, the service robot analyzes them using a multimodal large-scale model and actively controls the robot to initiate visual data acquisition. Specifically, the service robot's camera takes one or more photos or records a video; the photos or videos represent the acquired visual data.

[0117] S302. Based on a multimodal encoder, visual data, as well as voice command signals and / or text command signals, are encoded to obtain high-dimensional feature vectors; based on a pre-defined custom attention mechanism in the language model, the high-dimensional feature vectors are processed to obtain the target feature sequence.

[0118] Since a multimodal encoder includes an audio encoder, a text encoder, and a visual encoder, it encodes visual data, as well as speech command signals and / or text command signals, to obtain high-dimensional feature vectors. Specifically, this includes: encoding speech command signals based on the audio encoder to obtain audio feature vectors, and / or encoding text command signals based on the text encoder to obtain text feature vectors. Furthermore, it encodes visual data based on the visual encoder to obtain visual feature vectors, thus determining that the visual feature vectors, text feature vectors, and / or audio feature vectors are high-dimensional feature vectors.

[0119] The methods of obtaining audio feature vectors through an audio encoder and / or text feature vectors through a text encoder are existing technologies and will not be elaborated upon here. It should be noted that the visual encoder is a trained composite encoder, including an understanding sub-encoder and a generative sub-encoder. The visual encoder relies on the collaborative work of the understanding sub-encoder and the generative sub-encoder to generate visual feature vectors.

[0120] Explanatory Figure 4 This diagram illustrates the visual encoder training process of a data processing method based on a multimodal large model, as provided in an embodiment of this application. Figure 4 As shown, the training process of the visual encoder is as follows: Pre-collected historical visual data samples are input into a pre-created initial visual encoder. The initial visual encoder then segments the historical visual data samples into multiple data sample blocks. These data sample blocks are then pre-processed and input into the initial visual encoder, which includes a first data processing model, a second data processing model, and a third data processing model. The first data feature output by the first data processing model, the second data feature output by the second data processing model, and the third data feature output by the third data processing model are determined. The first, second, and third data features are then concatenated to obtain a data feature sequence. This sequence is then segmented according to a preset segmentation rule to obtain a first data feature subsequence and a second data feature subsequence. The first data feature subsequence is designated as the understanding sub-encoder, and the second data feature subsequence is designated as the generative sub-encoder. Based on the understanding sub-encoder and the generative sub-encoder, the visual encoder is generated.

[0121] Thus, a trained visual encoder is obtained. Since the first, second, and third data processing models used in the training process of the visual encoder are all existing open-source models, they can be used directly without fine-tuning or optimization after the trained visual encoder is obtained. Among them, the first data processing model is usually a basic vision-language model, the second data processing model is usually a feature extraction training model, and the third data processing model is usually a large vision-language model.

[0122] It should be noted that during the training of the visual encoder, the preprocessing of the data sample blocks is normalization. However, since the first, second, and third data processing models may rely on the positional information of the data sample blocks when processing them, the preprocessing of the data sample blocks includes not only normalization but also positional encoding interpolation.

[0123] according to Figure 4 The training process of the visual encoder shown involves two data processing models. The first model relies on the positional information of the data sample blocks when processing them, while the second and third models also rely on this positional information. Therefore, the preprocessing of the data sample blocks before inputting them into the first model consists of positional encoding interpolation and normalization, while the preprocessing of the data sample blocks before inputting them into the second and third models is simply normalization. Positional encoding interpolation and normalization are existing technologies and will not be elaborated upon here.

[0124] After obtaining the trained visual encoder, visual feature vectors can be generated using the interpretive sub-encoder and the generative sub-encoder included in the visual encoder. Specifically, the visual data is processed based on the interpretive sub-encoder to obtain visual features, and then the visual features are processed based on the generative sub-encoder to obtain visual feature vectors.

[0125] The visual feature vector is combined with the text feature vector and / or audio feature vector to obtain a high-dimensional feature vector. Next, based on a pre-defined custom attention mechanism in the language model, the high-dimensional feature vector is processed to obtain the target feature sequence.

[0126] It should be understood that the attention mechanism is the core of the language model. This embodiment integrates the existing multi-head latent attention mechanism with the linear attention mechanism to obtain a preset custom attention mechanism. The multi-head latent attention mechanism includes a cross-attention mechanism and the self-attention mechanism inherent in the language model. This embodiment replaces the self-attention mechanism in the multi-head latent attention mechanism with a linear attention mechanism to reduce the computational cost of attention calculations.

[0127] Specifically, based on the cross-attention mechanism in the custom attention mechanism, the high-dimensional feature vectors are aligned to map them to the word embedding space of the language model, resulting in a multimodal sequence. Then, based on the linear attention mechanism in the custom attention mechanism, attention is calculated on the multimodal sequence in the word embedding space to obtain the target feature sequence.

[0128] Explanatory Figure 5 This diagram illustrates the attention calculation process of a data processing method based on a multimodal large model, as provided in an embodiment of this application. Figure 5 As shown, it should first be noted that the steps of the attention mechanism are "normalization technique → attention mechanism → normalization technique → feedforward network", which will not be elaborated here.

[0129] Combination Figure 5 The steps for attention calculation using a linear attention mechanism are as follows:

[0130] Step a1: Based on the high-dimensional feature vectors of multimodal sequences in the word embedding space, calculate the query (Q) matrix, key (K) matrix, and value (V) matrix.

[0131] It should be understood that a multimodal sequence includes multiple high-dimensional feature vectors. Let the multimodal sequence be defined as X, and each high-dimensional feature vector be defined as a token, then X = {token1, token2, ..., tokenN}. Here, N represents the number of high-dimensional feature vectors in the multimodal sequence.

[0132] Next, the multimodal sequence is divided into two subsequences, X_V and X_KQ. X_V represents the sequence associated with the V matrix, and X_KQ represents the sequence associated with the K and Q matrices. Further, a preset dimensionality reduction matrix is ​​used to reduce the dimensionality of the corresponding subsequences. Specifically, the dimensionality reduction matrix W is used. DV The subsequence X_V is reduced in dimensionality to obtain the compressed latent vector c. t V :

[0133] c t V =W DV ·h 1t

[0134] Among them, h 1t W represents the input vector of the t-th token in the subsequence X_V. DV The dimensionality reduction matrix is ​​pre-designed and will not be elaborated here.

[0135] Using the dimensionality reduction matrix W DKQ The subsequence X_KQ is dimensionality reduced to obtain the compressed latent vector c.t KQ :

[0136] c t KQ =W DKQ ·h 2t

[0137] Among them, h 2t Let W represent the input vector of the t-th token in the subsequence X_KQ. DKQ The dimensionality reduction matrix is ​​pre-designed and will not be elaborated here.

[0138] It should be noted that during the dimensionality reduction process, the dimensionality reduction matrix W can also be used. DK The multimodal sequence X is directly subjected to dimensionality reduction to obtain the compressed latent vector c. t K :

[0139] c t K =W DK ·h t

[0140] Among them, h t Let X represent the input vector of the t-th token in the multimodal sequence X.

[0141] Next, for the compressed latent vector c t V With c t KQ Then, the corresponding dimension-upgrading matrix is ​​used to obtain the Q matrix, K matrix, and V matrix. Specifically, the dimension-upgrading matrix W is used. UV For the compressed latent vector c t V After performing dimensionality increase processing, we obtain the V matrix v. t C :

[0142] v t C =W UV ·c t V

[0143] Using an upgraded matrix W UK For the compressed latent vector c t KQ After dimensionality increase, we obtain the K matrix k. t C :

[0144] k t C =W UK ·c tKQ

[0145] It should be noted that the compressed latent vector c derived from the subsequence X_KQ can be applied only. t KQ After dimensionality increase, we obtain the K matrix k. t C It can also simultaneously compress the latent vector c derived from the subsequence X_KQ. t KQ , and the compressed latent vector c derived from the multimodal sequence X t K After dimensionality increase, we obtain the K matrix k. t C No specific limitations are specified here. The above equation gives the compressed latent vector c only for the subsequence X_KQ. t KQ After dimensionality increase, we obtain the K matrix k. t C .

[0146] Using an upgraded matrix W UQ For the compressed latent vector c t KQ After dimensionality increase, we obtain the Q matrix q. t C :

[0147] q t C =W UQ ·c t KQ

[0148] Among them, W UV W UK With W UQ All of these are pre-designed dimension-upgrading matrices, which will not be elaborated here.

[0149] It should be noted that after obtaining the V matrix, K matrix, and Q matrix, further processing of the K matrix and Q matrix is ​​required to obtain the K matrix k carrying the position information. t R With Q matrix q t R Specifically, for the compressed latent vector c t V and the input vector h 1t Mapping is performed in the spaces corresponding to the K and Q matrices respectively. Then, Rotation Position Embedding (RoPE) is performed to obtain the K matrix k carrying position information. t R With Q matrix q tR :

[0150] k t R =RoPE(W KR ·h 1t )

[0151] q t R =RoPE(W QR ·c t V )

[0152] Among them, W KR With W QR This is a pre-designed transformation matrix for RoPE processing.

[0153] Step a2: Calculate the target state matrix based on the K matrix and the V matrix.

[0154] Step a3: Perform matrix multiplication on the target state matrix and the Q matrix to obtain the target feature sequence.

[0155] Interpretive, typically, attention calculations begin after obtaining the V, K, and Q matrices:

[0156] q t,i =[q C t,i q R t,i ]

[0157] k t,i =[k C t,i k R t,i ]

[0158]

[0159] u t =W O [o t,1 ,o t,2 ,……,o t,nh ]

[0160] Where, q C t,i Let q represent the i-th element in the Q matrix. R t,i This represents the i-th element in the Q-matrix carrying location information, q t,i k represents the i-th element in the concatenated matrix formed by the Q matrix and the Q matrix carrying position information. C t,i Let k represent the i-th element in matrix K.R t,i This represents the i-th element in the K matrix carrying location information, k t,i This represents the i-th element in the concatenated matrix formed by the K matrix and the K matrix carrying positional information. t,i This represents the attention calculation result, Softmax represents the weighted processing, and v C j,i d represents the element in the j-th row and i-th column of matrix V. h q t,i (k t,i The projection of d onto the Q matrix (K matrix) h R q t,i (k t,i The projection of the position information onto the Q matrix (K matrix). t W represents the feature sequence obtained after attention computation. O A preset matrix for integrating attention calculation results.

[0161] It should be understood that the above describes the standard steps for attention calculation using the V, K, and Q matrices, with a computational complexity of O(n^2). 2 ·d), where n represents the length of the multimodal sequence and d represents the dimension of the multimodal sequence. This embodiment introduces a preset surrogate matrix a. t C To reduce the computational complexity of attention calculations, the pre-defined proxy matrix can be directly derived from c. t V We obtain it by performing feature transformation, i.e., a t C =W A ·c t V W A It is a preset feature transformation matrix.

[0162] After introducing the pre-defined proxy matrix, the attention calculation process is as follows:

[0163] k S A =Softmax(Matmul(k t C a t C ))

[0164] q S A =Softmax(Matmul(q t C a t C ))

[0165] h t out =Matmul(Matmul(v t C k S A ), q S A )

[0166] Here, Matmul represents matrix multiplication. S A Indicates the use of a t C Processing k t C The processed k obtained t C q S A Indicates the use of a t C Process q t C The processed q t C h t out This represents the feature sequence obtained after attention calculation. It should be noted that Matmul(v t C k S A ) indicates based on v t C With k S A The obtained target state matrix.

[0167] Explained, after introducing a preset proxy matrix, the K matrix is ​​processed based on the preset proxy matrix to obtain the processed K matrix, and the Q matrix is ​​processed based on the preset proxy matrix to obtain the processed Q matrix. Then, based on the V matrix and the processed K matrix, the target state matrix is ​​calculated, and then matrix multiplication is performed on the target state matrix and the processed Q matrix to obtain the target feature sequence.

[0168] It should be noted that after introducing a pre-defined surrogate matrix, the computational complexity of attention calculation decreases to O(nkd), where k represents the dimension of the surrogate matrix. When k << n, O(nkd) is linear, demonstrating that introducing a pre-defined surrogate matrix can significantly reduce the computational complexity of attention calculation.

[0169] It is worth noting that in language models, because a custom attention mechanism replaces the original self-attention mechanism, the language model will be affected by the new custom attention mechanism, leading to deviations in the output. Therefore, it is necessary to fine-tune the language model with the added custom attention mechanism to align it back to the original feature space. It should be understood that some existing efficient fine-tuning schemes can be used to fine-tune the language model, requiring only a small amount of data and training resources to complete the alignment process.

[0170] S303. Based on the multimodal decoder, perform autoregressive decoding processing on the target feature sequence to obtain the action command signal; execute the physical action indicated by the action command signal.

[0171] Since the multimodal decoder includes an audio decoder, a text decoder, and a visual decoder, it performs autoregressive decoding processing on the target feature sequence to obtain the action command signal. Specifically, this includes: based on the audio decoder, performing autoregressive decoding processing on the audio features in the target feature sequence to generate an audio response; and / or, based on the text decoder, performing autoregressive decoding processing on the text features in the target feature sequence to generate a text response. The audio response represents the audio information used to complete the voice command signal and / or the text command signal, and the text response represents the text information used to complete the voice command signal and / or the text command signal. Then, based on the visual encoder, it performs autoregressive decoding processing on the visual features in the target feature sequence to generate a target visual image. The target visual image represents the data information used to complete the voice command signal and / or the text command signal.

[0172] Generating audio responses via an audio decoder and / or text responses via a text decoder are existing techniques and will not be elaborated upon here. It should be noted that a pre-set autoregressive visual generation algorithm is added to the visual decoder, which the visual decoder relies on to generate the target visual image.

[0173] Specifically, Figure 6 This is a schematic diagram of the visual decoder decoding process of the data processing method based on a multimodal large model provided in an embodiment of this application. Figure 6 As shown, based on the visual features in the target feature sequence, a visual image at a preset resolution (l1) is generated using an autoregressive visual generation algorithm in the visual decoder. Then, based on the visual image at the preset resolution, a visual image at the next resolution (l2), adjacent to the preset resolution, is generated using the autoregressive visual generation algorithm. The next resolution is greater than the preset resolution. This process is repeated until the current resolution (l1) is reached. k If the resolution is the same as that of the visual data, then the visual image at the current resolution is determined as the target visual image.

[0174] Furthermore, after obtaining the target visual image, text response, and / or audio response, these are integrated to obtain an action command signal. This action command signal is used to perform the physical actions corresponding to the voice and / or text command signals. The service robot responds to the voice and / or text command signals by executing the physical actions indicated by the action command signal.

[0175] For example, integrating the target visual image, text response, and / or audio response can begin by aligning the visual encoder with the audio encoder and / or text encoder, followed by fine-tuning the visual encoder and the audio encoder and / or text encoder to align the generative capabilities of the multimodal large model. Finally, the target visual image, text response, and / or audio response are interleaved and fine-tuned based on various complex tasks. This example only illustrates one possible integration method, and this application does not limit the integration process of the target visual image, text response, and / or audio response.

[0176] This application provides a data processing method based on a multimodal large model, applied to service robots. The service robot has a built-in multimodal large model, which includes a multimodal encoder, a language model, and a multimodal decoder. First, based on voice and / or text command signals sent by the target user, a visual data acquisition operation is triggered to acquire visual data. The visual data represents image or video data used in response to the voice and / or text command signals. Then, based on the multimodal encoder, which includes an audio encoder, a text encoder, and a visual encoder, the voice and / or text command signals and the visual data are encoded respectively to obtain corresponding audio and / or text feature vectors and visual feature vectors. The visual feature vectors, text feature vectors, and / or audio feature vectors are determined as high-dimensional feature vectors. Further, based on the cross-attention mechanism included in the custom attention mechanism in the language model, the high-dimensional feature vectors are aligned to map them to the word embedding space of the language model, resulting in a multimodal sequence. Then, based on the linear attention mechanism included in the custom attention mechanism in the language model, the query matrix, key matrix, and value matrix are calculated according to the high-dimensional feature vectors of the multimodal sequence in the word embedding space. Based on a preset surrogate matrix, the key matrix is ​​processed to obtain a processed key matrix, and the query matrix is ​​processed based on the preset surrogate matrix to obtain a processed query matrix. The target state matrix is ​​calculated based on the value matrix and the processed key matrix. Matrix multiplication is then performed on the target state matrix and the processed query matrix to obtain the target feature sequence. Next, based on the audio decoder, text decoder, and visual decoder included in the multimodal decoder, autoregressive decoding processing is performed on the audio and / or text features and visual features in the target feature sequence, respectively, generating corresponding audio and / or text responses and a target visual image. The target visual image, audio response, and / or text response are integrated to obtain an action command signal, and the physical action indicated by the action command signal is executed. This application uses a custom attention mechanism with lower computational complexity to replace the original self-attention mechanism in the language model, thereby accelerating the data processing speed of the language model. Simultaneously, by performing autoregressive decoding on the target feature sequence through a multimodal decoder, high-quality decoding results can be directly generated without additional data filtering to ensure data quality. Therefore, the method in this application can improve the data processing efficiency of large multimodal models.

[0177] Figure 7 This is a schematic diagram of the structure of a data processing device based on a multimodal large model provided in an embodiment of this application, as shown below. Figure 7As shown, the data processing device 700 based on a multimodal large model is applied to a service robot. The service robot has a built-in multimodal large model, which includes a multimodal encoder, a language model, and a multimodal decoder. The data processing device 700 based on the multimodal large model includes: a processing module 701, an encoding module 702, a decoding module 703, and an execution module 704.

[0178] The processing module 701 is used to trigger a visual data acquisition operation based on the voice command signal and / or text command signal sent by the target user, and to acquire visual data; wherein the visual data represents image data or video data used in response to the voice command signal and / or text command signal.

[0179] The encoding module 702 is used to encode visual data, as well as voice command signals and / or text command signals based on a multimodal encoder, to obtain high-dimensional feature vectors.

[0180] The processing module 701 is also used to process the high-dimensional feature vector based on the preset custom attention mechanism in the language model to obtain the target feature sequence;

[0181] Decoding module 703 is used to perform autoregressive decoding processing on the target feature sequence based on a multimodal decoder to obtain the action command signal;

[0182] The execution module 704 is used to execute the physical action indicated by the action command signal.

[0183] Optional, preset custom attention mechanisms include cross-attention and linear attention mechanisms;

[0184] The processing module 701 is also used to perform alignment processing on high-dimensional feature vectors based on the cross-attention mechanism, so as to map the high-dimensional feature vectors to the word embedding space of the language model to obtain multimodal sequences;

[0185] The processing module 701 also includes a calculation module 705, which is used to perform attention calculation on multimodal sequences in the word embedding space based on a linear attention mechanism to obtain the target feature sequence.

[0186] Optionally, the computing module 705 is also used for:

[0187] Calculate the query matrix, key matrix, and value matrix based on the high-dimensional feature vectors of multimodal sequences in the word embedding space;

[0188] Based on the key matrix and value matrix, calculate the target state matrix; perform matrix multiplication on the target state matrix and the query matrix to obtain the target feature sequence.

[0189] Optionally, the processing module 701 is also used to process the key matrix based on the preset proxy matrix to obtain the processed key matrix, and to process the query matrix based on the preset proxy matrix to obtain the processed query matrix.

[0190] The calculation module 705 is also used to calculate the target state matrix based on the processed key matrix and value matrix; and to perform matrix multiplication on the target state matrix and the processed query matrix to obtain the target feature sequence.

[0191] Optionally, the multimodal decoder includes an audio decoder, a text decoder, and a visual decoder;

[0192] Decoding module 703 is also used for:

[0193] Based on an audio decoder, autoregressive decoding processing is performed on the audio features in the target feature sequence to generate an audio response, and / or, based on a text decoder, autoregressive decoding processing is performed on the text features in the target feature sequence to generate a text response; wherein, the audio response represents the audio information used to complete the voice command signal and / or text command signal, and the text response represents the text information used to complete the voice command signal and / or text command signal.

[0194] Based on a visual decoder, the visual features in the target feature sequence are subjected to autoregressive decoding processing to generate a target visual image; wherein, the target visual image represents the visual information used to complete the voice command signal and / or text command signal;

[0195] The processing module 701 is further configured to integrate the target visual image, text response and / or audio response to obtain an action command signal; wherein the action command signal represents the physical action used to complete the voice command signal and / or text command signal.

[0196] Optionally, the decoding module 703 further includes: a generation module 706, used for:

[0197] Based on the visual features in the target feature sequence, a visual image at a preset resolution is generated using an autoregressive visual generation algorithm in the visual decoder.

[0198] Based on the visual image at a preset resolution, an autoregressive visual generation algorithm is used to generate a visual image at the next resolution adjacent to the preset resolution; wherein, the next resolution is greater than the preset resolution.

[0199] The decoding module 703 further includes a determination module 707, which determines the visual image at the current resolution as the target visual image if the current resolution is the same as the resolution corresponding to the visual data.

[0200] Optionally, the multimodal encoder includes an audio encoder, a text encoder, and a visual encoder;

[0201] Encoding module 702 is also used for:

[0202] Based on an audio encoder, the voice command signal is encoded to obtain an audio feature vector, and / or based on a text encoder, the text command signal is encoded to obtain a text feature vector;

[0203] Based on the visual encoder, visual data is encoded to obtain visual feature vectors;

[0204] The determination module 707 is also used to determine visual feature vectors, text feature vectors, and / or audio feature vectors, which are high-dimensional feature vectors.

[0205] Optionally, the visual encoder includes a cognition sub-encoder and a generation sub-encoder;

[0206] Processing module 701 is also used for:

[0207] Based on the understanding sub-encoder, visual data is processed to obtain visual features;

[0208] Based on the generator sub-encoder, visual features are processed to obtain visual feature vectors.

[0209] Optionally, the visual encoder is a trained composite encoder. The data processing device 700 based on a multimodal large model also includes: a training module 708, used for:

[0210] Input the pre-collected historical visual data samples into the pre-created initial visual encoder;

[0211] Based on the initial visual encoder, the historical visual data samples are divided into blocks to obtain multiple data sample blocks.

[0212] The data sample blocks are preprocessed and then input into the first data processing model, the second data processing model and the third data processing model included in the initial visual encoder for further processing.

[0213] Determine the first data feature output by the first data processing model, the second data feature output by the second data processing model, and the third data feature output by the third data processing model;

[0214] The first data feature, the second data feature, and the third data feature are concatenated to obtain a data feature sequence.

[0215] Based on preset segmentation rules, the data feature sequence is segmented to obtain a first data feature subsequence and a second data feature subsequence; the first data feature subsequence is determined as the understanding sub-encoder, and the second data feature subsequence is determined as the generation sub-encoder; based on the understanding sub-encoder and the generation sub-encoder, a visual encoder is generated.

[0216] The data processing apparatus based on a multimodal large model provided in this application can be used to execute the data processing method based on a multimodal large model in any of the above embodiments. Its implementation principle and technical effect are similar, and will not be described again here.

[0217] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented via processing element calls in software, while others are implemented in hardware. Additionally, these modules can be fully or partially integrated together, or implemented independently. The processing element mentioned here can be an integrated circuit with signal processing capabilities. During implementation, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0218] Figure 8 A hardware structure diagram of the electronic device provided in this application. For example... Figure 8 As shown, the electronic device 800 includes:

[0219] Processor 801 and memory 802;

[0220] The memory stores the instructions that the computer executes;

[0221] The processor executes computer execution instructions stored in memory 802, causing the electronic device to perform the data processing method based on the multimodal large model described above.

[0222] It should be understood that the processor 801 described above can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor. The memory 802 may include high-speed random access memory (RAM), and may also include non-volatile memory (NVM), such as at least one disk storage device, or a USB flash drive, external hard drive, read-only memory, disk, or optical disc, etc.

[0223] The electronic device provided in this application embodiment can be used to execute the data processing method based on a multimodal large model provided in any of the above method embodiments. Its implementation principle and technical effect are similar, and will not be described again here.

[0224] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the data processing method based on a multimodal large model as described above.

[0225] This application also provides a computer program product, which includes a computer program that, when executed by a processor, is used to implement the data processing method based on a multimodal large model as described above.

[0226] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0227] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0228] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0229] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or in the form of software program modules.

[0230] When integrated units / modules are implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage unit can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc.

[0231] If the integrated unit / module is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0232] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0233] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0234] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A data processing method based on a multimodal large model, characterized in that, The method is applied to a service robot, which has a built-in multimodal large model. The multimodal large model includes a multimodal encoder, a language model, and a multimodal decoder. The method includes: Based on the voice command signal and / or text command signal sent by the target user, a visual data acquisition operation is triggered to acquire visual data; wherein, the visual data represents image data or video data used in response to the voice command signal and / or text command signal; Based on the multimodal encoder, the visual data, the voice command signal, and / or the text command signal are encoded to obtain a high-dimensional feature vector; based on the preset custom attention mechanism in the language model, the high-dimensional feature vector is processed to obtain the target feature sequence; Based on the multimodal decoder, the target feature sequence is subjected to autoregressive decoding processing to obtain an action command signal; the physical action indicated by the action command signal is executed.

2. The method according to claim 1, characterized in that, The preset custom attention mechanism includes cross attention mechanism and linear attention mechanism; Based on the pre-defined custom attention mechanism in the language model, the high-dimensional feature vector is processed to obtain the target feature sequence, including: Based on the cross-attention mechanism, the high-dimensional feature vector is aligned to map it to the word embedding space of the language model, thereby obtaining a multimodal sequence. Based on the linear attention mechanism, attention calculation is performed on the multimodal sequences in the word embedding space to obtain the target feature sequence.

3. The method according to claim 2, characterized in that, Based on the linear attention mechanism, attention is calculated on the multimodal sequences in the word embedding space to obtain the target feature sequence, including: Based on the high-dimensional feature vectors of the multimodal sequences in the word embedding space, the query matrix, key matrix, and value matrix are calculated; Based on the key matrix and the value matrix, the target state matrix is ​​calculated; matrix multiplication is performed on the target state matrix and the query matrix to obtain the target feature sequence.

4. The method according to claim 3, characterized in that, Calculate the target state matrix based on the key matrix and the value matrix; Perform matrix multiplication on the target state matrix and the query matrix to obtain the target feature sequence, including: Based on the preset proxy matrix, the key matrix is ​​processed to obtain the processed key matrix, and based on the preset proxy matrix, the query matrix is ​​processed to obtain the processed query matrix; Based on the processed key matrix and the value matrix, the target state matrix is ​​calculated; matrix multiplication is performed on the target state matrix and the processed query matrix to obtain the target feature sequence.

5. The method according to claim 1, characterized in that, The multimodal decoder includes an audio decoder, a text decoder, and a visual decoder; Based on the multimodal decoder, the target feature sequence is subjected to autoregressive decoding processing to obtain an action command signal, including: Based on the audio decoder, autoregressive decoding processing is performed on the audio features in the target feature sequence to generate an audio response, and / or, based on the text decoder, autoregressive decoding processing is performed on the text features in the target feature sequence to generate a text response; wherein, the audio response represents the audio information used to complete the voice command signal and / or text command signal, and the text response represents the text information used to complete the voice command signal and / or text command signal. Based on the visual decoder, the visual features in the target feature sequence are subjected to autoregressive decoding processing to generate a target visual image; wherein, the target visual image represents the visual information used to complete the voice command signal and / or text command signal; The target visual image, the text response, and / or the audio response are integrated to obtain an action command signal; wherein the action command signal represents the physical action used to complete the voice command signal and / or text command signal.

6. The method according to claim 5, characterized in that, Based on the visual decoder, autoregressive decoding processing is performed on the visual features in the target feature sequence to generate a target visual image, including: Based on the visual features in the target feature sequence, and using the autoregressive visual generation algorithm in the visual decoder, a visual image at a preset resolution is generated. Based on the visual image at the preset resolution, and using the autoregressive visual generation algorithm, a visual image at the next resolution adjacent to the preset resolution is generated; wherein, the next resolution is greater than the preset resolution. If the current resolution is the same as the resolution corresponding to the visual data, then the visual image at the current resolution is determined as the target visual image.

7. The method according to any one of claims 1 to 6, characterized in that, The multimodal encoder includes an audio encoder, a text encoder, and a visual encoder; Based on the multimodal encoder, the visual data, the speech command signal, and / or the text command signal are encoded to obtain a high-dimensional feature vector, including: Based on the audio encoder, the voice command signal is encoded to obtain an audio feature vector, and / or, based on the text encoder, the text command signal is encoded to obtain a text feature vector; Based on the visual encoder, the visual data is encoded to obtain a visual feature vector; The visual feature vector, the text feature vector, and / or the audio feature vector are determined to be the high-dimensional feature vector.

8. The method according to claim 7, characterized in that, The visual encoder includes a understanding sub-encoder and a generating sub-encoder; Based on the visual encoder, the visual data is encoded to obtain a visual feature vector, including: Based on the understanding sub-encoder, the visual data is processed to obtain visual features; Based on the generator sub-encoder, the visual features are processed to obtain a visual feature vector.

9. The method according to claim 7, wherein the visual encoder is a trained composite encoder, and the training process of the visual encoder includes: Input the pre-collected historical visual data samples into the pre-created initial visual encoder; Based on the initial visual encoder, the historical visual data samples are divided into blocks to obtain multiple data sample blocks; The data sample block is preprocessed, and the preprocessed data sample block is input into the first data processing model, the second data processing model and the third data processing model included in the initial visual encoder for processing; Determine the first data feature output by the first data processing model, the second data feature output by the second data processing model, and the third data feature output by the third data processing model; The first data feature, the second data feature, and the third data feature are concatenated to obtain a data feature sequence. Based on a preset segmentation rule, the data feature sequence is segmented to obtain a first data feature subsequence and a second data feature subsequence; the first data feature subsequence is determined as the understanding sub-encoder, and the second data feature subsequence is determined as the generating sub-encoder; based on the understanding sub-encoder and the generating sub-encoder, the visual encoder is generated.

10. A data processing device based on a multimodal large model, characterized in that, The device is applied to a service robot, which has a built-in multimodal large model. The multimodal large model includes a multimodal encoder, a language model, and a multimodal decoder. The device includes: The processing module is used to trigger a visual data acquisition operation based on the voice command signal and / or text command signal sent by the target user, and acquire visual data; wherein, the visual data represents image data or video data used in response to the voice command signal and / or text command signal; The encoding module is used to encode the visual data, the voice command signal and / or the text command signal based on the multimodal encoder to obtain a high-dimensional feature vector; The processing module is also used to process the high-dimensional feature vector based on the preset custom attention mechanism in the language model to obtain the target feature sequence; The decoding module is used to perform autoregressive decoding processing on the target feature sequence based on the multimodal decoder to obtain the action command signal; An execution module is used to execute the physical action indicated by the action command signal.

11. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the data processing method based on a multimodal large model as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the data processing method based on a multimodal large model as described in any one of claims 1 to 9.

Images