Data processing method and apparatus, device, and readable storage medium

Abstract

The present application discloses a data processing method and apparatus, a device, and a readable storage medium. The method comprises: combining M vision mapping vectors generated from media data and N text mapping vectors generated from text information into a mapping vector sequence, and in the mapping vector sequence, inserting a compression token vector between the M vision mapping vectors and the N text mapping vectors to obtain a vision compression sequence; performing attention processing on the vision compression sequence to obtain an attention result vector; determining a unit attention vector associated with the compression token vector in the attention result vector as a global compression vector, the vector length of the global compression vector being less than the sum of vector lengths of the M vision mapping vectors; and generating a question-answer result on the basis of the global compression vector and unit attention vectors associated with the N text mapping vectors. By using the present application, vision mapping vectors can be compressed, thereby reducing calculation costs while improving model performance.

Description

Data processing methods, apparatus, equipment and readable storage media

[0001] This application claims priority to Chinese Patent Application No. 202410782364.3, filed on June 17, 2024, entitled “Data Processing Method, Apparatus, Device and Readable Storage Medium”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of computer technology, and in particular to a data processing method, apparatus, device, and readable storage medium. Background Technology

[0003] Existing large vision-language models (LVLMs) are often limited by a finite context window, which is the length of the longest sequence of tokens that the model can accept as input. As the number of input images increases and the resolution improves, the context length becomes the main factor limiting the model's performance.

[0004] In image encoding, an image is divided into multiple smaller regions, each of which is encoded separately. The tokens output from each region are then concatenated to obtain a token sequence. High-resolution image or video inputs map to token sequences that occupy a large amount of context, often requiring segmentation for processing. This is computationally expensive. If the token sequence is segmented, each segment may only contain partial context, causing the model to fail to understand the complete contextual information. Segmentation can also lead to the loss of crucial information connecting the two ends, potentially resulting in feature loss or an inability to understand the context, leading to poor model performance. Summary of the Invention

[0005] This application provides a data processing method, apparatus, device, and readable storage medium that can compress visual mapping vectors, thereby improving model performance while reducing computational costs.

[0006] One embodiment of this application provides a data processing method, including:

[0007] Acquire at least two units of media data, perform vector mapping on each unit of media data, and obtain M visual mapping vectors, where M is a positive integer, and at least two units of media data belong to the media data.

[0008] Obtain text information, perform vector mapping on the text information to obtain N text mapping vectors, where N is a positive integer;

[0009] M visual mapping vectors and N text mapping vectors are combined into a mapping vector sequence. A compressed marker vector is then inserted between the M visual mapping vectors and the N text mapping vectors in the mapping vector sequence to obtain a visual compression sequence.

[0010] Attention processing is applied to the visual compression sequence to obtain an attention result vector, which includes unit attention vectors corresponding to M visual mapping vectors, compressed label vectors, and N text mapping vectors.

[0011] The unit attention vector associated with the compressed label vector in the attention result vector is determined as the global compressed vector. The global compressed vector is used to characterize the global semantics of the M visual mapping vectors, and / or the vector length of the global compressed vector is less than the sum of the vector lengths of the M visual mapping vectors.

[0012] Based on a unit attention vector associated with a global compression vector and N text mapping vectors, question-and-answer results are generated for media data and text information.

[0013] Specifically, the text information is vector-mapped to obtain N text mapping vectors, including:

[0014] The text information is input into the target visual compression model. In the target visual compression model, the text information is split into words to obtain the word sequence corresponding to the text information. The word sequence includes N words. A word in the word sequence refers to the smallest basic unit obtained after splitting the text information.

[0015] Each word in the word sequence is mapped separately to obtain N text mapping vectors.

[0016] The attention process involves applying attention to the visually compressed sequence to obtain an attention result vector, which includes:

[0017] The visual compression sequence is input into the target visual compression model; the target visual compression model includes an attention mask layer.

[0018] In the attention mask layer, the visual compressed sequence is multiplied by the query parameter matrix to obtain the query matrix, the visual compressed sequence is multiplied by the key parameter matrix to obtain the key matrix, and the visual compressed sequence is multiplied by the value parameter matrix to obtain the value matrix.

[0019] The mask matrix is ​​obtained based on the compressed tag vector. Based on the mask matrix, attention calculation is performed on the query matrix, key matrix, and value matrix to obtain the unit attention vectors corresponding to the M visual mapping vectors and the compressed tag vectors respectively. The unit attention vectors corresponding to the M visual mapping vectors and the compressed tag vectors respectively are determined as the attention result vectors. The mask matrix is ​​used to shield the attention of the N text mapping vectors on the M visual mapping vectors during the attention calculation process.

[0020] Specifically, a mask matrix is ​​obtained based on the compressed tag vector. Based on the mask matrix, attention calculation is performed on the query matrix, key matrix, and value matrix to obtain unit attention vectors corresponding to M visual mapping vectors and compressed tag vectors, respectively. The unit attention vectors corresponding to the M visual mapping vectors, compressed tag vectors, and N text mapping vectors are determined as the attention result vectors, including:

[0021] The query matrix and the transpose of the key matrix are multiplied by a dot product to obtain the attention score matrix. The masking matrix is ​​generated based on the square root of the dimension value of the key matrix and the attention score matrix to obtain the causal matrix; the causal matrix is ​​a lower triangular matrix.

[0022] The mask matrix is ​​obtained based on the compressed label vector and the matrix to be masked. The matrix to be masked, the causal matrix, and the mask matrix are multiplied by a dot product to obtain the mask fraction matrix. The mask fraction matrix is ​​normalized to obtain the normalized matrix. The normalized matrix is ​​multiplied by the value matrix to obtain the unit attention vectors corresponding to M visual mapping vectors, compressed label vectors, and N text mapping vectors. The unit attention vectors corresponding to the M visual mapping vectors, compressed label vectors, and N text mapping vectors are determined as the attention result vectors. The mask matrix is ​​specifically used to mask the elements in the matrix to be masked that represent the N text mapping vectors focusing on the M visual mapping vectors.

[0023] The process of obtaining the mask matrix based on the compressed marker vector and the matrix to be masked includes:

[0024] An initial matrix of M+N+1 rows and M+N+1 columns is generated based on the compressed marker vector. In the i-th row and j-th column of the initial matrix, if i belongs to the row number of the N text mapping vectors in the matrix to be masked, and j belongs to the column number of the M visual mapping vectors in the matrix to be masked, then the element in the i-th row and j-th column of the initial matrix is ​​set as the first mask value.

[0025] Set all elements in the initial matrix except those with the first mask value to the second mask value to obtain the mask matrix.

[0026] The attention process involves applying attention to the visually compressed sequence to obtain an attention result vector, which includes:

[0027] The mask matrix is ​​obtained based on the compressed tag vector. Attention processing is then performed on the visual compressed sequence based on the mask matrix to obtain the attention result vector. The mask matrix is ​​used to shield the attention of N text mapping vectors on M visual mapping vectors during the attention processing.

[0028] Specifically, the system generates question-and-answer results for media data and text information based on a unit attention vector associated with a global compression vector and N text mapping vectors, including:

[0029] The unit attention vector, which is associated with the global compression vector and N text mapping vectors, is input into the target visual compression model; the target visual compression model includes a feature understanding layer;

[0030] Based on the feature understanding layer, a unit attention vector associated with a global compression vector and N text mapping vectors is used to generate a prediction probability for media data. The prediction result with the highest prediction probability is determined as the question-and-answer result for media data and text information.

[0031] In this mapping vector sequence, the M visual mapping sequences are arranged before the N text mapping vectors; the compressed marker vector is inserted between the M visual mapping vectors and the N text mapping vectors in the mapping vector sequence, including:

[0032] If the resolution of the media data is greater than the compression threshold, a compressed vector value S is generated based on the resolution of the media data. S compressed marker vectors are inserted after the M visual mapping vectors in the mapping vector sequence to obtain the visual compression sequence; S is a positive integer; the S compressed marker vectors in the visual compression sequence are located before the N text mapping vectors.

[0033] Attention processing is applied to the visually compressed sequence to obtain an attention result vector, including:

[0034] Based on S compressed label vectors, attention processing is performed on the visual compressed sequence to obtain an attention result vector; the attention result vector includes unit attention vectors corresponding to M visual mapping vectors, S compressed label vectors, and N text mapping vectors respectively.

[0035] This application provides another data processing method, including:

[0036] Input the media data sample, along with the question text and answer labels for the media data sample, into the initial visual compression model;

[0037] The media data sample is divided into at least two units of media data using an initial visual compression model. Vector mapping is performed on each unit of media data to obtain M visual mapping vectors. Vector mapping is also performed on the question text corresponding to the media data sample to obtain N text mapping vectors. M and N are both positive integers.

[0038] M visual mapping vectors and N text mapping vectors are combined into a mapping vector sequence. A compressed marker vector is then inserted between the M visual mapping vectors and the N text mapping vectors in the mapping vector sequence to obtain a visual compression sequence.

[0039] The visual compression sequence is subjected to attention processing to obtain an attention result vector, which includes unit attention vectors corresponding to M visual mapping vectors and compressed label vectors respectively.

[0040] The unit attention vector associated with the compressed label vector in the attention result vector is determined as the global compression vector; the global compression vector is used to characterize the global semantics of the M visual mapping vectors, and / or the vector length of the global compression vector is less than the sum of the vector lengths of the M visual mapping vectors;

[0041] Based on the global compression vector and N text mapping vectors, the prediction results corresponding to the media data samples are generated. Based on the prediction results and answer labels, the model parameters of the initial visual compression model are adjusted until the initial visual compression model meets the model convergence condition, and the target visual compression model is obtained. The target visual compression model is used to generate question and answer results for media data and text information.

[0042] This also includes:

[0043] Based on the data distribution information corresponding to the media data samples, multimodal public data with similar data distribution information is obtained. The visual language model is trained based on the multimodal public data to obtain an initial visual compression model that converges in training on multiple modalities.

[0044] The prediction results are the label probability distributions. Based on the prediction results and answer labels, the model parameters of the initial visual compression model are adjusted until the initial visual compression model meets the model convergence condition, thus obtaining the target visual compression model, which includes:

[0045] The label probability distribution is generated based on the answer labels. The relative entropy between the label probability distribution and the predicted probability distribution is determined as the model loss value. The model parameters of the initial visual compression model are adjusted based on the model loss value until the initial visual compression model meets the model training conditions, thus obtaining the target visual compression model.

[0046] One embodiment of this application provides a data processing apparatus, including:

[0047] The visual encoding module is used to acquire at least two units of media data, perform vector mapping on each unit of media data, and obtain M visual mapping vectors, where M is a positive integer, and at least two units of media data belong to the media data.

[0048] The text encoding module is used to acquire text information, perform vector mapping on the text information, and obtain N text mapping vectors, where N is a positive integer.

[0049] The insertion processing module is used to combine M visual mapping vectors and N text mapping vectors into a mapping vector sequence, and insert the compressed marker vector between the M visual mapping vectors and the N text mapping vectors in the mapping vector sequence to obtain a visual compression sequence.

[0050] The compression processing module is used to perform attention processing on the visual compression sequence to obtain an attention result vector. The attention result vector includes unit attention vectors corresponding to M visual mapping vectors, compressed label vectors, and N text mapping vectors.

[0051] The compression determination module is used to determine the unit attention vector associated with the compression label vector in the attention result vector as the global compression vector. The global compression vector is used to characterize the global semantics of the M visual mapping vectors, and / or the vector length of the global compression vector is less than the sum of the vector lengths of the M visual mapping vectors.

[0052] The output module generates question-and-answer results for media data and text information based on a unit attention vector associated with a global compression vector and N text mapping vectors.

[0053] In one possible implementation, the text encoding module is used to perform vector mapping on the text information. When N text mapping vectors are obtained, it is specifically used to perform the following operations:

[0054] The text information is input into the target visual compression model. In the target visual compression model, the text information is split into words to obtain the word sequence corresponding to the text information. The word sequence includes N words. A word in the word sequence refers to the smallest basic unit obtained after splitting the text information.

[0055] Each word in the word sequence is mapped separately to obtain N text mapping vectors.

[0056] In one possible implementation, the compression processing module is used to perform attention processing on the visual compressed sequence. When obtaining the attention result vector, it is specifically used to perform the following operations:

[0057] The visual compression sequence is input into the target visual compression model; the target visual compression model includes an attention mask layer.

[0058] In the attention mask layer, the visual compressed sequence is multiplied by the query parameter matrix to obtain the query matrix, the visual compressed sequence is multiplied by the key parameter matrix to obtain the key matrix, and the visual compressed sequence is multiplied by the value parameter matrix to obtain the value matrix.

[0059] The mask matrix is ​​obtained based on the compressed tag vector. Based on the mask matrix, attention is calculated on the query matrix, key matrix, and value matrix to obtain the unit attention vectors corresponding to M visual mapping vectors, compressed tag vectors, and N text mapping vectors. The unit attention vectors corresponding to the M visual mapping vectors, compressed tag vectors, and N text mapping vectors are determined as the attention result vectors. The mask matrix is ​​used to shield the attention of the N text mapping vectors on the M visual mapping vectors during the attention calculation process.

[0060] In one possible implementation, the compression processing module is used to obtain a mask matrix based on the compressed label vector. Based on the mask matrix, attention calculation is performed on the query matrix, key matrix, and value matrix to obtain unit attention vectors corresponding to M visual mapping vectors and compressed label vectors, respectively. When the unit attention vectors corresponding to the M visual mapping vectors and compressed label vectors are determined as the attention result vectors, they are specifically used to perform the following operations:

[0061] The query matrix and the transpose of the key matrix are multiplied by a dot product to obtain the attention score matrix. The masking matrix is ​​generated based on the square root of the dimension value of the key matrix and the attention score matrix to obtain the causal matrix; the causal matrix is ​​a lower triangular matrix.

[0062] The mask matrix is ​​obtained based on the compressed label vector and the matrix to be masked. The matrix to be masked, the causal matrix, and the mask matrix are multiplied by a dot product to obtain the mask fraction matrix. The mask fraction matrix is ​​normalized to obtain the normalized matrix. The normalized matrix is ​​multiplied by the value matrix to obtain the unit attention vectors corresponding to M visual mapping vectors, compressed label vectors, and N text mapping vectors. The unit attention vectors corresponding to the M visual mapping vectors, compressed label vectors, and N text mapping vectors are determined as the attention result vectors. The mask matrix is ​​specifically used to mask the elements in the matrix to be masked that represent the N text mapping vectors focusing on the M visual mapping vectors.

[0063] In one possible implementation, when the compression processing module obtains the mask matrix based on the compression marker vector and the mask matrix to be masked, it specifically performs the following operations:

[0064] An initial matrix of M+N+1 rows and M+N+1 columns is generated based on the compressed marker vector. In the i-th row and j-th column of the initial matrix, if i belongs to the row number of the N text mapping vectors in the matrix to be masked, and j belongs to the column number of the M visual mapping vectors in the matrix to be masked, then the element in the i-th row and j-th column of the initial matrix is ​​set as the first mask value.

[0065] Set all elements in the initial matrix except those with the first mask value to the second mask value to obtain the mask matrix.

[0066] In one possible implementation, the compression processing module is used to perform attention processing on the visual compressed sequence. When obtaining the attention result vector, it is specifically used to perform the following operations:

[0067] The mask matrix is ​​obtained based on the compressed tag vector. Attention processing is then performed on the visual compressed sequence based on the mask matrix to obtain the attention result vector. The mask matrix is ​​used to shield the attention of N text mapping vectors on M visual mapping vectors during the attention processing.

[0068] In one possible implementation, when the output module generates question-and-answer results for media data and text information based on the global compression vector and N text mapping vectors, it specifically performs the following operations:

[0069] The unit attention vector, which is associated with the global compression vector and N text mapping vectors, is input into the target visual compression model; the target visual compression model includes a feature understanding layer;

[0070] Based on the feature understanding layer, a unit attention vector associated with a global compression vector and N text mapping vectors is used to generate a prediction probability for media data. The prediction result with the highest prediction probability is determined as the question-and-answer result for media data and text information.

[0071] In one possible implementation, the M visual mapping sequences in the mapping vector sequence are arranged before the N text mapping vectors. The insertion processing module is used to insert the compressed marker vector between the M visual mapping vectors and the N text mapping vectors in the mapping vector sequence. When obtaining the visual compressed sequence, it is specifically used to perform the following operations:

[0072] If the resolution of the media data is greater than the compression threshold, a compressed vector value S is generated based on the resolution of the media data. S compressed marker vectors are inserted after the M visual mapping vectors in the mapping vector sequence to obtain the visual compression sequence; S is a positive integer; the S compressed marker vectors in the visual compression sequence are located before the N text mapping vectors.

[0073] The compression processing module is used to perform attention processing on the visual compressed sequence. When obtaining the attention result vector, it is specifically used to perform the following operations:

[0074] Based on S compressed label vectors, attention processing is performed on the visual compressed sequence to obtain an attention result vector; the attention result vector includes unit attention vectors corresponding to M visual mapping vectors, S compressed label vectors, and N text mapping vectors respectively.

[0075] One embodiment of this application provides another data processing apparatus, including:

[0076] The sample input module is used to input media data samples, as well as question text and answer labels for the media data samples, into the initial visual compression model;

[0077] The initial encoding module is used to divide the media data sample into at least two units of media data using an initial visual compression model, perform vector mapping on each unit of media data to obtain M visual mapping vectors, and perform vector mapping on the question text corresponding to the media data sample to obtain N text mapping vectors; M and N are both positive integers.

[0078] The initial insertion module is used to combine M visual mapping vectors and N text mapping vectors into a mapping vector sequence. In the mapping vector sequence, the compressed marker vector is inserted between the M visual mapping vectors and the N text mapping vectors to obtain a visual compression sequence.

[0079] The initial compression processing module is used to perform attention processing on the visual compression sequence to obtain an attention result vector. The attention result vector includes unit attention vectors corresponding to M visual mapping vectors, compression tag vectors, and N text mapping vectors.

[0080] The initial compression determination module is used to determine the unit attention vector associated with the compression label vector in the attention result vector as the global compression vector; the global compression vector is used to characterize the global semantics of the M visual mapping vectors, and / or the vector length of the global compression vector is less than the sum of the vector lengths of the M visual mapping vectors;

[0081] The model adjustment module is used to generate prediction results for media data samples based on the global compression vector and the unit attention vector associated with N text mapping vectors. Based on the prediction results and answer labels, the model parameters of the initial visual compression model are adjusted until the initial visual compression model meets the model convergence condition, thus obtaining the target visual compression model. The target visual compression model is used to generate question-and-answer results for media data and text information.

[0082] In one possible implementation, the sample input module is also used to perform the following operations:

[0083] Based on the data distribution information corresponding to the media data samples, multimodal public data with similar data distribution information is obtained. The visual language model is trained based on the multimodal public data to obtain an initial visual compression model that converges in training on multiple modalities.

[0084] In one possible implementation, the prediction result is a label probability distribution; the model adjustment module is used to adjust the model parameters of the initial visual compression model based on the prediction result and the answer label until the initial visual compression model meets the model convergence condition. When the target visual compression model is obtained, it is specifically used to perform the following operations:

[0085] The label probability distribution is generated based on the answer labels. The relative entropy between the label probability distribution and the predicted probability distribution is determined as the model loss value. The model parameters of the initial visual compression model are adjusted based on the model loss value until the initial visual compression model meets the model training conditions, thus obtaining the target visual compression model.

[0086] One embodiment of this application provides a computer device, including: a processor, a memory, and a network interface;

[0087] The processor is connected to a memory and a network interface. The network interface is used to provide data communication functions, and the memory is used to store computer programs. When the computer program is executed by the processor, the computer device performs the method provided in the embodiments of this application.

[0088] One aspect of this application provides a computer-readable storage medium storing a computer program adapted to be loaded and executed by a processor, so that a computer device having the processor performs the method provided in this application.

[0089] One embodiment of this application provides a computer program product comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the method provided in this application embodiment.

[0090] This application embodiment acquires at least two units of media data, which are media data. Each unit of media data is vector-mapped to obtain M visual mapping vectors. Text information related to the media data is acquired and vector-mapped to obtain N text mapping vectors. The M visual mapping vectors and N text mapping vectors are combined into a mapping vector sequence. A compressed marker vector is inserted between the M visual mapping vectors and the N text mapping vectors to obtain a visual compression sequence. Attention processing is performed on the visual compression sequence to obtain an attention result vector. The attention result vector includes unit attention vectors corresponding to the M visual mapping vectors, the compressed marker vectors, and the N text mapping vectors. The unit attention vectors associated with the compressed marker vectors in the attention result vector are determined as global compression vectors. Since the number of compressed tag vectors is much smaller than the number of visual mapping vectors, the length of the global compressed vector is much shorter than the sum of the lengths of the M visual mapping vectors. This allows the massive visual mapping vectors to be compressed into the global compressed vector. When generating question-and-answer results for media data and text information based on the unit attention vector associated with the global compressed vector and the N text mapping vectors, caching costs can be significantly reduced. Furthermore, the context window occupied by the global compressed vector is much smaller than that of the M visual mapping vectors, eliminating the limitations imposed by the model's context window. The global compressed vector can also represent the global semantics of the M visual mapping vectors, enabling the model to better understand global contextual relationships and avoid context loss or forgetting issues, thus improving model performance. Moreover, computing with the shorter global compressed vector and the N text mapping vectors during the inference phase reduces computational costs and improves computational efficiency. Attached Figure Description

[0091] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0092] Figure 1 is a schematic diagram of a network architecture provided in an embodiment of this application;

[0093] Figure 2 is a schematic diagram of a data processing scenario provided in an embodiment of this application;

[0094] Figure 3 is a schematic diagram of a data processing scenario provided in an embodiment of this application;

[0095] Figure 4 is a schematic flowchart of a data processing method provided in an embodiment of this application;

[0096] Figure 5 is a schematic flowchart of a data processing method provided in an embodiment of this application;

[0097] Figure 6 is a schematic diagram of a model structure provided in an embodiment of this application;

[0098] Figure 7 is a schematic diagram of a data processing scenario provided in an embodiment of this application;

[0099] Figure 8 is a schematic flowchart of a data processing method provided in an embodiment of this application;

[0100] Figure 9 is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0101] Figure 10 is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0102] Figure 11 is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0103] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0104] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0105] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0106] Machine Learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0107] The solutions provided in this application relate to machine learning in artificial intelligence, and are specifically illustrated through the following embodiments.

[0108] Please refer to Figure 1, which is a schematic diagram of a network architecture provided in an embodiment of this application. As shown in Figure 1, the network architecture may include a service server 100 and a terminal device 200. There is a communication connection between the terminal device 200 and the service server 100. The communication connection is not limited to a specific method. It can be directly or indirectly connected via wired communication, or directly or indirectly connected via wireless communication, or in other ways. This application does not impose any restrictions on this method.

[0109] As shown in Figure 1, the terminal device 200 can send media data and text information to the business server 100. The business server 100 can build a target visual compression model. The business server 100 can use the target visual compression model to divide the media data into several unit media data (Patch), and perform vector mapping (Tokenization) on each unit media data to obtain M visual mapping vectors. The text information is also vector mapped to obtain N text mapping vectors. For example, the media data can be an image, and the text information can be a question text related to that image.

[0110] The business server 100 can combine M visual mapping vectors and N text mapping vectors into a mapping vector sequence. A compressed token vector is then inserted between the M visual mapping vectors and the N text mapping vectors to obtain a visual compression sequence. The compressed token vector is a special token in the vocabulary of the target visual compression model used to indicate attention processing (also known as causal transformation). The compressed token vector in the visual compression sequence can be located between the M visual mapping vectors and the N text mapping vectors.

[0111] The business server 100 can perform attention processing on the visually compressed sequence to obtain an attention result vector. This vector can include unit attention vectors corresponding to M visual mapping vectors, compressed marker vectors, and N text mapping vectors. The business server 100 can determine the unit attention vectors associated with the compressed marker vectors within the attention result vector as the global compression vector. The global compression vector represents the global semantics of the M visual mapping vectors, and its length is less than the sum of the lengths of the M visual mapping vectors. By performing attention processing on the visually compressed sequence, the compressed marker vectors can learn the global semantics of the M visual mapping vectors, thus compressing the M visual mapping vectors into a global compression vector.

[0112] The business server 100 can generate question-and-answer results for media data and text information based on a global compression vector and a unit attention vector associated with N text mapping vectors, and send the question-and-answer results to the terminal device 200. Taking the field of visual question answering as an example, the text information can be a question text for media data, and the question-and-answer result can be the answer to that question text.

[0113] It is understood that in the specific embodiments of this application, the user (object or player) data involved requires user permission or consent when the above and below embodiments of this application are applied to specific products or technologies, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant regions.

[0114] If this application requires the collection of object data (such as user data), a prompt interface or pop-up window will be displayed before and during the collection process. This prompt interface or pop-up window is used to inform the user that certain data is being collected. The data acquisition steps will only begin after the user confirms the prompt interface or pop-up window; otherwise, the process will end. Furthermore, the acquired user data will be used in reasonable and legal scenarios or for legitimate purposes. Optionally, in scenarios where user data needs to be used but user authorization has not been obtained, authorization can be requested from the user, and the user data can only be used after authorization is granted.

[0115] It's understandable that, since the number of compressed label vectors is much smaller than the number of visual mapping vectors, the length of the global compressed vector is much smaller than the sum of the lengths of the M visual mapping vectors. This allows the massive visual mapping vectors to be compressed into the global compressed vector. Furthermore, the context window occupied by the global compressed vector is much smaller than that of the M visual mapping vectors, eliminating the constraints of the model's context window. The global compressed vector can also represent the global semantics of the M visual mapping vectors, enabling the model to better understand global contextual relationships and avoiding context loss or forgetting issues, thus improving model performance. Moreover, calculating the unit attention vector associated with the N text mapping vectors using the shorter global compressed vector during the inference phase reduces computational cost and improves computational efficiency.

[0116] Please refer to Figure 2, which is a schematic diagram of a data processing scenario provided by an embodiment of this application. As shown in Figure 2, the media data is video data, which may include video frame data X1, ..., video frame data X i ..., video frame data X n Terminal device 200 can send media data and text information to business server 100, and business server 100 can build a target visual compression model.

[0117] The business server 100 can divide each video frame data in the media data using a target visual compression model to obtain several unit media data (Patch). Please also refer to Figure 3, which is a schematic diagram of a data processing scenario provided in an embodiment of this application. As shown in Figure 3, taking video frame data X... i For example, video frame data X i The media data can be divided into M units, which include media data P1, media data P2, ..., media data P... m The business server 100 can perform vector mapping on M units of media data to obtain video frame data X. i The corresponding M visual mapping vectors, video frame data X i The corresponding M visual mapping vectors may include visual mapping vector V1, visual mapping vector V2, ..., visual mapping vector V m The business server 100 can perform vector mapping on text information to obtain N text mapping vectors. These N text mapping vectors can include text mapping vector T1, text mapping vector T2, ..., text mapping vector T... n .

[0118] Business server 100 can transmit video frame data X iThe corresponding M visual mapping vectors and N text mapping vectors are combined to form a mapping vector sequence, which can be represented as {V1, V2, ..., V...} m ,T1,T2,…,T n The business server 100 can insert the compressed marker vector [VoCo] between M visual mapping vectors and N text mapping vectors in the mapping vector sequence to obtain a visual compression sequence, which can be represented as {V1, V2, ..., V...}. m ,[VoCo],T1,T2,…,T n} Wherein, the compressed token vector [VoCo] is a special token in the vocabulary of the target visual compression model used to indicate attention processing. The compressed token vector in the visual compression sequence can be located between M visual mapping vectors and N text mapping vectors. The number of compressed token vectors [VoCo] can be preset based on the resolution of the video frame data, for example, it can be 2, that is, it means that two compressed token vectors can be inserted into the mapping vector sequence corresponding to one video frame data. This embodiment of the application does not impose any limitation.

[0119] As shown in Figure 2, the business server 100 can process video frame data X i Attention processing is performed on the corresponding visual compression sequence to obtain video frame data X. i The corresponding attention result vector can include unit attention vectors corresponding to M visual mapping vectors, compressed tag vectors, and N text mapping vectors. The business server 100 can determine the unit attention vectors associated with the compressed tag vectors in the attention result vector as the global compression vector. For example, it can include the global compression vector VoCo. i,1 and global compression vector VoCo i,2 The visual mapping vector corresponding to each video frame can be compressed into two global compressed vectors through attention processing. The video frame data in the media data can be represented as {X1,…,X...} i ,…,X n The global compression vector corresponding to each video frame can be represented as {[VoCo}. 1,1 VoCo 1,2 ],…,[VoCo i,1 VoCo i,2 ]...,[VoCo n,1 VoCo n,2 ]}. Among them, the global compression vector is used to represent the global semantics of the M visual mapping vectors, and the vector length of the global compression vector is less than the sum of the vector lengths of the M visual mapping vectors.

[0120] The business server 100 can generate question-and-answer results for media data and text information based on the global compression vector of each video frame data and the unit attention vector associated with N text mapping vectors, and send the question-and-answer results to the terminal device 200. Taking the field of visual content recognition as an example, the text information can be a set of video tags corresponding to the media data and prompt words used to indicate the prediction of video tags, and the question-and-answer results can be the video tags to which the media data belongs.

[0121] The embodiments of this application can be applied to multiple fields and scenarios. In the field of image description, the question-and-answer result can be descriptive text for media data, providing a natural language description for the image. In the field of vision question-and-answer, the question-and-answer result can answer questions with textual information. In document understanding, the question-and-answer result can process document images in business image data, perform optical character recognition (OCR), and understand the document content.

[0122] When processing video data, this embodiment of the application obtains a global compressed vector by compressing the visual mapping vector corresponding to each video frame. The global compressed vector can be composed of a global compressed sequence based on the display order of the video frame data. The global compressed sequence contains the temporal relationship between video frames, thereby capturing the temporal dependency and event development relationship between visual mapping vectors through the global compressed sequence to adapt to time input and compress visual mapping vectors. Therefore, feature interaction through the unit attention vector associated with the global compressed sequence and the text mapping vector can better understand the temporal relationship between each video frame, simplify the video data understanding process, improve the accuracy of video understanding, and reduce computational costs.

[0123] Please refer to Figure 4, which is a schematic flowchart of a data processing method provided in an embodiment of this application. This data processing method can be executed by a computer device, which can be the business server 100 shown in Figure 1. The following description will take the execution of this data processing method by a computer device as an example. The data processing method may include at least the following steps S101-S106:

[0124] Step S101: Obtain at least two units of media data, perform vector mapping on each unit of media data to obtain M visual mapping vectors, where M is a positive integer, and at least two units of media data belong to media data.

[0125] Specifically, computer equipment can divide media data into several unit media data, and perform vector mapping (mapping processing) on ​​each unit media data to obtain M visual mapping vectors. The M visual mapping vectors can be represented as V = {V1, V2, ..., V...}m The vector mapping performed on unit media data can refer to visual encoding. Media data can be video data, and unit media data can be video frame data within video data.

[0126] Step S102: Obtain text information, perform vector mapping on the text information to obtain N text mapping vectors, where N is a positive integer;

[0127] Computer devices can perform vector mapping on text information, resulting in N text mapping vectors. These N text mapping vectors can be represented as T = {T1, T2, ..., T...}. n For example, media data can be an image, and text information can be a question text about that image. The vector mapping of text information can be text encoding. Computer devices can use a tokenizer to split and encode text information into a sequence of tokens. The computer device can then convert the sequence of tokens corresponding to the text information into a vector representation, resulting in N text mapping vectors.

[0128] Step S103: Combine the M visual mapping vectors and N text mapping vectors into a mapping vector sequence. Insert the compressed marker vector between the M visual mapping vectors and the N text mapping vectors in the mapping vector sequence to obtain a visual compression sequence.

[0129] Specifically, a computer device can combine M visual mapping vectors and N text mapping vectors into a mapping vector sequence, which can be represented as {V1, V2, ..., V...}. m ,T1,T2,…,T n The computer device inserts compressed marker vectors between M visual mapping vectors and N text mapping vectors in the mapping vector sequence to obtain a visual compression sequence X, which can be represented as X = {V1, V2, ..., V...}. m ,[VoCo],T1,T2,…,T n}. Here, [VoCo] is the compressed token vector, which is a special token in the vocabulary of the target visual compression model used to indicate attention processing in the causal transformation process. The compressed token vector in the visual compression sequence can be located between M visual mapping vectors and N text mapping vectors.

[0130] Step S104: Perform attention processing on the visual compression sequence to obtain an attention result vector. The attention result vector includes unit attention vectors corresponding to M visual mapping vectors, compressed label vectors, and N text mapping vectors.

[0131] Specifically, the computer device can obtain a mask matrix based on the compressed marker vector, and then perform attention processing on the visually compressed sequence using the mask matrix (e.g., masked attention processing) to obtain the attention result vector. The mask matrix is ​​used to shield the M visually mapped vectors from the attention of the N text mapping vectors in the visually compressed sequence during the attention processing. During this process, the M visually mapped vectors are isolated from the N text mapping vectors, allowing the visually mapped vectors to receive continuous attention from the compressed marker vector, thus achieving compression from the visually mapped vectors to the global compressed vector. The attention result vector includes the M visually mapped vectors, the compressed marker vector, and their corresponding unit attention vectors.

[0132] Step S105: The unit attention vector associated with the compressed label vector in the attention result vector is determined as the global compression vector. The global compression vector is used to characterize the global semantics of the M visual mapping vectors, and / or the vector length of the global compression vector is less than the sum of the vector lengths of the M visual mapping vectors.

[0133] Specifically, the computer device can determine the unit attention vector associated with the compressed label vector in the attention result vector as the global compressed vector. The global compressed vector is used to represent the global semantics of the M visual mapping vectors, and the vector length of the global compressed vector is less than the sum of the vector lengths of the M visual mapping vectors.

[0134] Step S106: Based on the global compression vector and the unit attention vector associated with N text mapping vectors, generate question-and-answer results for media data and text information.

[0135] Specifically, the computer device can input a unit attention vector, which is associated with a global compression vector and N text mapping vectors, into the feature understanding layer. In the feature understanding layer, a prediction probability for the media data is generated using the unit attention vector associated with the global compression vector and the N text mapping vectors. The prediction result with the highest prediction probability is determined as the question-and-answer result for the media data and text information. The process of generating question-and-answer results for media data based on the global compression vector and N text mapping vectors is the same as the process of understanding image tokens and text tokens using Visual Language Models (LVLMs) or Large Language Models (LLMs), and will not be elaborated further in this embodiment.

[0136] This application embodiment acquires at least two units of media data, which are media data. Each unit of media data is vector-mapped to obtain M visual mapping vectors. Text information related to the media data is acquired and vector-mapped to obtain N text mapping vectors. The M visual mapping vectors and N text mapping vectors are combined into a mapping vector sequence. A compressed marker vector is inserted between the M visual mapping vectors and the N text mapping vectors to obtain a visual compression sequence. Attention processing is performed on the visual compression sequence to obtain an attention result vector. The attention result vector includes unit attention vectors corresponding to the M visual mapping vectors, the compressed marker vectors, and the N text mapping vectors. The unit attention vectors associated with the compressed marker vectors in the attention result vector are determined as global compression vectors. Since the number of compressed tag vectors is much smaller than the number of visual mapping vectors, the length of the global compressed vector is much shorter than the sum of the lengths of the M visual mapping vectors. This allows the massive visual mapping vectors to be compressed into the global compressed vector. When generating question-and-answer results for media data and text information based on the unit attention vector associated with the global compressed vector and the N text mapping vectors, caching costs can be significantly reduced. Furthermore, the context window occupied by the global compressed vector is much smaller than that of the M visual mapping vectors, eliminating the limitations imposed by the model's context window. The global compressed vector can also represent the global semantics of the M visual mapping vectors, enabling the model to better understand global contextual relationships and avoid context loss or forgetting issues, thus improving model performance. Moreover, computing with the shorter global compressed vector and the N text mapping vectors during the inference phase reduces computational costs and improves computational efficiency.

[0137] Please refer to Figure 5, which is a schematic flowchart of a data processing method provided in an embodiment of this application. This data processing method can be executed by a computer device, which can be the business server 100 shown in Figure 1. The following description will use the execution of this data processing method by a computer device as an example. The data processing method may include at least the following steps S201-S207:

[0138] Step S201: Obtain at least two units of media data, perform vector mapping on each unit of media data to obtain M visual mapping vectors, where M is a positive integer, and at least two units of media data belong to media data;

[0139] Specifically, the computer device can divide the media data into several unit media data, and perform vector mapping on each unit media data to obtain M visual mapping vectors. Please refer to Figure 6, which is a schematic diagram of a model structure proposed in this embodiment. As shown in Figure 6, the target visual compression model (VoCo-LLaMA) includes an attention mask layer and a feature understanding layer. The feature understanding layer may include pooling layers, attention layers, normalization layers, and output layers used in LVLMs or LLMs for feature fusion and result output. The feature understanding layer can be used to generate a text sequence describing the image; this will not be elaborated further in this embodiment.

[0140] Media data can be video data, and a unit of media data can be a video frame. In the target visual compression model, the computer device can divide the media data into several units, i.e., several video frames. The computer device can perform image segmentation on the video frame data. The image segmentation method can be determined based on resolution. For example, if the size of the video frame data is 48 pixels × 48 pixels, the computer device can divide the video frame data into 9 unit image data. These 9 unit image data can include unit image data 1, unit image data 2, unit image data 3, ..., unit image data 9, and each unit image data can have a ratio of 16 pixels × 16 pixels. The computer device can embed the 9 unit image data into vectors of the same size using an encoder, obtaining a unit feature vector corresponding to each unit image data. The computer device can add a vector containing positional information to the unit feature vector, that is, it can perform positional encoding on the positional information of each unit image data in the sample image data, obtaining a positional feature vector corresponding to each unit image data. The computer device can concatenate the unit feature vector and the positional feature vector to obtain a visual mapping vector.

[0141] Step S202: Obtain text information, perform vector mapping on the text information to obtain N text mapping vectors, where N is a positive integer;

[0142] Specifically, text information is vector-mapped to obtain N text mapping vectors. Taking text information vector mapping as an example, the process of obtaining N text mapping vectors can include: inputting the text information into a target visual compression model; in the target visual compression model, the text information is split into sub-words to obtain a sequence of words corresponding to the text information; the sub-word sequence includes N words, where a word in the sub-word sequence refers to the smallest basic unit obtained after splitting the text information; each word in the sub-word sequence is mapped to obtain N text mapping vectors.

[0143] A computer device can input text information into a target visual compression model, perform text tokenization on the text information, and obtain the corresponding token sequence. For example, if the text information is "Let's do tokenization!", the corresponding token sequence can be represented as [Let,'s,do,token,ization]. Here, [Let,'s], [do], [token], and [ization] are all tokens. A token in the token sequence refers to the smallest basic unit obtained after the text information is tokenized. The computer device can perform mapping processing on each token in the token sequence to obtain N text mapping vectors. The text tokenization method can be word-based, character-based, or subword-based; this embodiment does not impose any limitations. The computer device can convert the token sequence corresponding to the text information into a vector representation to obtain N text mapping vectors.

[0144] Step S203: Combine the M visual mapping vectors and N text mapping vectors into a mapping vector sequence, and insert the compressed marker vector between the M visual mapping vectors and the N text mapping vectors in the mapping vector sequence to obtain a visual compression sequence;

[0145] Specifically, the computer device can combine M visual mapping vectors and N text mapping vectors into a mapping vector sequence. A compressed marker vector [VoCo] is then inserted between the M visual mapping vectors and the N text mapping vectors to obtain a visual compression sequence. The mapping vector sequence can be represented as {V1, V2, ..., V...} m ,T1,T2,…,T n The visual compression sequence can be represented as {V1, V2, ..., V}. m ,[VoCo],T1,T2,…,T n}

[0146] Optionally, the number of compression marker vectors [VoCo] can be preset based on the resolution of the video frame data, for example, it can be 2, meaning that two compression marker vectors can be inserted into the mapping vector sequence corresponding to one video frame data. This embodiment does not impose such a limitation. If the resolution of the media data is greater than the compression threshold, the computer device can generate a compression vector value S based on the resolution of the media data, and insert S compression marker vectors after the arrangement of M visual mapping vectors in the mapping vector sequence to obtain a visual compression sequence. The S compression marker vectors in the visual compression sequence are located before the N text mapping vectors. If the compression vector value S is 2, the visual compression sequence can be represented as {V1, V2, ..., V...} m ,[VoCo1],[VoCo2],T1,T2,…,T n}

[0147] Step S204: Input the visual compression sequence into the target visual compression model; the target visual compression model includes an attention mask layer; in the attention mask layer, the visual compression sequence is multiplied by the query parameter matrix to obtain the query matrix, the visual compression sequence is multiplied by the key parameter matrix to obtain the key matrix, and the visual compression sequence is multiplied by the value parameter matrix to obtain the value matrix.

[0148] Specifically, the computer device can input a visually compressed sequence into the attention mask layer. The visually compressed sequence X can be represented as X = {V1, V2, ..., V...} m ,[VoCo],T1,T2,…,T n The attention mask layer can include a query parameter matrix W. Q Key parameter matrix W K Sum parameter matrix W V Query parameter matrix W Q Key parameter matrix W K Sum parameter matrix W V Both are matrices composed of learnable parameters from the attention mask layer, and the query parameter matrix W Q Key parameter matrix W K Sum parameter matrix W V The matrix dimension can be smaller than the length of the visual compressed sequence.

[0149] The computer device performs a dot product operation between the visual compressed sequence X and the query parameter matrix WQ to obtain the query matrix Q, and then performs a dot product operation between the visual compressed sequence X and the key parameter matrix WQ. K Perform a dot product operation to obtain the key matrix K, and then combine the visual compressed sequence X with the value parameter matrix W. V Performing a dot product operation yields the value matrix V, as shown in formula (1): Q,K,V=[W Q WK W V Formula (1)

[0150] Step S205: Perform a dot product operation on the transpose of the query matrix and the key matrix to obtain the attention score matrix. Generate a masking matrix based on the square root of the dimension value of the key matrix and the attention score matrix to obtain the causal matrix. The causal matrix is ​​a lower triangular matrix. Obtain the masking matrix based on the compressed marker vector and the masking matrix. Perform a dot product operation on the masking matrix, the causal matrix, and the masking matrix to obtain the masking score matrix. Normalize the masking score matrix to obtain the normalized matrix. Perform a dot product operation on the normalized matrix and the value matrix to obtain the unit attention vectors corresponding to the M visual mapping vectors, the compressed marker vectors, and the N text mapping vectors. Determine the unit attention vectors corresponding to the M visual mapping vectors, the compressed marker vectors, and the N text mapping vectors as the attention result vectors. The masking matrix is ​​specifically used to mask the elements in the masking matrix that represent the N text mapping vectors focusing on the M visual mapping vectors.

[0151] Specifically, the computer device can perform a dot product operation on the query matrix Q and the transpose of the key matrix KT to obtain the attention score matrix. This is based on the dimension value d of the key matrix K. K square root The attention score matrix generates the masking matrix, and the computer device can obtain the causal matrix M. 1 The causal matrix can be a matrix used in attention processing (also known as causal transformation processing) to ensure the temporal causality of the generation process. For example, it can be a lower triangular matrix. Taking a visual compression sequence with a dimension of 4×4 as an example, the causal matrix M... 1 It can be shown in formula (2):

[0152] Computer devices can obtain the mask matrix M based on the compressed tag vector [VoCo] and the matrix to be masked. 2 Specifically, the computer device can generate an initial matrix of M+N+S rows and M+N+S columns based on the compressed marker vectors. In the i-th row and j-th column of the initial matrix, if i belongs to the row number corresponding to the N text mapping vectors in the matrix to be masked, and j belongs to the column number corresponding to the M visual mapping vectors in the matrix to be masked, then the element in the i-th row and j-th column of the initial matrix is ​​set as the first mask value. The first mask value can be False or 0, used to represent the padding value. Where S is the number of compressed marker vectors [VoCo].

[0153] In the initial matrix, if i belongs to the row number of the N text mapping vectors in the matrix to be masked, and j belongs to the column number of the compressed tag vector [VoCo] in the matrix to be masked, then the element in the initial matrix is ​​set as the second mask value. The second mask value can be True or 1, representing the true value. A computer device can set all elements in the initial matrix except those with the first mask value as the second mask value to obtain the mask matrix. The mask matrix M... 2 This is used to mask the attention of N text mapping vectors to M visual mapping vectors during attention calculation. For the mask matrix M... 2 The element M in the i-th row and j-th column i( The above process can be shown in formula (3):

[0154] Where [VoCo] represents the compressed marker vector, V represents the visual mapping vector, and M is the mask matrix. 2 ∈R (ABCBD) × (ABCBD) S is the value of the compressed vector, that is, the number of compressed marker vectors [VoCo], and R is the set of real numbers.

[0155] The computer device will mask the matrix M to be masked. 1 Causal matrix M 2 The dot product operation is performed with the mask matrix to obtain the mask fraction matrix. The mask fraction matrix is ​​then normalized (softmax) to obtain the normalized matrix. The dot product operation is performed between the normalized matrix and the value matrix to obtain the attention result vector Z. The process can be shown in formula (4):

[0156] The attention result vector includes unit attention vectors corresponding to M visual mapping vectors and compressed label vectors respectively. The mask score matrix is ​​normalized (softmax) to obtain a normalized matrix that can include the attention weights corresponding to M visual mapping vectors and compressed label vectors respectively. The unit attention vector can be the dot product of the attention weights corresponding to M visual mapping vectors and compressed label vectors with the value vector, which can be used to represent the correlation between the visual mapping vector or compressed label vector in the visual compression sequence.

[0157] Please also refer to Figure 7, which is a schematic diagram of a data processing scenario proposed in an embodiment of this application. As shown in Figure 7, by compressing the visual sequence {V1,V2,…,V…} mM1 V m [VoCo],T1,…,T n Attention processing is performed using the causal matrix M. 1This ensures that preceding tokens will not pay attention to tokens following them, using the mask matrix M. 2 This technique can be used to mask the attention of N text mapping vectors to M visual mapping vectors in a visually compressed sequence during attention processing, thereby isolating the M visual mapping vectors from the N text mapping vectors in the visually compressed sequence. By using a mask matrix M to set the attention weights between the text mapping vectors and the visual mapping vectors to False, the text mapping vectors can be effectively "isolated" from the visual mapping vectors, allowing the visual mapping vectors to receive continuous attention from the compressed marker vector [VoCo], thus achieving compression from the visual mapping vectors to the global compressed vector.

[0158] It can be understood that the above process can also be called causal transformation processing. Causal transformation processing introduces causal masking, i.e., the aforementioned causal matrix M. 1 To restrict the direction of information flow, this ensures that the model generates output using only historical or current inputs, preventing the leakage of future information. For example, in attention processing, the attention calculation at each position can only access the sequence information to its left (past), ensuring the temporal causality of the generation process. Furthermore, this embodiment introduces a mask matrix M. 2 By shielding the N text mapping vectors in the visual compression sequence from the M visual mapping vectors during the attention processing, the text mapping vectors can be effectively "isolated" from the visual mapping vectors, allowing the visual mapping vectors to receive continuous attention from the compressed label vector [VoCo]. This enhances the model's ability to model causal relationships, enabling the compressed label vector to learn the global features of the visual mapping vectors and achieve compression of the visual mapping vectors.

[0159] Step S206: The unit attention vector associated with the compressed label vector in the attention result vector is determined as the global compression vector. The global compression vector is used to characterize the global semantics of the M visual mapping vectors, and / or the vector length of the global compression vector is less than the sum of the vector lengths of the M visual mapping vectors.

[0160] Specifically, the computer device can determine the unit attention vector associated with the compressed label vector in the attention result vector as the global compressed vector. For example, the attention result vector can be represented as {A1, A2, ..., A...} m A [VOPO] ,B1,B2,…,B C}, where A1, A2, ..., A m A [VOPO] All are unit attention vectors, A1, A2, ..., A m Let A be the unit attention vector corresponding to each of the M visual mapping vectors. [VOPO]This represents the unit attention vector corresponding to the compressed label vector. B1, B2, ..., B C Let A be the unit attention vector corresponding to each of the N text mapping vectors. Therefore, the computer device can assign A to the attention result vector. [VOPO] It is determined to be a global compression vector.

[0161] When a computer device inserts multiple compressed label vectors into a mapping vector sequence, the attention result vector obtained by performing attention masking on the resulting visual compressed sequence can include unit attention vectors associated with the multiple compressed label vectors. For example, if the computer device inserts two compressed label vectors into the mapping vector sequence, the attention result vector obtained by performing attention processing on the resulting visual compressed sequence can be represented as {A1, A2, ..., A...}. m A [VOPO],1 A [VOPO],2 ,B1,B2,…,B C}, where A [VOPO],1 A [VOPO],2 For the unit attention vector associated with the two inserted compressed label vectors, the computer device can assign A to the attention result vector. [VOPO],1 A [VOPO],2 All are determined to be global compression vectors.

[0162] It is understood that the degree of compression can be flexibly adjusted by the number of compressed marker vectors inserted in this application embodiment in order to retain more comprehensive feature information. At the same time, the number of compressed marker vectors is much smaller than the number of visual mapping vectors. Therefore, by using attention processing, the compressed marker vectors learn the global features of the visual mapping vectors, thereby achieving compression of the visual mapping vectors.

[0163] Step S207: Input the global compression vector and the unit attention vector associated with N text mapping vectors into the target visual compression model; the target visual compression model includes a feature understanding layer; based on the feature understanding layer, generate the prediction probability for media data through the unit attention vector associated with the global compression vector and N text mapping vectors, and determine the prediction result with the highest prediction probability as the question-and-answer result for media data and text information.

[0164] Specifically, the computer device can input a global compression vector and a unit attention vector associated with N text mapping vectors into the feature understanding layer. The feature understanding layer can be a Visual Language Model (LVLM) or a Large Language Model (LLM). The unit attention vector associated with the N text mapping vectors can refer to the attention result vector {A1, A2, ..., A...}. m A [VOPO],1 A [VOPO],2 ,B1,B2,…,B CIn}, B1, B2, ..., B C Through the feature understanding layer, the computer device can generate prediction probabilities for media data and determine the prediction result with the highest prediction probability as the question-and-answer result for the media data. The process of generating question-and-answer results for media data and text information based on global compressed vectors and N text mapping vectors can be the same as the process of understanding image tokens and text tokens by visual language models (LVLMs) or large language models (LLMs), and will not be described in detail here.

[0165] It is understandable that when processing repetitive media data input, the output of the activation function in the Transformer for the global compression vector corresponding to the repetitive media data can be stored. The global compression vector is shorter and more densely represented, which can improve computational efficiency when processing the activation function. The target visual compression model can cache and reuse the global compression vector. Compared with the visual mapping vector, the computer device can reuse the output of the activation function, which can improve caching efficiency.

[0166] Optionally, if the media data includes T video frames, where T is a positive integer, the T video frames can be represented as {X1,…,X...} i Given the media data ,…,XT}, the computer device can divide the media data into units of media data corresponding to each video frame, perform vector mapping on these units of media data to obtain a visual mapping vector for each video frame, perform vector mapping on the text information corresponding to the media data to obtain a text mapping vector for the media data, insert a compression marker vector into the mapping vector sequence corresponding to each video frame to obtain a visual compression sequence corresponding to each video frame, and perform attention processing on the visual compression sequence to obtain a global compression vector corresponding to each video frame. The process of calculating the global compression vector through attention masking can be the same as the process in steps S204 to S205 above. Similarly, for each video frame, the computer device can obtain the unit attention vectors associated with the N text mapping vectors corresponding to each video frame in the same manner as in steps S204 to S205 above. Based on the display order of the T video frames in the media data, the computer device can arrange the T global compression vectors to obtain a global compression sequence, and arrange the unit attention vectors associated with the N text mapping vectors corresponding to each video frame to obtain a text attention sequence. The global compression sequence F can be represented as F = {VoCo1, VoCo2, ..., VoCo...} n Computer devices can generate question-and-answer results for media data and textual information based on global compressed sequences and textual attention sequences.

[0167] It is understandable that when the input media data includes T video frames, the target visual compression model can obtain a global compression vector by compressing the visual mapping vector corresponding to each video frame. The global compression vector can be composed of a global compression sequence based on the display order of the video frame data. The global compression sequence contains the temporal relationship between video frames, so the temporal dependency and event development relationship between visual mapping vectors can be captured through the global compression sequence to adapt to the temporal input and compress the visual mapping vector. Therefore, feature interaction between the global compression sequence and the text mapping vector can better understand the temporal relationship between each video frame, simplify the video data understanding process, improve the accuracy of video understanding, and reduce computational costs.

[0168] This application embodiment obtains a visual compression sequence by inserting compressed marker vectors between M visual mapping vectors and N text mapping vectors in the mapping vector sequence. A mask matrix is ​​then obtained. This mask matrix can mask the N text mapping vectors in the visual compression sequence during attention processing, isolating the M visual mapping vectors from the N text mapping vectors. This ensures that the visual mapping vectors receive continuous attention from the compressed marker vectors, achieving compression from visual mapping vectors to a global compressed vector. Since the number of compressed marker vectors is much smaller than the number of visual mapping vectors, the length of the global compressed vector is much smaller than the sum of the lengths of the M visual mapping vectors. This allows for the compression of the large visual mapping vectors into a global compressed vector. When generating question-and-answer results for media data and text information based on the global compressed vector and the N text mapping vectors, caching costs can be significantly reduced. Furthermore, the context window occupied by the global compressed vector is much smaller than that of the M visual mapping vectors, eliminating the limitations of the model's context window. The global compressed vector can also represent the global semantics of the M visual mapping vectors, enabling the model to better understand the global contextual relationships and avoiding context loss or forgetting issues, thus improving model performance. Furthermore, by using a globally compressed vector with a shorter vector length and N text mapping vectors for calculation during the inference phase, computational costs can be reduced and computational efficiency can be improved.

[0169] When processing repetitive media data input, the output of the activation function in the Transformer can be stored for the global compressed vector corresponding to the repetitive media data. The global compressed vector is shorter and denser, improving computational efficiency during activation function processing. The target visual compression model can cache and reuse the global compressed vector. Compared to visual mapping vectors, computer devices can reuse the output of the activation function, improving caching efficiency. When processing video data input, a global compressed vector can be obtained by compressing the visual mapping vector corresponding to each video frame. This global compressed vector can be composed of a global compressed sequence based on the display order of the video frames. This global compressed sequence contains the temporal relationships between video frames, allowing the capture of temporal dependencies and event progression relationships between visual mapping vectors. This adapts to temporal input and compresses the visual mapping vectors. Therefore, feature interaction between the global compressed sequence and the text mapping vector allows for a better understanding of the temporal relationships between each video frame, simplifying the video data understanding process, improving the accuracy of video understanding, and reducing computational costs.

[0170] Please refer to Figure 8, which is a schematic flowchart of a data processing method provided in an embodiment of this application. This data processing method can be executed by a computer device, which can be the business server 100 shown in Figure 1. The following description will take the execution of this data processing method by a computer device as an example. The data processing method may include at least the following steps S301-S306:

[0171] Step S301: Input the media data sample, along with the question text and answer labels for the media data sample, into the initial visual compression model;

[0172] Specifically, the computer equipment can use a pre-trained visual language model as the initial visual compression model. The computation can then leverage the data distribution information corresponding to the media data samples to obtain multimodal public data with similar characteristics to the data distribution information. Based on this multimodal public data, the visual language model is trained to obtain an initial visual compression model that converges across multiple modalities. This initial visual compression model can include the visual encoder (which encodes the input media data samples into visual mapping vectors) from the CLIP-ViT-L (Contrastive Language-Image Pre-training Vision Transformer) model and the text encoder from Vicuna-7B (Vicuna-7Billion, a large language model based on the Transformer architecture). Without introducing compressed tag vectors or global compression vectors, the visual encoder and language model are kept in a frozen state (preserving parameter weights). The visual encoder and text encoder are aligned using the CC3M dataset (Conceptual Captions 3M, a large-scale image description dataset) filtered by LLaVA (Large Language Visual Answering, a vision-language model).

[0173] Computer devices can input media data samples, along with question text and answer labels for those media data samples, into an initial visual compression model. The media data samples, along with the question text and answer labels for those media data samples, can be question-answering pairs (QA pairs) contained in the WebVid dataset (a video understanding dataset typically used to train and evaluate the performance of video understanding models) or question-answering pairs generated by Video-ChatGPT (Video-Chat Generative Pre-Trained).

[0174] It is understood that the embodiments of this application train the initial visual compression model by generating question-answer pairs on a wide range of datasets. During the training process, since the original visual mapping vectors are input, instead of hard compression of the original images or original tokens by other models, the model can understand the mapping relationship of visual mapping vectors on multiple modalities, thereby adapting to the compression of visual mapping vectors on multiple modalities and improving the versatility of the model.

[0175] Step S302: Divide the media data sample into at least two units of media data using the initial visual compression model, perform vector mapping on each unit of media data to obtain M visual mapping vectors, and perform vector mapping on the question text corresponding to the media data sample to obtain N text mapping vectors; M and N are both positive integers.

[0176] Specifically, the computer device can divide the media data sample into several unit media data using an initial visual compression model. The computer device can then perform vector mapping on each unit media data to obtain M visual mapping vectors. Furthermore, it can perform vector mapping on the question text corresponding to the media data sample to obtain N text mapping vectors. The vector mapping process can be found in the detailed description of steps S201 and S202 in the embodiment corresponding to Figure 5 above, and will not be repeated here.

[0177] Step S303: Combine the M visual mapping vectors and N text mapping vectors into a mapping vector sequence, and insert the compressed marker vector between the M visual mapping vectors and the N text mapping vectors in the mapping vector sequence to obtain a visual compression sequence;

[0178] Specifically, the computer device can combine M visual mapping vectors and N text mapping vectors into a mapping vector sequence, and insert compression token vectors into this sequence to obtain a visual compression sequence. The compression token vectors are special tokens in the vocabulary of the initial visual compression model used to indicate causal transformation processing. The compression token vectors in the visual compression sequence are located between the M visual mapping vectors and the N text mapping vectors.

[0179] Step S304: Perform attention processing on the visual compression sequence to obtain an attention result vector. The attention result vector includes unit attention vectors corresponding to M visual mapping vectors, compressed label vectors, and N text mapping vectors.

[0180] Specifically, the computer device can perform causal transformation processing on the visual compression sequence based on the compressed label vector to obtain a global compressed vector. The global compressed vector represents the global semantics of the M visual mapping vectors, and its length is less than the sum of the lengths of the M visual mapping vectors. The causal transformation processing can be implemented based on the causal matrix and mask matrix in the Causal Transformer. The process can be found in the detailed description of steps S203 to S204 of the embodiment corresponding to Figure 5 above, and will not be repeated here.

[0181] Step S305: The unit attention vector associated with the compressed label vector in the attention result vector is determined as the global compression vector; the global compression vector is used to characterize the global semantics of the M visual mapping vectors, and / or the vector length of the global compression vector is less than the sum of the vector lengths of the M visual mapping vectors;

[0182] For details, please refer to the specific description of step S206 in the embodiment corresponding to Figure 5 above, which will not be repeated here.

[0183] Step S306: Based on the global compression vector and the unit attention vector associated with N text mapping vectors, generate the prediction result corresponding to the media data sample. Based on the prediction result and answer label, adjust the model parameters of the initial visual compression model until the initial visual compression model meets the model convergence condition, and obtain the target visual compression model. The target visual compression model is used to generate question and answer results for media data and text information.

[0184] Specifically, the computer device can input a global compressed vector and N text mapping vectors into the feature understanding layer. Taking a classification task as an example, the feature understanding layer can generate predicted probabilities for media data on multiple classification labels (each predicted probability can correspond to a predicted result y) through the unit attention vector associated with the global compressed vector and the N text mapping vectors, thus obtaining the predicted probability distribution p. VOPO (y|F,T). Computer devices can generate a label probability distribution p based on the answer labels. QU The relative entropy between the label probability distribution and the predicted probability distribution is determined as the model loss value L. W The process can be shown in formula (5): L W =E T [D KZ (p QU ||p VOPO (y|F,T)] Formula (5)

[0185] Where ET refers to the expected mean of the text information, and D... KZ Used to represent the label probability distribution p QU With the predicted probability distribution p VOPO The relative entropy KL between (y|V,T). F represents the sequence of globally compressed vectors, and T represents the text mapping vector. Computer devices can use the model loss value L... W The initial visual compression model is adjusted in terms of model parameters until it meets the model training conditions, thus obtaining the target visual compression model (VoCo-LLaMA). The model training conditions can be the model loss value L... WStable, no longer significantly decreasing after multiple consecutive epochs (training cycles). Or, performance metrics (such as accuracy, precision, recall, etc.) no longer improve or reach the predetermined target, etc., which are not limited in the embodiments of this application.

[0186] It is understood that during the visual instruction adjustment phase, the global compression vector can directly interact with the text mapping vector, and the text mapping vector only interacts with the global compression vector. Therefore, during the model parameter adjustment process, the global compression vector can learn and establish the interaction path between the two modalities with the text mapping vector. There is no need to design a special alignment module to achieve modal alignment. Through the standard visual instruction adjustment learning process, the training cost of the embodiments of this application is reduced.

[0187] In the experimental phase of this application, experiments were conducted on common visual understanding benchmarks to evaluate the compression effect of visual mapping vectors. Test datasets in the image domain may include GQA (General Question Answering), MMB (MMBench), MME (Multimodal Large Language Model Evaluation), POPE (a benchmark specifically designed to evaluate whether multimodal large language models exhibit hallucination problems in visual question answering tasks), SEED-Bench (a benchmark for multimodal large language models), SQAI (an image-based version of ScienceQA), and VQAv2. Test datasets in the video domain can include MSVD QA (a video QA dataset containing 1,970 video clips and 50,505 QA pairs), MSRVTT-QA (a large-scale video QA dataset containing 10,000 videos and 243,000 QA pairs), MSR-VTT dataset, and ActivityNet-QA (a fully annotated video QA dataset containing 58,000 QA pairs derived from 5,800 complex network videos in the ActivityNet dataset).

[0188] By observing the target visual compression model before and after visual compression (i.e., before and after using compressed token vectors and global compression vectors), the compression retention rate and compression ratio of the target visual compression model were tested on these image and video understanding benchmarks. It can be observed that even at a high compression ratio of 576x, the performance loss achieved by the embodiments of this application is significantly lower than the compression ratio by two orders of magnitude. An average compression retention rate of 83.7% was achieved on widely used benchmarks. On MMBeen (an evaluation benchmark for multimodal large language models), a compression retention rate of 91.3% was achieved. The results show that the target visual compression model can effectively compress visual tokens. The benefits of saved context length and improved computational efficiency far outweigh the performance loss caused by compression.

[0189] Furthermore, this embodiment allows setting the number of compressed marker vectors inserted into the mapping vector sequence. This embodiment gradually increases the number of compressed marker vectors on widely used benchmarks, i.e., increases the number of global compressed vectors. As the number of global compressed vectors increases, the overall compression performance of the model shows an upward trend. An average compression retention rate of 89.1% can be achieved with approximately five global compressed vectors, and an average compression retention rate of 95.3% is achieved when the number of global compressed vectors reaches 128. This indicates that when the number of global compressed vectors exceeds 128, the performance loss due to compression is almost negligible. A comparative analysis was conducted on CUDA (Compute Unified Device Architecture) time, FLOPs (Floating Point Operations Per Second), and Key-Value Cache Storage Size during the inference process. Compared with a single-line caching strategy, this embodiment achieves a significant reduction of 69.6% in CUDA time and a significant reduction of 94.8% in FLOPs. Compared with the full caching strategy, the embodiments of this application save 99.8% of cache storage while achieving lower CUDA time and FLOPs.

[0190] The initial visual compression model in this embodiment can consist of a visual encoder from a pre-trained CLIP-ViT-L model and a text encoder from Vicuna-7B. The visual encoder and text encoder are aligned using a filtered CC3M dataset. Question-answer pairs are generated on a broad dataset to train the initial visual compression model. A special token for a predetermined compression tag vector is set in the vocabulary of the initial visual compression model, thereby enabling causal transformation of the visual compression sequence. The visual compression sequence consists of a visual mapping vector corresponding to the media data sample, a compression tag vector, and a text mapping vector corresponding to the question text, resulting in a global compression vector. Prediction results for the media data sample are generated based on the global compression vector and N text mapping vectors. The model parameters of the initial visual compression model are adjusted based on the prediction results and answer labels until the initial visual compression model meets the model convergence condition, resulting in the target visual compression model. The target visual compression model is used to generate question-answer results for the media data. Compressing the large visual mapping vector into a global compression vector, where the global compression vector represents the global semantics of the M visual mapping vectors, allows the model to better understand global contextual relationships and avoids contextual forgetting issues. Furthermore, due to the input of the original visual mapping vector, the model can adapt to the compression of visual mapping vectors across multiple modalities, improving the model's versatility. The vector length of the global compressed vector is much smaller than the sum of the vector lengths of the M visual mapping vectors, which can greatly reduce caching costs and computational costs during the inference phase, thus improving computational efficiency. During the visual instruction adjustment phase, the global compressed vector can directly interact with the text mapping vector, and the text mapping vector only interacts with the global compressed vector. Therefore, during the model parameter adjustment process, the global compressed vector can learn and establish the interaction path between the two modalities with the text mapping vector. There is no need to design a dedicated alignment module to achieve modal alignment; the standard visual instruction adjustment learning process can be used, reducing the training cost of this embodiment.

[0191] Please refer to Figure 9, which is a schematic diagram of the structure of a data processing device provided in an embodiment of this application. As shown in Figure 9, the data processing device 1 includes a visual encoding module 910, a text encoding module 920, an insertion processing module 930, a compression processing module 940, a compression determination module 950, and an output module 960.

[0192] Visual encoding module 910 is used to acquire at least two units of media data, perform vector mapping on each unit of media data, and obtain M visual mapping vectors, where M is a positive integer, and at least two units of media data belong to media data.

[0193] The text encoding module 920 is used to acquire text information, perform vector mapping on the text information, and obtain N text mapping vectors, where N is a positive integer.

[0194] The insertion processing module 930 is used to combine M visual mapping vectors and N text mapping vectors into a mapping vector sequence, and insert a compressed marker vector between the M visual mapping vectors and the N text mapping vectors in the mapping vector sequence to obtain a visual compression sequence.

[0195] The compression processing module 940 is used to perform attention processing on the visual compression sequence to obtain an attention result vector. The attention result vector includes unit attention vectors corresponding to M visual mapping vectors, compressed label vectors, and N text mapping vectors.

[0196] The compression determination module 950 is used to determine the unit attention vector associated with the compression label vector in the attention result vector as the global compression vector. The global compression vector is used to characterize the global semantics of the M visual mapping vectors, and / or the vector length of the global compression vector is less than the sum of the vector lengths of the M visual mapping vectors.

[0197] Output module 960 is used to generate question-and-answer results for media data and text information based on a unit attention vector associated with a global compression vector and N text mapping vectors.

[0198] In one possible implementation, the text encoding module 920 is used to perform vector mapping on the text information. When N text mapping vectors are obtained, it is specifically used to perform the following operations:

[0199] The text information is input into the target visual compression model. In the target visual compression model, the text information is split into words to obtain the word sequence corresponding to the text information. The word sequence includes N words. A word in the word sequence refers to the smallest basic unit obtained after splitting the text information.

[0200] Each word in the word sequence is mapped separately to obtain N text mapping vectors.

[0201] In one possible implementation, the compression processing module 940 performs attention processing on the visual compression sequence to obtain an attention result vector, specifically performing the following operations:

[0202] The visual compression sequence is input into the target visual compression model; the target visual compression model includes an attention mask layer.

[0203] In the attention mask layer, the visual compressed sequence is multiplied by the query parameter matrix to obtain the query matrix, the visual compressed sequence is multiplied by the key parameter matrix to obtain the key matrix, and the visual compressed sequence is multiplied by the value parameter matrix to obtain the value matrix.

[0204] The mask matrix is ​​obtained based on the compressed tag vector. Based on the mask matrix, attention is calculated on the query matrix, key matrix, and value matrix to obtain the unit attention vectors corresponding to M visual mapping vectors, compressed tag vectors, and N text mapping vectors. The unit attention vectors corresponding to the M visual mapping vectors, compressed tag vectors, and N text mapping vectors are determined as the attention result vectors. The mask matrix is ​​used to shield the attention of the N text mapping vectors on the M visual mapping vectors during the attention calculation process.

[0205] In one possible implementation, the compression processing module 940 is used to obtain a mask matrix based on the compressed marker vector, and based on the mask matrix, perform attention calculation on the query matrix, key matrix, and value matrix to obtain unit attention vectors corresponding to M visual mapping vectors and compressed marker vectors respectively. When determining the unit attention vectors corresponding to the M visual mapping vectors and compressed marker vectors as the attention result vectors, it is specifically used to perform the following operations:

[0206] The query matrix and the transpose of the key matrix are multiplied by a dot product to obtain the attention score matrix. The masking matrix is ​​generated based on the square root of the dimension value of the key matrix and the attention score matrix to obtain the causal matrix; the causal matrix is ​​a lower triangular matrix.

[0207] The mask matrix is ​​obtained based on the compressed label vector and the matrix to be masked. The matrix to be masked, the causal matrix, and the mask matrix are multiplied by a dot product to obtain the mask fraction matrix. The mask fraction matrix is ​​normalized to obtain the normalized matrix. The normalized matrix is ​​multiplied by the value matrix to obtain the unit attention vectors corresponding to M visual mapping vectors, compressed label vectors, and N text mapping vectors. The unit attention vectors corresponding to the M visual mapping vectors, compressed label vectors, and N text mapping vectors are determined as the attention result vectors. The mask matrix is ​​specifically used to mask the elements in the matrix to be masked that represent the N text mapping vectors focusing on the M visual mapping vectors.

[0208] In one possible implementation, when the compression processing module 940 obtains the mask matrix based on the compression marker vector and the mask matrix to be masked, it specifically performs the following operations:

[0209] An initial matrix of M+N+1 rows and M+N+1 columns is generated based on the compressed marker vector. In the i-th row and j-th column of the initial matrix, if i belongs to the row number of the N text mapping vectors in the matrix to be masked, and j belongs to the column number of the M visual mapping vectors in the matrix to be masked, then the element in the i-th row and j-th column of the initial matrix is ​​set as the first mask value.

[0210] Set all elements in the initial matrix except those with the first mask value to the second mask value to obtain the mask matrix.

[0211] In one possible implementation, the compression processing module 940 performs attention processing on the visual compression sequence to obtain an attention result vector, specifically performing the following operations:

[0212] The mask matrix is ​​obtained based on the compressed tag vector. Attention processing is then performed on the visual compressed sequence based on the mask matrix to obtain the attention result vector. The mask matrix is ​​used to shield the attention of N text mapping vectors on M visual mapping vectors during the attention processing.

[0213] In one possible implementation, when the output module 960 generates question-and-answer results for media data and text information based on the global compression vector and N text mapping vectors, it specifically performs the following operations:

[0214] The unit attention vector, which is associated with the global compression vector and N text mapping vectors, is input into the target visual compression model; the target visual compression model includes a feature understanding layer;

[0215] Based on the feature understanding layer, a unit attention vector associated with a global compression vector and N text mapping vectors is used to generate a prediction probability for media data. The prediction result with the highest prediction probability is determined as the question-and-answer result for media data and text information.

[0216] In one possible implementation, the M visual mapping sequences in the mapping vector sequence are arranged before the N text mapping vectors; the insertion processing module 930 is used to insert the compressed marker vector between the M visual mapping vectors and the N text mapping vectors in the mapping vector sequence to obtain the visual compressed sequence, specifically by performing the following operations:

[0217] If the resolution of the media data is greater than the compression threshold, a compressed vector value S is generated based on the resolution of the media data. S compressed marker vectors are inserted after the M visual mapping vectors in the mapping vector sequence to obtain the visual compression sequence; S is a positive integer; the S compressed marker vectors in the visual compression sequence are located before the N text mapping vectors.

[0218] The compression processing module 940 is used to perform attention processing on the visual compression sequence. When obtaining the attention result vector, it is specifically used to perform the following operations:

[0219] Based on S compressed label vectors, attention processing is performed on the visual compressed sequence to obtain an attention result vector; the attention result vector includes unit attention vectors corresponding to M visual mapping vectors, S compressed label vectors, and N text mapping vectors respectively.

[0220] This application embodiment obtains a visual compression sequence by inserting compressed marker vectors between M visual mapping vectors and N text mapping vectors in the mapping vector sequence. A mask matrix is ​​then obtained. This mask matrix can mask the N text mapping vectors in the visual compression sequence during attention processing, isolating the M visual mapping vectors from the N text mapping vectors. This ensures that the visual mapping vectors receive continuous attention from the compressed marker vectors, achieving compression from visual mapping vectors to a global compressed vector. Since the number of compressed marker vectors is much smaller than the number of visual mapping vectors, the length of the global compressed vector is much smaller than the sum of the lengths of the M visual mapping vectors. This allows for the compression of the large visual mapping vectors into a global compressed vector. When generating question-and-answer results for media data and text information based on the global compressed vector and the N text mapping vectors, caching costs can be significantly reduced. Furthermore, the context window occupied by the global compressed vector is much smaller than that of the M visual mapping vectors, eliminating the limitations of the model's context window. The global compressed vector can also represent the global semantics of the M visual mapping vectors, enabling the model to better understand the global contextual relationships and avoiding context loss or forgetting issues, thus improving model performance. Furthermore, by using a globally compressed vector with a shorter vector length and N text mapping vectors for calculation during the inference phase, computational costs can be reduced and computational efficiency can be improved.

[0221] When processing repetitive media data input, the output of the activation function in the Transformer can be stored for the global compressed vector corresponding to the repetitive media data. The global compressed vector is shorter and denser, improving computational efficiency during activation function processing. The target visual compression model can cache and reuse the global compressed vector. Compared to visual mapping vectors, computer devices can reuse the output of the activation function, improving caching efficiency. When processing video data input, a global compressed vector can be obtained by compressing the visual mapping vector corresponding to each video frame. This global compressed vector can be composed of a global compressed sequence based on the display order of the video frames. This global compressed sequence contains the temporal relationships between video frames, allowing the capture of temporal dependencies and event progression relationships between visual mapping vectors. This adapts to temporal input and compresses the visual mapping vectors. Therefore, feature interaction between the global compressed sequence and the text mapping vector allows for a better understanding of the temporal relationships between each video frame, simplifying the video data understanding process, improving the accuracy of video understanding, and reducing computational costs.

[0222] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0223] Please refer to Figure 10, which is a second structural schematic diagram of a data processing device provided in an embodiment of this application. As shown in Figure 10, the data processing device 2 includes a sample input module 1010, an initial encoding module 1020, an initial insertion module 1030, an initial compression processing module 1040, an initial compression determination module 1050, and a model adjustment module 1060.

[0224] The sample input module 1010 is used to input media data samples, as well as question text and answer labels for the media data samples, into the initial visual compression model;

[0225] The initial encoding module 1020 is used to divide the media data sample into at least two units of media data through the initial visual compression model, perform vector mapping on each unit of media data to obtain M visual mapping vectors, and perform vector mapping on the question text corresponding to the media data sample to obtain N text mapping vectors; M and N are both positive integers.

[0226] The initial insertion module 1030 is used to combine M visual mapping vectors and N text mapping vectors into a mapping vector sequence, and insert a compressed marker vector between the M visual mapping vectors and the N text mapping vectors in the mapping vector sequence to obtain a visual compression sequence.

[0227] The initial compression processing module 1040 is used to perform attention processing on the visual compression sequence to obtain an attention result vector. The attention result vector includes unit attention vectors corresponding to M visual mapping vectors, compression tag vectors, and N text mapping vectors.

[0228] The initial compression determination module 1050 is used to determine the unit attention vector associated with the compression label vector in the attention result vector as the global compression vector; the global compression vector is used to characterize the global semantics of the M visual mapping vectors, and / or the vector length of the global compression vector is less than the sum of the vector lengths of the M visual mapping vectors;

[0229] The model adjustment module 1060 is used to generate prediction results corresponding to media data samples based on the unit attention vector associated with the global compression vector and N text mapping vectors. Based on the prediction results and answer labels, the model parameters of the initial visual compression model are adjusted until the initial visual compression model meets the model convergence condition, thus obtaining the target visual compression model. The target visual compression model is used to generate question-and-answer results for media data and text information.

[0230] In one possible implementation, the sample input module 1010 is also used to perform the following operations:

[0231] Based on the data distribution information corresponding to the media data samples, multimodal public data with similar data distribution information is obtained. The visual language model is trained based on the multimodal public data to obtain an initial visual compression model that converges in training on multiple modalities.

[0232] In one possible implementation, the prediction result is a label probability distribution; the model adjustment module 1060 is used to adjust the model parameters of the initial visual compression model based on the prediction result and the answer label until the initial visual compression model meets the model convergence condition. When the target visual compression model is obtained, it is specifically used to perform the following operations:

[0233] The label probability distribution is generated based on the answer labels. The relative entropy between the label probability distribution and the predicted probability distribution is determined as the model loss value. The model parameters of the initial visual compression model are adjusted based on the model loss value until the initial visual compression model meets the model training conditions, thus obtaining the target visual compression model.

[0234] The initial visual compression model in this embodiment can consist of a visual encoder from a pre-trained CLIP-ViT-L model and a text encoder from Vicuna-7B. The visual encoder and text encoder are aligned using a filtered CC3M dataset. Question-answer pairs are generated on a broad dataset to train the initial visual compression model. A special token for a predetermined compression tag vector is set in the vocabulary of the initial visual compression model, thereby enabling causal transformation of the visual compression sequence. The visual compression sequence consists of a visual mapping vector corresponding to the media data sample, a compression tag vector, and a text mapping vector corresponding to the question text, resulting in a global compression vector. Prediction results for the media data sample are generated based on the global compression vector and N text mapping vectors. The model parameters of the initial visual compression model are adjusted based on the prediction results and answer labels until the initial visual compression model meets the model convergence condition, resulting in the target visual compression model. The target visual compression model is used to generate question-answer results for the media data. Compressing the large visual mapping vector into a global compression vector, where the global compression vector represents the global semantics of the M visual mapping vectors, allows the model to better understand global contextual relationships and avoids contextual forgetting issues. Furthermore, due to the input of the original visual mapping vector, the model can adapt to the compression of visual mapping vectors across multiple modalities, improving the model's versatility. The vector length of the global compressed vector is much smaller than the sum of the vector lengths of the M visual mapping vectors, which can greatly reduce caching costs and computational costs during the inference phase, thus improving computational efficiency. During the visual instruction adjustment phase, the global compressed vector can directly interact with the text mapping vector, and the text mapping vector only interacts with the global compressed vector. Therefore, during the model parameter adjustment process, the global compressed vector can learn and establish the interaction path between the two modalities with the text mapping vector. There is no need to design a dedicated alignment module to achieve modal alignment; the standard visual instruction adjustment learning process can be used, reducing the training cost of this embodiment.

[0235] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0236] Please refer to Figure 11, which is a schematic diagram of the structure of a computer device provided in an embodiment of this application. As shown in Figure 11, the computer device 1000 may include: a processor 1001, a network interface 1004, and a memory 1005. Furthermore, the computer device 1000 may also include: a user interface 1003, and at least one communication bus 1002. The communication bus 1002 is used to implement communication between these components. The user interface 1003 may include a display screen and a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed RAM memory or a non-volatile memory, such as at least one disk storage device. Optionally, the memory 1005 may also be at least one storage device located remotely from the aforementioned processor 1001. As shown in Figure 11, the memory 1005, as a computer-readable storage medium, may include an operating system, a network communication module, a user interface module, and a device control application.

[0237] In the computer device 1000 shown in Figure 11, the network interface 1004 provides network communication elements; the user interface 1003 is mainly used to provide an input interface for users; and the processor 1001 can be used to call the device control application stored in the memory 1005.

[0238] When the computer device 1000 is used to implement the data processing device 1, it can perform:

[0239] Acquire at least two units of media data, perform vector mapping on each unit of media data, and obtain M visual mapping vectors, where M is a positive integer, and at least two units of media data belong to the media data.

[0240] Obtain text information, perform vector mapping on the text information to obtain N text mapping vectors, where N is a positive integer;

[0241] M visual mapping vectors and N text mapping vectors are combined into a mapping vector sequence. A compressed marker vector is then inserted between the M visual mapping vectors and the N text mapping vectors in the mapping vector sequence to obtain a visual compression sequence.

[0242] Attention processing is applied to the visual compression sequence to obtain an attention result vector, which includes unit attention vectors corresponding to M visual mapping vectors, compressed label vectors, and N text mapping vectors.

[0243] The unit attention vector associated with the compressed label vector in the attention result vector is determined as the global compressed vector. The global compressed vector is used to characterize the global semantics of the M visual mapping vectors, and / or the vector length of the global compressed vector is less than the sum of the vector lengths of the M visual mapping vectors.

[0244] Based on a unit attention vector associated with a global compression vector and N text mapping vectors, question-and-answer results are generated for media data and text information.

[0245] When computer device 1000 is used to implement data processing device 2, it can perform:

[0246] Input the media data sample, along with the question text and answer labels for the media data sample, into the initial visual compression model;

[0247] The media data sample is divided into at least two units of media data using an initial visual compression model. Vector mapping is performed on each unit of media data to obtain M visual mapping vectors. Vector mapping is also performed on the question text corresponding to the media data sample to obtain N text mapping vectors. M and N are both positive integers.

[0248] M visual mapping vectors and N text mapping vectors are combined into a mapping vector sequence. A compressed marker vector is then inserted between the M visual mapping vectors and the N text mapping vectors in the mapping vector sequence to obtain a visual compression sequence.

[0249] Attention processing is applied to the visual compression sequence to obtain an attention result vector, which includes unit attention vectors corresponding to M visual mapping vectors, compressed label vectors, and N text mapping vectors.

[0250] The unit attention vector associated with the compressed label vector in the attention result vector is determined as the global compression vector; the global compression vector is used to characterize the global semantics of the M visual mapping vectors, and / or the vector length of the global compression vector is less than the sum of the vector lengths of the M visual mapping vectors;

[0251] The prediction results for the media data samples are generated based on the unit attention vector associated with the global compression vector and N text mapping vectors. The model parameters of the initial visual compression model are adjusted based on the prediction results and answer labels until the initial visual compression model meets the model convergence condition, thus obtaining the target visual compression model. The target visual compression model is used to generate question-and-answer results for media data and text information.

[0252] It should be understood that the computer device 1000 described in the embodiments of this application can execute the data processing method described in any of the embodiments corresponding to Figures 4, 5, and 8 above, and will not be repeated here. In addition, the beneficial effects of using the same method will not be repeated here either.

[0253] Furthermore, it should be noted that this application also provides a computer-readable storage medium storing a computer program. When the processor executes the computer program, it can perform the data processing method described in any of the embodiments corresponding to Figures 4, 5, and 8 above. Therefore, it will not be repeated here. Additionally, the beneficial effects of using the same method will not be repeated here either. For technical details not disclosed in the embodiments of the computer-readable storage medium involved in this application, please refer to the description of the method embodiments of this application.

[0254] The aforementioned computer-readable storage medium can be an internal storage unit of the data processing apparatus or computer device provided in any of the foregoing embodiments, such as a hard disk or memory of the computer device. The computer-readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., provided on the computer device. Furthermore, the computer-readable storage medium can include both internal and external storage units of the computer device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium can also be used to temporarily store data that has been displayed or will be displayed.

[0255] Furthermore, it should be noted that this application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the method provided in any of the embodiments corresponding to Figures 4, 5, and 8 above.

[0256] The terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the term "comprising," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or device that includes a series of steps or units is not limited to the listed steps or modules, but may optionally include steps or modules not listed, or may optionally include other step units inherent to these processes, methods, apparatuses, products, or devices.

[0257] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the foregoing description as a network element. Whether these network elements are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can implement the described network elements using different methods for each specific application, but such implementation should not be considered beyond the scope of this application.

[0258] The methods and related apparatus provided in this application are described with reference to the method flowcharts and / or structural diagrams provided in this application. Specifically, each block of the method flowcharts and / or structural diagrams, as well as combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable device, create means for implementing the functions specified in one or more blocks of the flowcharts and / or one or more blocks of the structural diagrams. These computer program instructions can also be stored in a computer-readable storage medium capable of directing a computer or other programmable device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more blocks of the flowcharts and / or one or more blocks of the structural diagrams. These computer program instructions may also be loaded onto a computer or other programmable device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable device, provide steps for implementing the functions specified in one or more flowcharts and / or one or more blocks in the structural diagram.

[0259] The steps in the method of this application embodiment can be adjusted, combined, or deleted according to actual needs.

[0260] The modules in the device of this application embodiment can be merged, divided, and deleted according to actual needs.

[0261] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A data processing method, characterized in that, include: Acquire at least two units of media data, perform vector mapping on each unit of media data to obtain M visual mapping vectors, where M is a positive integer, and the at least two units of media data belong to media data; Obtain text information, perform vector mapping on the text information to obtain N text mapping vectors, where N is a positive integer; The M visual mapping vectors and the N text mapping vectors are combined into a mapping vector sequence. The compressed marker vector is then inserted between the M visual mapping vectors and the N text mapping vectors in the mapping vector sequence to obtain a visual compression sequence. Attention processing is performed on the visual compression sequence to obtain an attention result vector, which includes unit attention vectors corresponding to the M visual mapping vectors, the compression marker vector, and the N text mapping vectors, respectively. The unit attention vector associated with the compressed label vector in the attention result vector is determined as the global compression vector. The global compression vector is used to characterize the global semantics of the M visual mapping vectors, and / or the vector length of the global compression vector is less than the sum of the vector lengths of the M visual mapping vectors. Based on the global compression vector and the unit attention vector associated with the N text mapping vectors, question-and-answer results are generated for the media data and the text information.

2. The method according to claim 1, characterized in that, The vector mapping of the text information yields N text mapping vectors, including: The text information is input into a target visual compression model. In the target visual compression model, the text information is split into words to obtain a word sequence corresponding to the text information. The word sequence includes N words, and a word in the word sequence refers to the smallest basic unit obtained after splitting the text information. Each word in the word sequence is mapped to obtain an N text mapping vector.

3. The method according to claim 1, characterized in that, The attention processing of the visual compression sequence to obtain the attention result vector includes: The visual compression sequence is input into the target visual compression model; the target visual compression model includes an attention mask layer. In the attention mask layer, the visual compression sequence is multiplied by the query parameter matrix to obtain the query matrix, the visual compression sequence is multiplied by the key parameter matrix to obtain the key matrix, and the visual compression sequence is multiplied by the value parameter matrix to obtain the value matrix. A mask matrix is ​​obtained based on the compressed tag vector. Based on the mask matrix, attention calculation is performed on the query matrix, key matrix, and value matrix to obtain unit attention vectors corresponding to the M visual mapping vectors, the compressed tag vector, and the N text mapping vectors. The unit attention vectors corresponding to the M visual mapping vectors, the compressed tag vector, and the N text mapping vectors are determined as attention result vectors. The mask matrix is ​​used to shield the N text mapping vectors from the attention of the M visual mapping vectors during the attention calculation process.

4. The method according to claim 3, characterized in that, The process involves obtaining a mask matrix based on the compressed marker vector, performing attention calculations on the query matrix, key matrix, and value matrix based on the mask matrix to obtain unit attention vectors corresponding to the M visual mapping vectors and the compressed marker vector, and determining the unit attention vectors corresponding to the M visual mapping vectors and the compressed marker vector as the attention result vector, including: The query matrix and the transpose of the key matrix are multiplied by a dot product to obtain the attention score matrix. A masking matrix is ​​generated based on the square root of the dimension value of the key matrix and the attention score matrix to obtain the causal matrix; the causal matrix is ​​a lower triangular matrix. A mask matrix is ​​obtained based on the compressed marker vector and the matrix to be masked. A dot product operation is performed between the matrix to be masked, the causal matrix, and the mask matrix to obtain a mask fraction matrix. The mask fraction matrix is ​​then normalized to obtain a normalized matrix. A dot product operation is performed between the normalized matrix and the value matrix to obtain unit attention vectors corresponding to the M visual mapping vectors, the compressed marker vector, and the N text mapping vectors. These unit attention vectors are then determined as attention result vectors. Specifically, the mask matrix is ​​used to mask the elements in the matrix to be masked that represent the N text mapping vectors focusing on the M visual mapping vectors.

5. The method according to claim 4, characterized in that, The step of obtaining the mask matrix based on the compressed marker vector and the matrix to be masked includes: Based on the compressed marker vector, an initial matrix of M+N+1 rows and M+N+1 columns is generated. In the i-th row and j-th column of the initial matrix, if i belongs to the row number corresponding to the N text mapping vectors in the matrix to be masked, and j belongs to the column number corresponding to the M visual mapping vectors in the matrix to be masked, then the element in the i-th row and j-th column of the initial matrix is ​​set as the first mask value. The elements in the initial matrix other than those with the first mask value are set as the second mask value to obtain the mask matrix.

6. The method according to claim 1, characterized in that, The attention processing of the visual compression sequence to obtain the attention result vector includes: A mask matrix is ​​obtained based on the compressed marker vector, and attention processing is performed on the visual compressed sequence according to the mask matrix to obtain an attention result vector; the mask matrix is ​​used to shield the attention of the N text mapping vectors on the M visual mapping vectors during the attention processing.

7. The method according to claim 1, characterized in that, The generation of question-and-answer results for media data and text information based on the unit attention vector associated with the global compression vector and the N text mapping vectors includes: The unit attention vector associated with the global compression vector and the N text mapping vectors is input into the target visual compression model; the target visual compression model includes a feature understanding layer; Based on the feature understanding layer, a prediction probability for media data is generated by the unit attention vector associated with the global compression vector and the N text mapping vectors. The prediction result with the highest prediction probability is determined as the question-and-answer result for the media data and the text information.

8. The method according to claim 1, characterized in that, The M visual mapping sequences in the mapping vector sequence are arranged before the N text mapping vectors; the step of inserting the compressed marker vector between the M visual mapping vectors and the N text mapping vectors in the mapping vector sequence to obtain a visual compression sequence includes: If the resolution of the media data is greater than the compression threshold, a compression vector value S is generated based on the resolution of the media data. S compression marker vectors are inserted after the M visual mapping vectors in the mapping vector sequence to obtain a visual compression sequence; S is a positive integer; the S compression marker vectors in the visual compression sequence are located before the N text mapping vectors. The attention processing of the visual compression sequence to obtain the attention result vector includes: Based on the S compressed label vectors, attention processing is performed on the visual compression sequence to obtain an attention result vector; the attention result vector includes the unit attention vectors corresponding to the M visual mapping vectors, the S compressed label vectors, and the N text mapping vectors, respectively.

9. A data processing method, characterized in that, include: Input the media data sample, along with the question text and answer labels for the media data sample, into the initial visual compression model; The media data sample is divided into at least two units of media data using an initial visual compression model. Vector mapping is performed on each unit of media data to obtain M visual mapping vectors. Vector mapping is also performed on the question text corresponding to the media data sample to obtain N text mapping vectors. M and N are both positive integers. The M visual mapping vectors and the N text mapping vectors are combined into a mapping vector sequence. The compressed marker vector is then inserted between the M visual mapping vectors and the N text mapping vectors in the mapping vector sequence to obtain a visual compression sequence. Attention processing is performed on the visual compression sequence to obtain an attention result vector, which includes unit attention vectors corresponding to the M visual mapping vectors, the compressed tag vector, and the N text mapping vectors, respectively. The unit attention vector associated with the compressed tag vector in the attention result vector is determined as the global compression vector; The global compression vector is used to characterize the global semantics of the M visual mapping vectors, and / or the vector length of the global compression vector is less than the sum of the vector lengths of the M visual mapping vectors; Based on the global compression vector and the unit attention vector associated with the N text mapping vectors, the prediction result corresponding to the media data sample is generated. Based on the prediction result and the answer label, the model parameters of the initial visual compression model are adjusted until the initial visual compression model meets the model convergence condition, and the target visual compression model is obtained. The target visual compression model is used to generate question-and-answer results for the media data and text information.

10. The method according to claim 9, characterized in that, Also includes: Based on the data distribution information corresponding to the media data samples, multimodal public data with data similarity characteristics to the data distribution information is obtained. The visual language model is trained based on the multimodal public data to obtain an initial visual compression model that converges in training on multiple modalities.

11. The method according to claim 9, characterized in that, The prediction result is a label probability distribution; the step of adjusting the model parameters of the initial visual compression model based on the prediction result and the answer label until the initial visual compression model meets the model convergence condition, to obtain the target visual compression model, includes: A label probability distribution is generated based on the answer labels. The relative entropy between the label probability distribution and the predicted probability distribution is determined as the model loss value. The model parameters of the initial visual compression model are adjusted using the model loss value until the initial visual compression model meets the model training conditions, thus obtaining the target visual compression model.

12. A data processing apparatus, characterized in that, include: The visual encoding module is used to acquire at least two units of media data, perform vector mapping on each unit of media data, and obtain M visual mapping vectors, where M is a positive integer, and the at least two units of media data belong to media data. The text encoding module is used to acquire text information, perform vector mapping on the text information, and obtain N text mapping vectors, where N is a positive integer; An insertion processing module is used to combine the M visual mapping vectors and the N text mapping vectors into a mapping vector sequence, and to insert the compressed marker vector between the M visual mapping vectors and the N text mapping vectors in the mapping vector sequence to obtain a visual compression sequence; The compression processing module is used to perform attention processing on the visual compression sequence to obtain an attention result vector. The attention result vector includes the unit attention vectors corresponding to the M visual mapping vectors and the unit attention vectors associated with the compressed label vectors. The compression determination module is used to determine the unit attention vector associated with the compression label vector in the attention result vector as a global compression vector. The global compression vector is used to characterize the global semantics of the M visual mapping vectors, and / or the vector length of the global compression vector is less than the sum of the vector lengths of the M visual mapping vectors. The output module is used to generate question-and-answer results for the media data and the text information based on the unit attention vector associated with the global compression vector and the N text mapping vectors.

13. A data processing apparatus, characterized in that, include: The sample input module is used to input media data samples, as well as question text and answer labels for the media data samples, into the initial visual compression model; The initial encoding module is used to divide the media data sample into at least two units of media data using an initial visual compression model, perform vector mapping on each unit of media data to obtain M visual mapping vectors, and perform vector mapping on the question text corresponding to the media data sample to obtain N text mapping vectors; M and N are both positive integers. An initial insertion module is used to combine the M visual mapping vectors and the N text mapping vectors into a mapping vector sequence, and to insert the compressed marker vector between the M visual mapping vectors and the N text mapping vectors in the mapping vector sequence to obtain a visual compression sequence; An initial compression processing module is used to perform attention processing on the visual compression sequence to obtain an attention result vector, wherein the attention result vector includes unit attention vectors corresponding to the M visual mapping vectors, the compression marker vector, and the N text mapping vectors, respectively; An initial compression determination module is used to determine the unit attention vector associated with the compression marker vector in the attention result vector as the global compression vector; The global compression vector is used to characterize the global semantics of the M visual mapping vectors, and / or the vector length of the global compression vector is less than the sum of the vector lengths of the M visual mapping vectors; The model adjustment module is used to generate prediction results corresponding to the media data samples based on the global compression vector and the unit attention vector associated with the N text mapping vectors, and to adjust the model parameters of the initial visual compression model based on the prediction results and the answer labels until the initial visual compression model meets the model convergence condition, thereby obtaining the target visual compression model. The target visual compression model is used to generate question-and-answer results for the media data and text information.

14. A computer device, characterized in that, include: Processor, memory, and network interface; The processor is connected to the memory and the network interface, wherein the network interface is used to provide data communication functions, the memory is used to store computer programs, and the processor is used to call the computer programs so that the computer device executes the method according to any one of claims 1-11.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded and executed by a processor to cause a computer device having the processor to perform the method of any one of claims 1-11.

16. A computer program product, characterized in that, The computer program product includes a computer program stored in a computer-readable storage medium and adapted to be read and executed by a processor so that a computer device having the processor performs the method of any one of claims 1-11.

Images