Data processing method and device, equipment and storage medium

Abstract

The embodiment of the invention discloses a data processing method and device, equipment and a storage medium. The data processing method comprises the steps that when the sequence length of a multimedia sequence to be processed is larger than the limited recognition length of a data recognition model, the multimedia sequence is subjected to block processing according to the limited recognition length, and N media sequence blocks are obtained; performing feature coding on the N media sequence blocks through a coder to obtain N media features; key media features of the multimedia sequence are retrieved from the N media features through a querier and a decoder which are used for feature retrieval; and according to the key media features, identifying the multimedia sequence to obtain a multimedia identification result of the multimedia sequence. By adopting the method and the device, the multimedia sequence with any sequence length can be processed by the data identification model, and the identification accuracy of the multimedia sequence is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, and particularly relates to a data processing method and device, equipment and a storage medium. BACKGROUND

[0002] With the development of Internet technology, data recognition models for multimedia (such as text, images, audio, etc.) recognition processing are widely used, which brings great convenience to multimedia recognition processing. For example, the data recognition model can be a dialogue summary model, a context rewriting model, an archiving classification model, etc. in the field of intelligent customer service, and can also be a video recognition model, a video editing model, etc. in the field of video recognition.

[0003] At present, the data recognition model usually has a predefined limited recognition length. When the sequence length of the multimedia sequence input into the data recognition model is greater than the limited recognition length of the data recognition model, the front sequence part (the sequence length of the front sequence part is less than or equal to the limited recognition length) in the multimedia sequence is truncated, or part of the sequence data in the multimedia sequence is deleted to shorten the length of the multimedia sequence, so that the sequence length of the multimedia sequence is less than or equal to the limited recognition length of the data recognition model. This results in that the multimedia sequence finally input into the data recognition model is incomplete, and some key information is largely lost, thereby resulting in low recognition accuracy of the multimedia sequence. SUMMARY

[0004] The embodiments of the present application provide a data processing method, device, equipment and storage medium, which can enable the data recognition model to process multimedia sequences of any sequence length and improve the recognition accuracy of the multimedia sequence.

[0005] In one aspect, the embodiments of the present application provide a data processing method, comprising:

[0006] When the sequence length of the multimedia sequence to be processed is greater than the limited recognition length of the data recognition model, the multimedia sequence is processed by block according to the limited recognition length, to obtain N media sequence blocks; the data recognition model comprises an encoder and a decoder, and N is an integer greater than 1;

[0007] The N media sequence blocks are feature encoded by the encoder to obtain media features corresponding to the N media sequence blocks, respectively;

[0008] The key media features of the multimedia sequence are retrieved from the media features corresponding to the N media sequence blocks by a queryer for feature retrieval and the decoder;

[0009] The multimedia sequence is identified according to the key media feature, and a multimedia identification result of the multimedia sequence is obtained; the multimedia identification result is used to reflect at least one of a multimedia attribute and a multimedia key content of the multimedia sequence.

[0010] The embodiment of the present application provides a data processing apparatus, which comprises:

[0011] The block module is used for performing block processing on the multimedia sequence according to the limited identification length when the sequence length of the multimedia sequence to be processed is greater than the limited identification length of the data identification model, and obtaining N media sequence blocks; the data identification model comprises a decoder and an encoder; N is an integer greater than 1;

[0012] The encoding module is used for performing feature coding on the N media sequence blocks by using the encoder, and obtaining media features corresponding to the N media sequence blocks respectively;

[0013] The searching module is used for searching out the key media feature of the multimedia sequence from the media features corresponding to the N media sequence blocks respectively by using a queryer for feature searching and the decoder;

[0014] The identification module is used for identifying the multimedia sequence according to the key media feature, and obtaining a multimedia identification result of the multimedia sequence; the multimedia identification result is used to reflect at least one of a multimedia attribute and a multimedia key content of the multimedia sequence.

[0015] The embodiment of the present application provides a computer readable storage medium, and the computer readable storage medium stores a computer program.

[0016] The embodiment of the present application provides a computer program product or a computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method provided by the embodiment of the present application.

[0017] The embodiment of the present application provides a method for expanding the input sequence length of a general data recognition model, and the data recognition model is obtained by optimizing the trained recognition model through a queryer for feature retrieval. Since the queryer for feature retrieval is used to retrieve key media features from the media features corresponding to the multimedia sequence to be processed, the code in the data recognition model is not changed, and no additional learning weight is added, so that the data recognition model does not need to be retrained, and the data recognition model can process multimedia sequences with any sequence length, and the recognition accuracy of the multimedia sequence is improved. Specifically, when the sequence length of the multimedia sequence to be processed is greater than the limited recognition length of the data recognition model, the multimedia sequence can be blocked according to the limited recognition length, and N media sequence blocks are obtained. It can be understood that the sequence length of the N media sequence blocks is less than or equal to the limited recognition length, so that the N media sequence blocks can be input into the data recognition model, the features of the N media sequence blocks are encoded through the encoder in the data recognition model, and the media features corresponding to the N media sequence blocks are obtained. Further, the key media features of the multimedia sequence are retrieved from the media features corresponding to the N media sequence blocks through the queryer and the decoder in the data recognition model. According to the key media features, the multimedia sequence is recognized, and the multimedia recognition result of the multimedia sequence is obtained, which is used to reflect at least one of the multimedia attribute and the multimedia key content of the multimedia sequence. It can be seen that the present application does not need to use the truncation method of the front sequence part in the multimedia sequence, or delete part of the sequence in the multimedia sequence to shorten the length of the multimedia sequence, but the entire multimedia sequence is input into the data recognition model, and the key media features are extracted from the N media features corresponding to the multimedia sequence. In this way, the loss of key information of the multimedia sequence caused by truncation or deletion can be avoided, the data recognition model can accept the input of the multimedia sequence with any sequence length, and the multimedia sequence can be accurately recognized without retraining, and the accuracy of the multimedia sequence recognition is improved. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0019] Figure 1 is a structural schematic diagram of a data processing system provided by the embodiment of the present application;

[0020] Figure 2is a schematic diagram of a general agent model training method provided by an embodiment of the present application.

[0021] Figure 3 is a flow schematic diagram of a data processing method provided by an embodiment of the present application.

[0022] Figure 4 is a schematic diagram of a multimedia sequence block processing provided by an embodiment of the present application.

[0023] Figure 5 is a schematic diagram of a text sequence data with a sequence length greater than a limit recognition length for media recognition provided by an embodiment of the present application.

[0024] Figure 6 is a flow schematic diagram of a data processing method provided by an embodiment of the present application.

[0025] Figure 7 is a structural schematic diagram of a data processing apparatus provided by an embodiment of the present application.

[0026] Figure 8 is a schematic diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0028] The present application relates to the technical field of artificial intelligence, and the embodiments of the present application can specifically add a data recognition model of a queryer, can recognize and process multimedia sequences with arbitrary sequence lengths, can realize that a data recognition model can process input with arbitrary sequence lengths, and can also improve the efficiency and accuracy of recognizing and processing multimedia sequences.

[0029] Among them, artificial intelligence (Artificial Intelligence, AI) is to use digital computers or digital computer controlled machine simulation, extension and expansion of human intelligence, perception of the environment, knowledge acquisition and use of knowledge to obtain the best results of theory, method, technology and application system. In other words, artificial intelligence is a comprehensive technology of computer science, which tries to understand the essence of intelligence, and produces a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making. Artificial intelligence technology is a comprehensive discipline, which involves a wide range of fields, both hardware and software technologies. Artificial intelligence basic technology generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning, automatic driving, intelligent transportation and other major directions.

[0030] Specifically, the present application specifically relates to machine learning under artificial intelligence. Machine learning (Machine Learning, ML) is a multi-disciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It is a special study of how computers simulate or implement human learning behavior to acquire new knowledge or skills, reorganize existing knowledge structure to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental approach to making computers intelligent, and its applications are widespread in various fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and rule-based learning.

[0031] Please refer to Figure 1 , Figure 1 is a structural schematic diagram of a data processing system provided by an embodiment of the present application. As Figure 1 indicated, the data processing system can include a server 10 and a terminal device cluster. The terminal device cluster can include one or more terminal devices, which will not be limited in number here. As Figure 1 indicated, it can specifically include terminal device 100a, terminal device 100b, terminal device 100c, …, terminal device 100n. As Figure 1As shown, the terminal device 100a, the terminal device 100b, the terminal device 100c, ..., the terminal device 100n can respectively establish a network connection with the server 10, so that each terminal device can exchange data with the server 10 through the network connection. Of course, the terminal device 100a, the terminal device 100b, the terminal device 100c, ..., the terminal device 100n can communicate with each other through a direct network connection, that is, point-to-point communication can be achieved between each terminal device; that is, when data needs to be exchanged between each two terminal devices, one terminal device (i.e., the sending terminal device) can send data directly to the other terminal device (i.e., the receiving terminal).

[0032] Each terminal device in the terminal device cluster may include: smart phones, tablet computers, laptop computers, desktop computers, intelligent voice interaction devices, smart home appliances (e.g., smart TVs), wearable devices, vehicle-mounted terminals and other intelligent terminals with media data processing capabilities. Figure 1 Each terminal device in the terminal device cluster shown can be installed with an application with a media data processing function. When the application runs in each terminal device, it can be respectively connected to the above Figure 1 Data interaction is performed between the servers 10 shown, such as applications that may include media processing applications, text applications, video applications, audio applications, etc. For ease of understanding, the embodiments of the present application may be Figure 1 Select one terminal device from the multiple terminal devices shown as the target terminal device. For example, the embodiment of the present application can Figure 1 The terminal device 100a shown is used as a target terminal device. An application with a media data processing function may be installed in the target terminal device. At this time, the target terminal device can implement media data interaction with the server 10 through the application in the target terminal device.

[0033] Among them, Figure 1 As shown, the server 10 can be a device that provides background services for applications in terminal devices. The server 10 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0034] It should be understood that based on Figure 1The data processing system in one of the embodiments can be suitable for the identification processing of a multimedia sequence. It can be understood that media refers to the form of information, such as media can include text, sound, image, animation, etc. The multimedia sequence in the embodiments of the present application can refer to text sequence data, video sequence data, image sequence data, and audio sequence data, etc. Among them, the text sequence data can include intelligent question and answer sequence data, intelligent customer service sequence data, information sequence data, book sequence data, etc. Among them, the video sequence data can include film and television video sequence data, live video sequence data, etc. The data recognition model can be used for media recognition of the multimedia sequence to obtain a multimedia recognition result corresponding to the multimedia sequence, and the multimedia recognition result is used to reflect at least one of the multimedia attribute and the multimedia key content of the multimedia sequence. Among them, the multimedia attribute can be used to reflect the category and nature of the multimedia sequence, and the multimedia key content can be the media meaning, media summary, media context rewriting, etc. of the multimedia sequence. For example, when the multimedia sequence is text sequence data, the multimedia recognition result of the text sequence data can be the text summary, context rewriting result, and archiving classification category of the text sequence data. Among them, the data recognition model can be an encoder-decoder model based on the Transformer structure. The Transformer is a neural network structure based on the self-attention mechanism, and the data recognition model includes an encoder and a decoder. The encoder is responsible for processing the input data and converting it into an internal representation, and the decoder generates the output according to the internal representation. The self-attention mechanism is the key, which allows the data recognition model to consider all other words or tokens in the input sequence when processing each word or token, so that the data recognition model can capture long-distance dependencies and understand the complex relationships between words or tokens.

[0035] For example, the data recognition model can be a pre-trained model (i.e., already trained) such as a BART model, a Longformer model, an LLM (Large Language Model) model, etc. Among them, the BART model (Bidirectional and Auto-Regressive Transformers) is a very popular model in the field of natural language processing, which combines the advantages of bidirectional encoder and autoregressive decoder, so that the model can understand the text while generating high-quality text. The Longformer model is a Transformer model specially designed for processing long sequence text. The attention mechanism of the Longformer model combines the characteristics of local attention and global attention. The local attention mechanism allows the model to focus only on the local information near the current position when processing long sequences, thereby significantly reducing the computational complexity. The global attention mechanism ensures that the model can still capture important global information in the sequence. The LLM model, also known as a large language model, is an artificial intelligence model designed to understand and generate human language. The LLM model is characterized by its large size, containing billions of parameters, which helps it learn complex patterns in language data.

[0036] Among them, the data recognition model in the embodiments of the present application can be trained, i.e., the data recognition model is in a convergent state, such as the model recognition loss of the data recognition model being less than or equal to a loss threshold or the number of iterations of the data recognition model reaching a target number. Since each input token (such as a character (Token), e.g., when the multimedia sequence is text sequence data, each input token can be a word) in the input multimedia sequence needs to calculate attention weights with other tokens, this results in a time complexity of the attention calculation operation being the square of the length of the input content. Therefore, when the sequence length of the input multimedia sequence is very long, the computational overhead in the data recognition model becomes very large, which can cause the data recognition model to be difficult to run. Therefore, the data recognition model usually has a predefined limit recognition length. Currently, most data recognition models accept input sequence lengths of 1024 lengths. If the input sequence length is too long, the input multimedia sequence will be truncated, or some intermediate media data in the input multimedia sequence will be deleted to shorten the overall context length. However, an incomplete context will largely lose some key information, which will affect the output effect of the data recognition model.

[0037] However, about 40% of the current multimedia sequences have context sequence lengths exceeding 1024 (e.g., 1024 characters), and about 10% of the multimedia sequences have context sequence lengths exceeding 2048. If some additional supplementary information is added, the overall sequence length may exceed 3000. Therefore, how to enable the data recognition model to receive the ultra-long text under the condition of limited resources is a problem to be solved. Based on this, the embodiments of the present application provide an optimization scheme for the data recognition model, so that the data recognition model can accept input with unlimited sequence length, greatly expand the context length that the data recognition model can perceive, and thus improve the recognition accuracy of the input multimedia sequence.

[0038] Specifically, when the sequence length of the multimedia sequence to be processed is greater than the limited recognition length of the data recognition model, the multimedia sequence is processed in blocks according to the limited recognition length of the data recognition model, to obtain N media sequence blocks. N is a positive integer greater than 1, that is, N can be 2, 3, 4, etc.

[0039] The N media sequences are feature encoded by the encoder in the data recognition model to obtain media features corresponding to the N media sequence blocks respectively. The encoder can be an encoder based on the Transformer structure, which can include a self-attention layer and a feedforward neural network. The self-attention layer can enable the current node to not only focus on the current token, but also obtain the semantic context of the multimedia sequence. The feedforward neural network can be used for linear transformation. The number of encoders can be one or more. It can be understood that when the number of encoders is multiple, each of the multiple encoders can be used to feature encode the N media sequence blocks to obtain media features corresponding to the N media sequence blocks respectively. When the number of encoders is multiple, the input of the previous encoder can be used as the input of the next encoder. For example, the multiple encoders include a first encoder and a second encoder, the first encoder is the previous encoder of the second encoder, and the output of the first encoder can be used as the input of the second encoder.

[0040] Further, the key media features of the multimedia sequence are retrieved from the media features corresponding to the N media sequence blocks respectively by a queryer for feature retrieval and a decoder in the data recognition model, and the key media features can be more important parts of the media features corresponding to the N media sequence blocks respectively. The decoder can be a decoder based on a Transformer structure, and the decoder can include a cross-attention layer and a decoding layer. The cross-attention layer adopts a cross-attention mechanism, which is a special calculation method in the attention mechanism. In the cross-attention mechanism, a position in one sequence is calculated with attention to all positions in another sequence. In simple terms, the cross-attention mechanism allows the data recognition model to find and focus on relevant information in different sequences or data sets, so as to more accurately understand the input multimedia sequence and extract effective media features from the input multimedia sequence. The decoding layer is used to decode the effective media features in the multimedia sequence to obtain the multimedia recognition result of the multimedia sequence. The number of decoders can be one or more.

[0041] The queryer for feature retrieval can be a KNN (k-Nearest Neighbor) search algorithm, an ANN (Approximate Nearest Neighbor) method, etc. The KNN search algorithm is a K-Nearest Neighbor (KNN) classification algorithm, which is used to retrieve media features in the data storage area based on the retrieval attention parameters of each decoder to obtain the most relevant media features corresponding to each decoder as the initial key media features corresponding to the corresponding decoder. The ANN (Approximate Nearest Neighbor) method is a technology for quickly finding the most similar items in a large data set based on a given query. The ANN method usually uses heuristic techniques to speed up the search process, such as pruning and approximate search. These methods can effectively reduce the amount of data that needs to be compared, thereby speeding up the search.

[0042] In this way, the key media feature can be searched from the whole multimedia sequence, the key calculation is performed on the key media feature (i.e., the more important part), without needing to truncate the multimedia sequence and input only the first k tags of the multimedia sequence, or delete part of the multimedia sequence, thereby improving the recognition accuracy of the multimedia sequence, and realizing that the data recognition model can accept a multimedia sequence of an arbitrary sequence length, thereby improving the recognition accuracy of the data recognition model. According to the key media feature, the media recognition is performed on the multimedia sequence, to obtain a multimedia recognition result of the multimedia sequence, the multimedia recognition result is used to reflect at least one of a multimedia attribute and a multimedia key content of the multimedia sequence, the multimedia attribute can be used to reflect a category and a property of the multimedia sequence, and the multimedia key content can be a media meaning, a media summary, a media context rewriting, and the like of the multimedia sequence.

[0043] The queryer can be located in the data recognition model (e.g., the queryer is a network layer in the data recognition model), or can be located outside the data recognition model (i.e., the queryer is an independent component and can be called by the data recognition model).

[0044] Optionally, the queryer is located in the data recognition model, i.e., the data recognition model in the embodiment of the application can be obtained by adding the queryer to a trained recognition model, such as a network layer in the data recognition model. Since the queryer is used to search the key media feature from the media feature corresponding to the multimedia sequence to be processed, no additional learning weight is added, and the recognition model is already trained. Therefore, the data recognition model can be improved without any further training of the data recognition model, and accurate recognition of the multimedia sequence can be realized by the data recognition model without training, which can reduce the training cost and improve the media recognition efficiency. Of course, if the data recognition model can be further fine-tuned, the performance of the data recognition model can be further improved.

[0045] It can be understood that, in the embodiment of the application, the queryer can be added to the trained recognition model to obtain the data recognition model, and the calculation method in the data recognition model can be improved to adapt to the newly added queryer, so that the data recognition model can accept an input of an unlimited sequence length, and the context length that can be perceived by the data recognition model is greatly expanded. At the same time, without training the data recognition model, the data recognition model can also accurately recognize the multimedia sequence, which can avoid the training cost caused by model training and improve the media recognition efficiency.

[0046] Optionally, the queryer can be located outside the data recognition model (i.e., the queryer is an independent component and can be called by the data recognition model), and at this time the data recognition model can be a trained recognition model. Since the queryer is located outside the data recognition model, the code in the data recognition model will not be changed, and no additional learning weights will be added. Therefore, the data recognition model does not need to be retrained, and the data recognition model can process multimedia sequences of any sequence length and improve the recognition accuracy of the multimedia sequence.

[0047] As shown in Figure 2 , Figure 2 is a schematic diagram of a method for implementing a data recognition model to accept any sequence length provided by an embodiment of the present application, as shown in Figure 2 , the terminal device 20a can be any terminal device in the terminal device cluster described above Figure 1 , such as the terminal device 100b in the terminal device 20a described above Figure 1 , the terminal device 20a installs an application, and the terminal device 20a can realize data interaction between the application and the server 20b, and the server 20b can be the server 10 described above Figure 1 . The terminal device 20a can obtain a multimedia sequence to be processed, which can be intelligent question and answer sequence data, and send the multimedia sequence to be processed to the server 20b, request the server 20b to perform media recognition on the multimedia sequence, and return the media recognition result of the multimedia sequence. The multimedia recognition result is used to reflect at least one of the multimedia attribute and the multimedia key content of the multimedia sequence, the multimedia attribute can be used to reflect the category, nature, etc. of the multimedia sequence, and the multimedia key content can be the media meaning, media summary, media context rewriting, etc. of the multimedia sequence.

[0048] The server 20b can obtain the sequence length of the multimedia sequence, such as obtaining the character length of the multimedia sequence, and take the character length of the multimedia sequence as the sequence length of the multimedia sequence. Further, the server 20b can compare the sequence length of the multimedia sequence with the limited recognition length of the data recognition model to obtain a length comparison result. When the length comparison result indicates that the sequence length of the multimedia sequence is greater than the limited recognition length of the data recognition model, the server 20b can divide the multimedia sequence according to the limited recognition length of the data recognition model to obtain N media sequence blocks. Each media sequence block in the N media sequence blocks has a sequence length less than the limited recognition length of the data recognition model. In this way, the N media sequence blocks can be input into the data recognition model (the limited recognition length of the added recognition model is the same as the limited recognition length of the data recognition model), and the media recognition of the multimedia sequence is performed by the data recognition model. In this way, the multimedia sequence does not need to be truncated and only the first k tokens of the multimedia sequence are input or part of the multimedia sequence is deleted, which improves the recognition accuracy of the multimedia sequence, and also realizes that the data recognition model can accept multimedia sequences of any sequence length, which improves the recognition accuracy of the data recognition model.

[0049] Of course, when the length comparison result indicates that the sequence length of the multimedia sequence is less than or equal to the limited recognition length of the data recognition model, the multimedia sequence does not need to be divided, and the multimedia sequence can be normally input into the data recognition model, and the media recognition of the multimedia sequence is performed by the encoder and the decoder in the data recognition model to obtain the media recognition result of the multimedia sequence. At this time, when the sequence length of the multimedia sequence is less than or equal to the limited recognition length of the data recognition model, the queryer does not need to be called for related processing, that is, the processing can be performed according to the normal processing process of the encoder and the decoder.

[0050] For example, the server 20b can obtain the sequence length of the multimedia sequence, such as obtaining the character length of the multimedia sequence, and take the character length of the multimedia sequence as the sequence length of the multimedia sequence. Further, the server 20b can compare the sequence length of the multimedia sequence with the limited recognition length of the data recognition model to obtain a length comparison result. When the length comparison result indicates that the sequence length of the multimedia sequence is greater than the limited recognition length of the data recognition model, the server 20b can divide the multimedia sequence according to the limited recognition length of the data recognition model to obtain N media sequence blocks. Each media sequence block in the N media sequence blocks has a sequence length less than the limited recognition length of the data recognition model. In this way, the N media sequence blocks can be input into the data recognition model (the limited recognition length of the added recognition model is the same as the limited recognition length of the data recognition model), and the media recognition of the multimedia sequence is performed by the data recognition model. In this way, the multimedia sequence does not need to be truncated and only the first k tokens of the multimedia sequence are input or part of the multimedia sequence is deleted, which improves the recognition accuracy of the multimedia sequence, and also realizes that the data recognition model can accept multimedia sequences of any sequence length, which improves the recognition accuracy of the data recognition model. Figure 2As shown, the data recognition model is a queryer for feature retrieval added to a trained recognition model including an encoder and a decoder, which can be an encoder-decoder model based on a Transformer structure, the encoder can be an encoder of a Transformer structure, and the decoder can be a decoder of a Transformer structure. It can be understood that when the length comparison result indicates that the sequence length of the multimedia sequence is greater than the limited recognition length of the data recognition model, the multimedia sequence is recognized by the encoder, the decoder, and the queryer in the data recognition model to obtain a multimedia recognition result of the multimedia sequence. When the length comparison result indicates that the sequence length of the multimedia sequence is less than or equal to the limited recognition length of the data recognition model, the multimedia sequence is directly recognized by the encoder and the decoder in the data recognition model to obtain a multimedia recognition result of the multimedia sequence.

[0051] Specifically, when the comparison result indicates that the sequence length of the multimedia sequence is greater than the limited recognition length of the data recognition model, after the multimedia sequence is divided into N media sequence blocks, the server 20b can input the N media sequence blocks into the data recognition model, encode the N media sequence blocks by the encoder in the data recognition model respectively to obtain media features corresponding to the N media sequence blocks respectively. Further, the server 20b can obtain retrieval attention parameters of the decoder by the decoder in the data recognition model, and retrieve key media features in the multimedia sequence from the media features corresponding to the N media sequence blocks respectively by the queryer according to the retrieval attention parameters of the decoder. The key media features are decoded by the decoder to output decoded media features, and the decoded media features are further recognized to obtain a multimedia recognition result of the multimedia sequence. It can be seen that the decoder only needs to decode the key media features and does not need to decode all the media features. It can be seen that by adding the queryer to the trained recognition model, without additional learning weights and without modifying the codes of the encoder and the decoder, the input arbitrary sequence length of the data recognition model can be expanded, that is, the data recognition model can accept multimedia sequences of arbitrary sequence length, and the applicability of the data recognition model is improved. At the same time, the multimedia sequence does not need to be truncated and only the first k tokens of the multimedia sequence are input or part of the sequence in the multimedia sequence is deleted, and the recognition accuracy of the multimedia sequence is improved. In addition, the data recognition model does not need to be trained to accurately recognize the multimedia sequence, and the training cost caused by model training is avoided.

[0052] Further, please refer to Figure 3 , Figure 3is a flowchart of a data processing method provided by an embodiment of the present application. As shown in Figure 3 The method can be executed by any terminal device in Figure 1 , can also be executed by the server 10 in Figure 1 , and can also be executed by the terminal device and the server 10 in Figure 1 together. In the present application, the device used to execute the data processing method can be collectively referred to as a computer device. The data processing method can include, but is not limited to, the following steps:

[0053] S101, when the sequence length of the multimedia sequence to be processed is greater than the limited recognition length of the data recognition model, the multimedia sequence is processed by block according to the limited recognition length, and N media sequence blocks are obtained.

[0054] Specifically, when the sequence length of the multimedia sequence to be processed is greater than the limited recognition length of the data recognition model, the computer device can process the multimedia sequence by block according to the limited recognition length, and obtain N media sequence blocks, where N is an integer greater than 1. Since the data recognition model usually has a predefined limited recognition length, i.e., the data recognition model does not accept multimedia sequences with a sequence length greater than the limited recognition length, when the sequence length of the multimedia sequence is greater than the limited recognition length, the computer device can process the multimedia sequence by block to obtain N media sequence blocks. The sequence length of each media sequence block in the N media sequence blocks is less than the limited recognition length, so that the N media sequence blocks can be input into the data recognition model, and the data recognition model can recognize the N media sequence blocks. In this way, it is realized to input multimedia sequences with a sequence length greater than the limited recognition length of the data recognition model into the data recognition model. The data recognition model can include an encoder and a decoder, and can be a pre-trained model such as a BART model, a Longformer model, an LLM (Large Language Model) model, etc.

[0055] Optionally, the specific manner in which the computer device processes the multimedia sequence by block can include: when the sequence length of the multimedia sequence to be processed is greater than the limited recognition length of the data recognition model, determining the window size of a sliding block window according to the limited recognition length. The multimedia sequence is processed by block according to the sliding block window with the window size and the sliding step of the sliding block window, and N media sequence blocks are obtained; the sliding step is less than the limited recognition length.

[0056] Specifically, the computer device can determine the window size of the sliding chunk window according to the limit recognition length. Specifically, the computer device can determine the limit recognition length as the window size of the sliding chunk window, or the computer device can determine a length (e.g., two-thirds of the limit recognition length, or half of the limit recognition length, etc.) smaller than the limit recognition length as the window size of the sliding chunk window, as long as the window size of the sliding chunk window is smaller than or equal to the limit recognition length, the length can be set according to specific requirements, and the embodiments of the present application do not limit the window size of the sliding chunk window. The computer device can obtain a sliding step size of the sliding chunk window, and the sliding step size is smaller than or equal to the limit recognition length. For example, the sliding step size can be equal to the limit recognition length, or the sliding step size can be half of the limit recognition length, or the sliding step size can be one-third of the limit recognition length, and the length of the sliding step size can be set according to specific requirements, and the embodiments of the present application do not limit the length of the sliding step size.

[0057] Further, the computer device can perform chunk processing on the multimedia sequence according to the sliding chunk window with the window size and the sliding step size of the sliding chunk window, to obtain N media sequence blocks. For example, when the limit recognition length is determined as the window size of the sliding chunk window, and half of the limit recognition length is used as the sliding step size, the computer device can divide the starting point of the sliding chunk window at the starting point of the multimedia sequence, and divide the sequence covered by the sliding chunk window in the multimedia sequence as the first media sequence block, and the sequence length of the first media sequence block is equal to the limit recognition length. Further, the computer device can continue to use the midpoint of the first media sequence block in the multimedia sequence (i.e., the position of one-half of the first media sequence block in the multimedia sequence) as the starting point of the second media sequence block, divide the starting point of the sliding chunk window at the starting point of the second media sequence block, and continue to divide the multimedia sequence to obtain the second media sequence block. In this way, the multimedia sequence is continuously chunked to obtain N media sequence blocks.

[0058] Optionally, the multimedia sequence can be a question and answer sequence data, and the question and answer sequence data includes P questions and O answers, and P and Q are positive integers. The specific manner in which the computer device performs chunk processing on the multimedia sequence can include: when the sequence length of the multimedia sequence to be processed is greater than the limit recognition length of the data recognition model, sorting the P questions and the Q answers according to the generation time corresponding to each of the P questions and the generation time corresponding to each of the Q answers, to obtain sorted question and answer sequence data. According to the limit recognition length, the sorted question and answer sequence data is chunked to obtain N media sequence blocks; one of the N media sequence blocks includes at least one question and an answer corresponding to the at least one question.

[0059] Specifically, the P questions and the O answers each have a corresponding generation time, and when the sequence length of the multimedia sequence to be processed is greater than the limited recognition length of the data recognition model, the computer device can sort the P questions and the Q answers from early to late according to the generation times of the P questions and the generation times of the Q answers respectively, to obtain sorted question and answer sequence data. Further, the computer device can perform block processing on the sorted question and answer sequence data according to the limited recognition length, to obtain N media sequence blocks. Each of the N media sequence blocks includes at least one question and an answer corresponding to the at least one question, and the sequence length of each of the N media sequence blocks is less than the limited recognition length. For example, one of the N media sequence blocks includes 5 questions and answers corresponding to the 5 questions, or one of the N media sequence blocks includes 7 questions and answers corresponding to the 7 questions. The number of questions included in each of the N media sequence blocks can be set according to specific requirements, and the number of questions included in each of the N media sequence blocks can be the same or different. The computer device can obtain semantic information of the question and answer sequence data, and dynamically adjust the window size of the sliding block window of the question and answer sequence data according to the semantic information of the question and answer sequence data.

[0060] As shown in Figure 4 , Figure 4 is a schematic diagram of block processing of a multimedia sequence provided by an embodiment of the present application, as shown in Figure 4 , the multimedia sequence can be intelligent customer service sequence data, and the intelligent customer service sequence data includes a plurality of questions and a plurality of answers. When the sequence length of the intelligent customer service sequence data is greater than the limited recognition length of the data recognition model, the computer device can perform overlapping block encoding on the intelligent customer service sequence data to obtain N media sequence blocks. Specifically, the computer device can sort the plurality of questions and the plurality of answers from early to late according to the generation times of the plurality of questions and the generation times of the plurality of answers respectively, to obtain sorted plurality of questions and plurality of answers. Further, the computer device can set the window size of the sliding block window to be less than the limited recognition length of the data recognition model, and perform block processing on the sorted plurality of questions and plurality of answers according to the set sliding block window, to obtain N media sequence blocks. Each of the N media sequence blocks includes at least one question and an answer corresponding to the at least one question, and the sequence length of each of the N media sequence blocks is less than the limited recognition length.

[0061] As shown in Figure 4As shown, the computer device can divide the question "Hello, what can I do for you" in the intelligent customer service sequence data, the answer "I always get the easy card to pay money, I want to turn it off, how to turn it off", the question "You mean to bind the relative card bound by others, do you need to unbind now", and the answer "Yes, I can't unbind" into one media sequence block, obtaining the media sequence block 40a. The computer device can divide the question "Are you the card giver or the receiver" in the intelligent customer service sequence data, the question "No, how do I know, I was clicked on an easy card last time, I don't know myself", the answer "A method of operation unbinding will be pushed to you by the incoming call message later, and you can unbind according to the unbinding method", the question "My service opens it to let you give, there is no unbinding operation", the question "Have you successfully given", and the answer "Consumed several times" into one media sequence block, obtaining the media sequence block 40b. The computer device can divide the question "My service opens it to let you give, there is no unbinding operation" in the intelligent customer service sequence data, the question "Have you successfully given", the answer "Consumed several times", the answer "According to the message sent to you, it can be successfully unbound" into one media sequence block, obtaining the media sequence block 40d. It can be seen that when the sequence length of the intelligent customer service sequence data is greater than the limited recognition length of the data recognition model, the computer device can perform overlapping block coding on the intelligent customer service sequence data to ensure that each media sequence block has an overlapping part with other media sequence blocks, and at the same time, ensure that each media sequence block has sufficient context.

[0062] Of course, the computer device can take one question in the intelligent customer service sequence data and the answer corresponding to the one question as a basic unit of one session round, and sequentially intercept the conversation content according to 5 rounds for overlapping block division. Specifically, the computer device can represent the multimedia sequence as D, divide the multimedia sequence by a sliding block window to obtain N chunks (i.e., N media sequence blocks), and after each media sequence block passes through an encoder (i.e., an Encoder module), output the hidden layer state S (i.e., the media feature) of each media sequence block. Finally, the multimedia sequence D is represented as a list composed of multiple hidden layer states, D = [S1, S2, S3, …, SN]. n ]。

[0063] In S102, the encoder is used to perform feature coding on the N media sequence blocks to obtain the media features corresponding to the N media sequence blocks respectively.

[0064] Specifically, the computer device can encode features of the N media sequence blocks through an encoder in the data recognition model to obtain media features corresponding to the N media sequence blocks respectively. The number of the encoder can be one or more. When the number of the encoder is one, the computer device can encode features of the N media sequence blocks through the one encoder to obtain media features corresponding to the N media sequence blocks respectively. Taking a tth media sequence block in the N media sequence blocks as an example, the computer device can extract features of the tth media sequence block to obtain media features corresponding to the tth media sequence block, where t is a positive integer less than or equal to N. When the number of the encoder is more than one, the computer device can input the N media sequence blocks into a first encoder, encode features of the N media sequence blocks through the first encoder to obtain initial media features corresponding to the N media sequence blocks output by the first encoder. Further, the computer device can input the initial media features corresponding to the N media sequence blocks output by the first encoder into a second encoder, encode the initial media features corresponding to the N media sequence blocks output by the first encoder through the second encoder to obtain initial media features corresponding to the N media sequence blocks output by the second encoder. In this way, until the initial media features corresponding to the N media sequence blocks output by a last encoder in the multiple encoders are obtained as the media features corresponding to the N media sequence blocks.

[0065] Optionally, the encoder can be an encoder of a Transformer structure, and a specific manner in which the computer device encodes features of the N media sequence blocks through the encoder can include: obtaining a media embedding vector of a tth media sequence block in the N media sequence blocks, performing attention calculation on the media embedding vector of the tth media sequence block through the encoder to obtain attention features of the tth media sequence block, where t is a positive integer less than or equal to N. Fusing the media embedding vector and the attention features of the tth media sequence block to obtain fused attention features, performing linear transformation on the fused attention features to obtain transformed attention features of the tth media sequence block. Fusing the fused attention features and the transformed attention features of the tth media sequence block to obtain media features corresponding to the tth media sequence block.

[0066] Specifically, the encoder can include a feature embedding layer, a self-attention layer, a first fusion layer, a first feedforward neural network, and a second fusion layer. Specifically, taking the tth media sequence block in the N media sequence blocks as an example, the computer device can obtain the media embedding vector of the tth media sequence block in the N media sequence blocks, and perform attention calculation on the media embedding vector of the tth media sequence block through the self-attention layer in the encoder to obtain the attention feature corresponding to the media embedding vector of the tth media sequence block. The tth media sequence block can be composed of multiple tokens, and the self-attention layer can not only focus on the information of the current token in the tth media sequence block, but also obtain the association information between the current token in the tth media sequence block and other tokens in the tth media sequence block, i.e., the semantic of the context of the tth media sequence block.

[0067] Further, the computer device can fuse the media embedding vector and the attention feature of the tth media sequence block through the first fusion layer in the encoder to obtain the fused attention feature. The first fusion layer can be Add&normalize, through which the media embedding vector and the attention feature of the tth media sequence block can be residual connected (i.e., Add) to obtain an initial fused attention feature, and the initial fused attention feature can be further normalized (i.e., normalize) to obtain the fused attention feature. Further, the computer device can perform linear transformation on the attention feature of the tth media sequence block through the first feedforward neural network in the encoder to obtain the transformed attention feature of the tth media sequence block. Further, the computer device can fuse the fused attention feature and the transformed attention feature of the tth media sequence block through the second fusion layer in the encoder to obtain the media feature corresponding to the tth media sequence block. The second fusion layer can also be Add&normalize, through which the fused attention feature and the transformed attention feature of the tth media sequence block can be residual connected (i.e., Add) to obtain an initial media feature, and the initial media feature can be further normalized (i.e., normalize) to obtain the media feature corresponding to the tth media sequence block. It can be understood that the media features of other media sequence blocks in the N media sequence blocks can be obtained by referring to the manner of obtaining the media feature of the tth media sequence block, so as to obtain the media features corresponding to the N media sequence blocks respectively.

[0068] Optionally, the computer device can store the media features corresponding to the N media sequence blocks in a data storage area, which can be a GPU (Graphics Processing Unit) memory area, or a CPU (Central Processing Unit) memory area, etc. Since the GPU memory area has low latency and the CPU memory area has high throughput, efficient storage and retrieval can be achieved, facilitating subsequent retrieval of key media features from the media features corresponding to the N media sequence blocks.

[0069] In S103, the key media features of the multimedia sequence are retrieved from the media features corresponding to the N media sequence blocks by using the queryer and the decoder for feature retrieval.

[0070] Specifically, the computer device can retrieve the key media features of the multimedia sequence from the media features corresponding to the N media sequence blocks by using the queryer and the decoder for feature retrieval. It can be understood that when the sequence length of the multimedia sequence is greater than the limited recognition length, if the multimedia sequence is truncated and the first k tokens (the sequence length of the first k tokens is the same as the limited recognition length) of the multimedia sequence are input into the data recognition model, or if part of the sequence of the multimedia sequence is deleted (the sequence length of the deleted multimedia sequence is less than or equal to the limited recognition length) and input into the data recognition model, the key information of the multimedia sequence can be lost. Based on this, when the sequence length of the multimedia sequence to be processed is greater than the limited recognition length, the multimedia sequence is processed by block and feature encoding in the embodiments of the present application to obtain media features corresponding to the N media sequence blocks. Since the queryer is used to retrieve the key media features from the media features corresponding to the multimedia sequence to be processed, no additional learning weight is added, and by using the queryer, the key media features of the multimedia sequence are retrieved from the media features corresponding to the N media sequence blocks according to the retrieval attention parameters in the decoder, so as to facilitate subsequent decoding processing and media recognition processing. Therefore, the data recognition model can be improved without any further training of the data recognition model, and the multimedia sequence can be accurately recognized by the data recognition model without training, which can reduce the training cost and improve the media recognition efficiency. Of course, if the data recognition model can be further fine-tuned, the performance of the data recognition model can be further improved.

[0071] In this way, since the key media features of the multimedia sequence are retrieved from the media features corresponding to the N media sequence blocks (i.e., the media features of the entire multimedia sequence), the multimedia sequence does not need to be truncated and media recognition is performed on the first k tokens (the sequence length of the first k tokens is the same as the limited recognition length), nor does the multimedia sequence need to be partially deleted and then media recognition is performed. The problem of loss of key information of the multimedia sequence caused by truncation or deletion, which leads to low media recognition accuracy of the multimedia sequence, can be avoided, and the data recognition model can accept multimedia sequences of any sequence length, thereby improving the accuracy of multimedia recognition.

[0072] It can be understood that when the sequence length of the multimedia sequence is greater than the limited recognition length, the multimedia sequence will be truncated and only the first k tokens of the multimedia sequence will be used as the input of the encoder, or part of the sequence of the multimedia sequence will be deleted and then used as the input of the encoder. That is, in the original data recognition model, only the first k tokens of the multimedia sequence or part of the sequence of the multimedia sequence are deleted and then media recognition is performed, which will lose the key information in the multimedia sequence, and thus lead to low recognition accuracy of the multimedia sequence. The application can process the multimedia sequence in blocks to obtain N media sequence blocks, and use the N media sequence blocks as the input of the encoder to obtain the media features corresponding to the N media sequence blocks. In this way, the multimedia sequence as a whole is input into the encoder, and the key information in the multimedia sequence is not lost, and the recognition accuracy of the multimedia sequence can be improved.

[0073] Meanwhile, the computer device can retrieve, through the queryer, the initial key media features corresponding to each decoder from the media features corresponding to the N media sequence blocks, decode the initial key media features, and obtain the output (i.e., the decoded media features) of each decoder. In this way, the codes in the encoder and the decoder are not changed, and no additional learning weights are needed, so the data recognition model does not need to be trained, and any sequence length of the multimedia sequence can be accurately recognized, thereby expanding the length of the multimedia sequence input to the data recognition model.

[0074] Optionally, the queryer can be a module using a KNN search algorithm, which is a K-Nearest Neighbor (KNN) classification algorithm, used to retrieve the media features in the data storage area based on the retrieval attention parameters of each decoder, to obtain the most relevant media features corresponding to each decoder as the initial key media features corresponding to the corresponding decoder.

[0075] Optionally, if the sequence length of the multimedia sequence is greater than the target length, the queryer can use an ANN (Approximate Nearest Neighbor) method (i.e., an approximate nearest neighbor search method) for retrieval to improve retrieval efficiency and stability. Among them, the ANN (Approximate Nearest Neighbor) method is a technique for quickly finding the most similar item to a given query in a large dataset. The ANN method usually uses heuristic techniques to speed up the search process, such as pruning and approximate search. These methods can effectively reduce the amount of data that needs to be compared, thereby speeding up the search process. Specifically, the ANN method can be divided into the following categories: first, Hashing-based ANN algorithm: Hashing-based ANN algorithm can use hashing technology to map data points into a low-dimensional space and perform approximate nearest neighbor search in this low-dimensional space. Common hashing methods include Locality Sensitive Hashing (LSH). Second, Tree-based ANN algorithm: Tree-based ANN algorithm uses tree structure to organize data, such as KD-Tree, Ball-Tree, etc. They store data points in tree nodes by recursively dividing the data space, and use pruning and approximate search techniques to speed up the search process. Third, Vector Quantization-based ANN algorithm: Vector Quantization-based ANN algorithm encodes points in a vector space using a finite subset of them. In vector quantization encoding, each data item is represented as a code, which can significantly reduce the storage space overhead.

[0076] Among them, the queryer can be located in the data recognition model (such as the queryer being a network layer in the data recognition model), or outside the data recognition model (i.e., the queryer being an independent component that can be called by the data recognition model).

[0077] Optionally, the queryer is located in the data recognition model, that is, the data recognition model in the embodiment of the application can be obtained by adding the queryer to the trained recognition model, such as the queryer can be a network layer in the data recognition model. Specifically, the computer device can add the queryer for feature retrieval to the trained recognition model to obtain the data recognition model. The trained recognition model includes an encoder and a decoder, so that the encoder, the queryer, and the decoder in the data recognition model are implemented by the encoder, the queryer, and the decoder in the data recognition model, so as to enable the data recognition model to process an input (i.e., a multimedia sequence) of an arbitrary length, and improve the applicability of the data recognition model. At the same time, it is not necessary to truncate the multimedia sequence and input only the first k tokens of the multimedia sequence (i.e., the sequence length of the first k tokens is equal to the limited recognition length of the data recognition model), or delete part of the sequence in the multimedia sequence, so as to avoid missing the key information in the multimedia sequence, and improve the recognition accuracy of the multimedia sequence.

[0078] wherein the recognition model is trained, that is, the data recognition model is in a convergent state, such as the model recognition loss of the recognition model is less than or equal to a loss threshold, or the iteration number of the recognition model reaches a target number, and the multimedia sequence can be accurately recognized by the recognition model. The embodiment of the application adds the queryer to the trained recognition model, and does not change the code (such as the code in the encoder and the decoder) in the recognition model, and does not need to increase additional learning weights, so that the data recognition model obtained by adding does not need to be retrained, and can also accurately recognize the multimedia sequence. Of course, the computer device can continue to fine-tune the data recognition model to further improve the performance of the data recognition model and further improve the recognition accuracy of the multimedia sequence. Wherein, adding the queryer in the recognition model does not affect the limited recognition length of the input (i.e., the multimedia sequence) that the recognition model can accept, so that the limited recognition length of the input that the recognition model can accept is the same as the limited recognition length of the input that the data recognition model after adding the query model can accept, that is, adding the queryer does not affect the limited recognition length of the recognition model.

[0079] Optionally, the queryer can be located outside the data recognition model (i.e., the queryer is a separate component and can be called by the data recognition model), and at this time, the data recognition model can be a trained recognition model. Since the queryer is located outside the data recognition model, the code in the data recognition model will not be changed, and no additional learning weights will be added. Therefore, the data recognition model does not need to be retrained, and the data recognition model can process multimedia sequences of any sequence length and improve the recognition accuracy of the multimedia sequence. Specifically, when the computer device obtains the media features corresponding to the N media sequence blocks, the computer device can retrieve the key media features of the multimedia sequence from the media features corresponding to the N media sequence blocks through the queryer and the decoder, and then accurately recognize the multimedia sequence according to the key media features.

[0080] Optionally, the number of decoders can be one or more. When the number of decoders is one, the specific manner in which the computer device retrieves the key media features of the multimedia sequence from the media features corresponding to the N media sequence blocks through the queryer and the decoder can include: obtaining, by the decoder, a retrieval attention parameter of the decoder. Specifically, the computer device can obtain, by the decoder, a decoding start token vector, which is obtained by vector conversion on a decoding start token. The decoding start token is used to indicate that the decoder starts to perform a decoding operation to generate a decoding media feature, such as an SOS. Further, the decoder can include a third cross-attention layer, and the computer device can perform linear transformation on the decoding start token vector through a query weight matrix in the third cross-attention layer to obtain a processed decoding start token vector. Specifically, the computer device can multiply the query weight matrix in the third cross-attention layer and the decoding start token vector to obtain the processed decoding start token vector.

[0081] Further, the computer device can perform linear transformation on the processed decoding start token vector through a key weight matrix in the third cross-attention layer to obtain the retrieval attention parameter of the decoder. Specifically, the computer device can obtain the transpose of the key weight matrix in the third cross-attention layer, and multiply the processed decoding start token vector and the transpose of the key weight matrix in the third cross-attention layer to obtain the retrieval attention parameter of the decoder.

[0082] It can be understood that when the number of decoders is one, the decoding start token vector can be used as the decoder state of the decoder, and let h d indicates the decoder state, W q is the query weight matrix in the third cross-attention layer in the decoder, W kThe key weight matrix in the third cross-attention layer is denoted as W, and the retrieval attention parameter of the one decoder can be h d *W q *(W k ) T Further, the computer device can retrieve, through the retriever, the initial key media feature corresponding to the one decoder from the media features corresponding to the N media sequence blocks according to the retrieval attention parameter of the one decoder, and determine the initial key media feature corresponding to the one decoder as the key media feature of the multimedia sequence.

[0083] Optionally, the number of decoders is M, M is a positive integer greater than 1, that is, the number of decoders is multiple, and the specific manner in which the computer device retrieves, through the retriever and the decoders, the key media feature of the multimedia sequence from the media features corresponding to the N media sequence blocks can include: determining, through the i+1th decoder, the retrieval attention parameter of the i+1th decoder according to the decoding media feature output by the i th decoder; the decoding media feature output by the i th decoder is obtained by decoding the initial key media feature corresponding to the i th decoder through the i th decoder; i is a positive integer less than M. Retrieving, through the retriever, the initial key media feature corresponding to the i+1th decoder from the media features corresponding to the N media sequence blocks according to the retrieval attention parameter of the i+1th decoder. The initial key media feature corresponding to the Mth decoder is determined as the key media feature of the multimedia sequence.

[0084] Specifically, taking the i+1th decoder in the M decoders as an example, the computer device can determine, through the i+1th decoder, the retrieval attention parameter of the i+1th decoder according to the decoding media feature output by the i th decoder. Wherein, i is a positive integer less than M. Specifically, when i = 1, the decoding media feature output by the 1st decoder is obtained by decoding the initial key media feature corresponding to the 1st decoder through the 1st decoder. The initial key media feature of the 1st decoder is retrieved from the media features corresponding to the N media sequence blocks through the retriever according to the retrieval attention parameter of the 1st decoder. Wherein, the retrieval attention parameter of the 1st decoder is determined based on a decoding start token vector, the decoding start token vector is obtained by vector conversion on a decoding start token, and the decoding start token is used to indicate that the decoder starts decoding operation to generate decoding media feature, such as the decoding start token can be SOS. The determination manner of the retrieval attention parameter of the 1st decoder can refer to the determination manner of the retrieval attention parameter of the one decoder when the decoder is one, and the embodiments of the present application will not be repeated here.

[0085] Specifically, when i is a positive integer greater than 1 and less than M, the decoded media feature output by the i-th decoder is obtained by decoding, by the i-th decoder, the initial key media feature corresponding to the i-th decoder. The initial key media feature corresponding to the i-th decoder is retrieved from the media features respectively corresponding to the N media sequence blocks by the retriever according to the search attention parameter corresponding to the i-th decoder. The search attention parameter corresponding to the i-th decoder is determined by the i-th decoder according to the decoded media feature output by the (i-1)-th decoder.

[0086] Further, the computer device can retrieve, by the retriever, the initial key media feature corresponding to the (i+1)-th decoder from the media features respectively corresponding to the N media sequence blocks according to the search attention parameter of the (i+1)-th decoder. It can be understood that when the number of decoders is M, each of the M decoders needs to retrieve the corresponding initial key media feature from the media features respectively corresponding to the N media sequence blocks by the retriever. In this way, the initial key media features respectively corresponding to the M decoders are obtained, and the computer device can determine the initial key media feature corresponding to the M-th decoder in the M decoders as the key media feature of the multimedia sequence. The initial key media feature corresponding to the M-th decoder is retrieved from the media features respectively corresponding to the N media sequence blocks by the retriever according to the search attention parameter of the M-th decoder, and the search attention parameter of the M-th decoder is determined according to the decoded media feature output by the (M-1)-th decoder.

[0087] It can be understood that, taking the first decoder and the second decoder in the M decoders as an example, the first decoder and the second decoder are adjacent, and the second decoder is the next decoder of the first decoder. The computer device can generate the retrieval attention parameter of the first decoder according to the decoding start token vector through the first decoder. Then, through the retriever, the retrieval attention parameter of the first decoder retrieves the initial key media feature corresponding to the first decoder from the media features corresponding to the N media sequence blocks respectively. The computer device can perform decoding processing on the initial key media feature corresponding to the first decoder through the first decoder to obtain the decoding media feature output by the first decoder. Further, the computer device can determine the retrieval attention parameter of the second decoder according to the decoding media feature output by the first decoder through the second decoder. Further according to the retriever, according to the retrieval attention parameter of the second decoder, the initial key media feature corresponding to the second decoder is retrieved from the media features corresponding to the N media sequence blocks respectively. Through the second decoder, the initial key media feature corresponding to the second decoder is decoded to obtain the decoding media feature output by the second decoder. In this way, the initial key media feature corresponding to the Mth decoder in the M decoders is obtained, and the initial key media feature corresponding to the Mth decoder is determined as the key media feature of the multimedia sequence.

[0088] Optionally, when i is a positive integer greater than 1 and less than M, taking the (i+1)th decoder in the M decoders as an example, the process of determining the retrieval attention parameter is specifically described below. Specifically, the (i+1)th decoder includes a first cross-attention layer, and the computer device can determine the retrieval attention parameter of the (i+1)th decoder according to the decoding media feature output by the ith decoder through the (i+1)th decoder. The specific manner can include: through the first cross-attention layer, according to the query weight matrix in the first cross-attention layer, linearly transforming the decoding media feature output by the ith decoder to obtain the processed decoding media feature. According to the key weight matrix in the first cross-attention layer, linearly transforming the processed decoding media feature to obtain the retrieval attention parameter of the (i+1)th decoder.

[0089] Specifically, the computer device can perform linear transformation on the decoding media feature output by the i-th decoder according to the query weight matrix in the first cross-attention layer to obtain a processed decoding media feature. Wherein, the computer device can multiply the query weight matrix in the first cross-attention layer with the decoding media feature output by the i-th decoder to obtain the processed decoding media feature. Further, the computer device performs linear transformation on the processed decoding media feature according to the key weight matrix in the first cross-attention layer to obtain the retrieval attention parameter of the i+1-th decoder. Wherein, the computer device can obtain the transpose of the key weight matrix in the first cross-attention layer, multiply the processed decoding media feature with the transpose of the key weight matrix in the first cross-attention layer to obtain the retrieval attention parameter of the i+1-th decoder.

[0090] Wherein, when i is a positive integer greater than 1 and less than M, the computer device can take the decoding media feature output by the i-th decoder as the decoder state h d of the i+1-th decoder. d Further, the computer device can multiply the decoding media feature output by the i-th decoder with the query weight matrix W q of the cross-attention layer in the i+1-th decoder to obtain a processed decoding media feature (i.e., h d *W q ). Further, the computer device can multiply the processed decoding media feature (i.e., h d *W q ) with the transpose of the key weight matrix W k of the cross-attention layer in the i+1-th decoder to obtain the retrieval attention parameter (i.e., h d *W q *(W k ) T ) of the i+1-th decoder. It can be understood that when i is a positive integer greater than 1 and less than M, the computer device takes the decoding media feature output by the i-th decoder as the decoder state of the i+1-th decoder. When i=1, the decoder state of the first decoder is the decoding start token vector. In this way, when i is a positive integer greater than 1 and less than M, the retrieval attention parameter of the i-th decoder, the retrieval attention parameter of the M-1-th decoder, the retrieval attention parameter of the M-th decoder, and the parameters of other decoders can all refer to the determination process of the retrieval attention parameter of the i+1-th decoder.

[0091] Optionally, taking the i+1th decoder in the M decoders as an example, the process of determining the initial key media feature is specifically described below. The computer device retrieves the initial key media feature corresponding to the i+1th decoder from the N media features corresponding to the N media sequence blocks according to the retrieval attention parameter of the i+1th decoder through the queryer. The specific manner can include: determining the importance degrees of the N media features corresponding to the N media sequence blocks according to the retrieval attention parameter of the i+1th decoder through the queryer. The media feature corresponding to the maximum importance degree in the N media features is determined as the initial key media feature corresponding to the i+1th decoder.

[0092] Specifically, the computer device can determine the importance degrees of the N media features corresponding to the N media sequence blocks according to the retrieval attention parameter of the i+1th decoder through the queryer. Further, the media feature corresponding to the maximum importance degree in the N media features is determined as the initial key media feature corresponding to the i+1th decoder. It can be understood that the computer device can retrieve the initial key media feature corresponding to each decoder from the N media features corresponding to the N media sequence blocks according to the retrieval attention parameter of each decoder through the queryer.

[0093] Optionally, the specific manner in which the computer device determines the importance degrees of the N media features according to the retrieval attention parameter of the i+1th decoder through the queryer can include: obtaining, through the queryer, a parameter matrix reflecting the retrieval attention parameter of the i+1th encoder and a feature matrix reflecting the jth media feature in the N media features. The transpose of the parameter matrix and the feature matrix are multiplied to obtain an association degree matrix reflecting the association degree between the jth media feature and media recognition. The importance degree of the jth media feature is determined according to the association degree matrix.

[0094] Specifically, taking the feature matrix corresponding to the jth media feature in the N media features as an example, the process of determining the importance degree of the jth media feature is specifically described, and the j is a positive integer less than or equal to N. The computer device can obtain, through the queryer, a parameter matrix reflecting the retrieval attention parameter of the i+1th encoder (i.e., h d *W q *W k T ), and a feature matrix reflecting the jth media feature in the N media features. Further, the computer device can obtain the transpose of the feature matrix corresponding to the jth media feature, multiply the parameter matrix corresponding to the i+1th encoder and the transpose of the feature matrix corresponding to the jth media feature to obtain an association degree matrix reflecting the association degree between the jth media feature and media recognition.

[0095] It can be understood that the computer device can take the feature matrix corresponding to the jth media feature as the hidden state h e of the encoder, take the parameter matrix corresponding to the retrieval attention parameter of the i+1th encoder (i.e., h d *W q *W k T , and multiply the transpose h e of the feature matrix corresponding to the jth media feature to obtain the association degree matrix (i.e., h T *W d *W q k T *h e T reflecting the association degree between the jth media feature and the media identification. Further, the computer device can determine the importance degree corresponding to the jth media feature according to the association degree matrix through the projection layer in the retriever. Specifically, the computer device can obtain the importance degree corresponding to the jth media feature by performing sum-averaging processing on the association degree matrix through the projection layer in the retriever.

[0096] Specifically, in the process of decoding the initial key media feature by the decoder, the attention key (i.e., the product of the initial key media feature and the key weight matrix of the cross-attention layer in the decoder) and the attention value (i.e., the product of the initial key media feature and the value weight matrix of the cross-attention layer in the decoder) of the cross-attention layer in each decoder need to be obtained according to the initial key media feature of each decoder, and a separate data storage area is constructed for the attention key and the attention value of the cross-attention layer in each decoder. In this way, a total of 2*L*H data storage areas are needed, where L is the number of layers of the decoder and H is the number of attention heads of the cross-attention layer in each decoder. For each attention head of the cross-attention layer in each decoder, constructing a separate data storage area is both time-consuming and space-consuming.

[0097] Therefore, the embodiment of the present application provides an efficient computing method, which can realize decoding of the initial key media feature by all decoders through one data storage area, thereby reducing time consumption and saving space. Specifically, in the cross-attention layer of the traditional transformer structure, the calculation formula of a single head attention in the cross-attention layer is as follows (1).

[0098]

[0099] Wherein, Attn(Q, K, V) in formula (1) is to extract effective media features (i.e. the features needed to pay attention to in the initial key media features) from the initial key media features, Softmax is a function commonly used in machine learning and deep learning, mainly used to normalize the classification results to form a probability distribution. Q is the decoder state h d of the decoder d *W q , wherein when the number of decoders is one, the decoder state of the decoder is the decoding start token vector; when the number of decoders is M, the decoder state of the first decoder in the M decoders is the decoding start token vector, and the decoder state of the other decoders (i.e. the decoders other than the first decoder) is the decoding media feature output by the previous decoder. Wherein, K is the hidden state (i.e. the media feature, h e ) output by the encoder multiplied by the key weight matrix (i.e. the key weight matrix) in the cross-attention layer, i.e. W k *h e , K T is the transpose of K. Wherein, V is the hidden state (i.e. the media feature, h e ) output by the encoder multiplied by the value weight matrix (i.e. the value weight matrix) in the cross-attention layer, i.e. W v *h e . Wherein, when the number of encoders is one, the hidden state output by the encoder is the media feature output by the one encoder, and when the number of encoders is multiple, the hidden state output by the encoder is the media feature output by the last encoder. Wherein, d k is the feature dimension.

[0100] It can be understood that the purpose of the queryer in the embodiment of the application is to retrieve an initial key media feature for each decoder from the media features corresponding to the N media sequence blocks output by the encoder, to determine a group of final K values, i.e. K best , and then maximize QK T , so as to realize attention calculation from the media feature of the most relevant media sequence block in the entire multimedia sequence. The embodiment of the application reconstructs the attention calculation process of QK T , and obtains the following formula (2).

[0101] QK T =(h d *W q )(W k *h e ) T

[0102] =(h d *W q )W k T *h e T (2)

[0103] =(h d *W q W k T )*h e T

[0104] It can be understood that the retrieval process (i.e., the retrieval purpose) of the query can be equivalent to retrieving an initial key media feature from the media features corresponding to the N media sequence blocks, so that the cross attention layer (h d *W q W k T )*h e T Get the maximum value.

[0105] The embodiment of this application is to QK T The advantages of the reconstructed attention calculation process include but are not limited to the following: First, there is no need to build a separate data storage area for the attention key and attention value of each cross-attention layer in the decoder. Instead, only the media features (i.e., hidden layer states) of the N media sequence blocks output by the encoder can be stored and the query weight matrix (i.e., W) of each attention head in the cross-attention layer can be used to store the media features of the N media sequence blocks output by the encoder. q ) and the key weight matrix (ie W k ) to project the decoder input (i.e., the decoder state of the decoder) onto (h d *W q W k T ) is used as the decoder’s retrieval attention parameter. According to the query, the media features of the N media sequence blocks are retrieved according to the decoder’s retrieval attention parameter, and the initial key media features of each decoder are retrieved. Second, V can be easily calculated by the initial key media features of each decoder, so the attention key and attention value of the cross-attention layer in each decoder are stored in different data storage areas respectively. It can be seen that the embodiment of the present application has a significant impact on QK T After the attention calculation process is reconstructed, only one data storage area is needed to store the media features (i.e., hidden layer states) of N media sequence blocks, so that all decoders can decode the initial key media features, which can reduce time consumption and space occupation.

[0106] By the above method, the embodiment of the application transforms each attention head in the cross-attention layer of each decoder into a retrieval enhanced attention head, almost without increasing the time required by the calculation process, while also reducing the storage space occupied, without the need to retrain the data recognition model, so that the data recognition model can process multimedia sequences of arbitrary sequence length, improve the applicability of the data recognition model, and at the same time improve the recognition accuracy of the multimedia sequence.

[0107] In S104, the multimedia sequence is recognized according to the key media feature, and a multimedia recognition result of the multimedia sequence is obtained.

[0108] Specifically, the computer device can recognize the multimedia sequence according to the key media feature in the multimedia sequence, and obtain a multimedia recognition result of the multimedia sequence. The multimedia recognition result is used to reflect at least one of a multimedia attribute and a multimedia key content of the multimedia sequence. The multimedia attribute can be used to reflect the category, nature, etc. of the multimedia sequence, and the multimedia key content can be the media meaning, media summary, media context rewriting, etc. of the multimedia sequence. For example, when the multimedia sequence is text sequence data, the multimedia recognition result of the text sequence data can be the text summary, context rewriting result, archiving classification category, etc. of the text sequence data.

[0109] Optionally, the data recognition model further includes an identification layer, which can include a Linear sub-layer and a Softmax sub-layer. The Linear sub-layer is a fully connected network layer, which is used to perform feature mapping on the output of the decoder (i.e. the decoded media feature), and map the output of the decoder to a Lotigs (i.e. probability) vector. Through the Softmax sub-layer, the Lotigs vector output by the Linear sub-layer is identified to obtain the multimedia recognition result of the multimedia sequence. Specifically, the computer device can recognize the multimedia sequence according to the key media feature to obtain the multimedia recognition result of the multimedia sequence. The specific manner can include: through the Mth decoder, the initial key media feature corresponding to the Mth decoder is decoded to obtain the decoded media feature output by the Mth decoder. Through the identification layer, the decoded media feature output by the Mth decoder is identified to obtain the multimedia recognition result of the multimedia sequence.

[0110] Specifically, the computer device can decode the initial key media feature corresponding to the Mth decoder by the Mth decoder to obtain a decoded media feature output by the Mth decoder. Further, the computer device can identify a Linear sub-layer in the layer and linearly process the decoded media feature output by the Mth decoder to obtain a linearly processed decoded media feature. Further, the computer device can identify a Softmax sub-layer in the layer and calculate the linearly processed decoded media feature to obtain a multimedia recognition result of the multimedia sequence.

[0111] Optionally, the Mth decoder includes a second cross-attention layer and a decoding layer, and the computer device decodes the initial key media feature corresponding to the Mth decoder by the Mth decoder to obtain a decoded media feature output by the Mth decoder in the following manner: the second cross-attention layer extracts cross-attention features from the initial key media feature corresponding to the Mth decoder to obtain effective media features; and the decoding layer decodes the effective media features to obtain the decoded media feature output by the Mth decoder.

[0112] Specifically, the computer device can extract cross-attention features from the initial key media feature corresponding to the Mth decoder by the second cross-attention layer to obtain effective media features. Further, the decoding layer can include a third fusion layer, a second feedforward neural network layer, and a fourth fusion layer, and the third fusion layer and the fourth fusion layer can both be Add&normalize. The computer device can perform residual connection (i.e., Add) on the effective media features and the decoder state of the Mth decoder (i.e., the decoded media feature output by the M-1th decoder) by the third fusion layer to obtain initial fused effective media features. The initial fused effective media features are further normalized (i.e., normalize) to obtain fused effective media features. Further, the computer device can linearly transform the fused effective media features by the second feedforward neural network layer to obtain transformed effective media features, and then fuse the transformed effective media features and the fused effective media features by the fourth fusion layer to obtain the decoded media feature output by the Mth decoder.

[0113] Optionally, the computer device can obtain, by the second cross-attention layer, a correlation matrix reflecting the correlation between the initial key media feature corresponding to the Mth decoder and the media recognition. The computer device can perform linear transformation on the initial key media feature corresponding to the Mth decoder according to the value weight matrix in the second cross-attention layer to obtain a processed initial key media feature. The computer device can perform effective feature extraction on the processed initial key media feature according to the correlation matrix of the initial key media feature corresponding to the Mth decoder by the second cross-attention layer to obtain an effective media feature.

[0114] Specifically, the computer device can obtain, by the second cross-attention layer, a correlation matrix reflecting the correlation between the initial key media feature corresponding to the Mth decoder and the media recognition. It can be understood that the second cross-attention layer in the Mth decoder pre-computes the retrieval attention parameters of the Mth encoder, i.e., (h d *W q W k T ), and multiplies the parameter matrix of the retrieval attention parameters of the Mth encoder with the feature matrix corresponding to each media feature in the N media features to obtain a correlation matrix corresponding to the correlation between each media feature and the media recognition, and further determines the importance of the corresponding media feature according to the correlation matrix corresponding to each media feature. The initial key media feature corresponding to the Mth decoder is the media feature with the maximum importance in the N media features, and therefore, the correlation matrix corresponding to the correlation between the initial key media feature corresponding to the Mth decoder and the media recognition is calculated by the queryer, and the second cross-attention layer can directly obtain the correlation matrix reflecting the correlation between the initial key media feature corresponding to the Mth decoder and the media recognition.

[0115] Further, the computer device can perform linear transformation on the initial key media feature corresponding to the Mth decoder according to the value weight matrix in the second cross-attention layer to obtain a processed initial key media feature. Specifically, the computer device can perform multiplication processing on the value weight matrix in the second cross-attention layer and the initial key media feature corresponding to the Mth decoder to obtain the processed initial key media feature. Further, the computer device can perform effective feature extraction on the processed initial key media feature according to the correlation matrix of the initial key media feature corresponding to the Mth decoder through the second cross-attention layer to obtain an effective media feature. Specifically, the computer device can perform normalization processing on the correlation matrix of the initial key media feature corresponding to the Mth decoder through the second cross-attention layer to obtain a normalized correlation matrix, and perform multiplication processing on the normalized correlation matrix and the processed initial key media feature to obtain the effective media feature.

[0116] As shown in Figure 5 , Figure 5 is a schematic diagram of media recognition of text sequence data with a sequence length greater than a limited recognition length provided by an embodiment of the present application, as shown in Figure 5 , taking a multimedia sequence as the text sequence data 50a as an example. When the sequence length of the text sequence data 50a is greater than the limited recognition length of the data recognition model, the computer device can perform block processing on the text sequence data 50a according to the limited recognition length of the data recognition model to obtain N text blocks 50b, i.e., text block 1, text block 2, text block 3, …, and text block n. Wherein, the sequence length of each text block in the N text blocks 50b is less than or equal to the limited recognition length of the data recognition model. Further, since the sequence length of each text block in the N text blocks 50b is less than or equal to the limited recognition length of the data recognition model, the computer device can input the N text blocks 50b into the encoder 50c. Through the encoder 50c, each text block in the N text blocks 50b is encoded to obtain the hidden state 50d (i.e., the text feature) corresponding to the N text blocks 50b, i.e., the hidden state 1 corresponding to the text block 1, the hidden state 2 corresponding to the text block 2, the hidden state 3 corresponding to the text block 3, …, and the hidden state n corresponding to the text block n. Wherein, each text block includes a group of tokens in the multimedia sequence.

[0117] Further, the computer device can construct a data storage area 50e for the hidden states 50d corresponding to the N text blocks respectively, generate an index of the hidden state corresponding to each text block in the data storage area 50e, and use the index of the hidden state corresponding to each text block to obtain the hidden state corresponding to each text block. The hidden state corresponding to each text block and the index of the hidden state corresponding to each text block are stored in the GPU memory area or the CPU memory area, and the initial key hidden state (i.e., the initial key media feature) corresponding to the decoder 50g is efficiently retrieved in sub-linear time through the queryer 50f and the decoder 50g. Specifically, the index of the initial key hidden state corresponding to the decoder 50g can be calculated by the queryer 50f (which can be a KNN search algorithm), and then the initial key hidden state corresponding to the decoder 50g is obtained.

[0118] When the number of decoders 50g is one, the retrieval attention parameter (i.e., h d *W q W k T ) of the one decoder can be calculated in advance through the cross-attention layer in the one decoder 50g. The initial key hidden state (i.e., the most relevant hidden state h e ) of the one decoder 50g is obtained from the hidden states 50d corresponding to the N text blocks respectively in the data storage area by the queryer 50f according to the retrieval attention parameter of the one decoder 50g. Further, the initial key hidden state of the one decoder 50g is taken as the key media feature of the text sequence data, and the computer device can perform decoding processing on the key media feature of the text sequence data through the cross-attention layer and the decoding layer in the one decoder 50g to obtain the multimedia recognition result 50h of the text sequence data.

[0119] When the number of the decoders 50g is M, M is a positive integer greater than 1, and taking the i+1th decoder in the M decoders as an example, i is a positive integer less than M, the computer device can determine the retrieval attention parameter of the i+1th decoder according to the decoded media feature output by the i th decoder through the i+1th decoder. Further, the initial key hidden state corresponding to the i+1th decoder is retrieved from the hidden states 50d corresponding to the N text blocks respectively according to the retrieval attention parameter of the i+1th decoder through the retriever 50f (i.e., a set of the most relevant tokens are retrieved from the multimedia sequence). In this way, the initial key hidden states corresponding to the M decoders are retrieved from the hidden states 50d corresponding to the N text blocks respectively through the cross-attention layers in the decoders 50g and the retriever. The initial key hidden state of the Mth decoder (i.e., the last decoder) in the M decoders is determined as the key media feature in the text sequence data, and the multimedia recognition result 50h of the text sequence data is obtained by decoding the key media feature in the text sequence data according to the cross-attention layer and the decoding layer in the Mth decoder.

[0120] It can be understood that the embodiments of the present application optimize the trained recognition model through the retriever for feature retrieval to obtain a data recognition model, and the trained recognition model can be an encoder-decoder model of a pre-trained Transformer architecture. In this way, the data recognition model can accurately process multimedia sequences of any sequence length, and even if the data recognition model can process multimedia sequences of any sequence length, the context length of the multimedia sequence that the data recognition model can perceive is greatly expanded. The decoder in the data recognition model will focus on the top-level hidden state of the encoder (i.e., the media feature output by the encoder), and the encoder will usually truncate the input and only encode the first k tokens of the multimedia sequence, so that the key information of the multimedia sequence will be lost. The data recognition model obtained by the embodiments of the present application retrieves the initial key hidden state of the cross-attention layer in each decoder from the N hidden states (i.e., N media features) corresponding to the multimedia sequence through the retriever (which can be a KNN retrieval module), and only focuses on the initial key hidden state. In this way, retrieval can be performed from the entire multimedia sequence without truncating or deleting the multimedia sequence.

[0121] Wherein, a data storage area can be constructed for N hidden states, which can be located in a GPU memory area or a CPU memory area, and can perform a sub-linear query. In this way, a queryer can be implemented to efficiently retrieve the initial key hidden state of the cross-attention layer in each decoder from the data storage area. At the same time, by using the characteristic of the softmax function in the cross-attention layer in each decoder, which is sensitive to the maximum value, the more important part of the multimedia sequence (i.e. N hidden states) can be retrieved for key calculation. The optimization method of the data recognition model can be applied to any Encoder-Decoder model based on the transformer structure, so that the data recognition model can process input of unlimited length without the need for retraining.

[0122] It can be understood that, by using the queryer, the trained recognition model is optimized, and before entering the cross-attention layer in the decoder, the queryer performs KNN search query in the N hidden states (i.e. media features of N media sequence blocks) stored in the external data storage area. The cross-attention layer of each decoder can include multiple attention heads, so that each attention head in the cross-attention layer selects a most relevant context window (i.e. a most relevant hidden state) from the N hidden states of the entire multimedia sequence for attention calculation. The data recognition model optimization method in the present application can be directly applied to any existing Encoder-Decoder model based on the transformer structure, and can accurately recognize the multimedia sequence without any further training. Of course, if the optimized data recognition model can be further fine-tuned, the performance of the data recognition model can be further improved.

[0123] Specifically, referring to Table 1, several representative classification models and generation models based on the Transformer architecture are selected for comparative experiments, and multimedia sequences of several sequence lengths of 512, 2048, more than 2048, 4096 and more than 4096 are selected for testing. Among them, the classification model uses the accuracy index, and the generation model uses the ROUGE-L index.

[0124] Model Max length Accuracy ROUGE-L Archive (classification) 512 92.8％ \ Archive (classification) + retrieval attention 512 92.2％ Archive (classification) 2048 90.1％ \ Archive (classification) + retrieval attention 2048 91.9％ Archive (classification) 2048+ 84.7％ \ Archive (classification) + retrieval attention 2048+ 91.3％ \ Summary (generation) 4096 \ 0.54 Summary (generation) + retrieval attention 4096 \ 0.52 Summary (generation) 4096+ \ 0.39 Summary (generation) + retrieval attention 4096+ \ 0.50

[0125] Table 1

[0126] It can be seen from the experimental data in Table 1 that in the classification model, when the input multimedia sequence is less than 2048 in length, the input length of the generation model itself is sufficient to cope with it, and the accuracy index is relatively stable, but when the length exceeds 2048, the accuracy of the generation model decreases sharply. After adding the retrieval attention of the query, not only the index performance of the shorter input length is basically the same as the original generation model, but also the index basically remains stable even in the summary generation task when the input increases and exceeds 2048, with a decrease of less than 5%, indicating that the retrieval attention of the query plays a significant role in the super-long multimedia sequence. Moreover, this method does not require retraining the data recognition model, greatly reducing the optimization cost.

[0127] The embodiment of the present application proposes a method for expanding the input sequence length of a general data recognition model, which optimizes the trained recognition model through a query for feature retrieval to obtain a data recognition model. Since the role of the query for feature retrieval is to retrieve key media features from the media features corresponding to the multimedia sequence to be processed, it does not change the code in the data recognition model or add additional learning weights, so it is not necessary to retrain the data recognition model to enable the data recognition model to process multimedia sequences of any sequence length and improve the recognition accuracy of multimedia sequences. Specifically, when the sequence length of the multimedia sequence to be processed is greater than the limited recognition length of the data recognition model, the multimedia sequence can be blocked according to the limited recognition length to obtain N media sequence blocks. It can be understood that the sequence length of the N media sequence blocks is less than or equal to the limited recognition length, so the N media sequence blocks can be input into the data recognition model, and the features of the N media sequence blocks are encoded by the encoder in the data recognition model to obtain the media features corresponding to the N media sequence blocks, respectively. Further, the key media features of the multimedia sequence are retrieved from the media features corresponding to the N media sequence blocks by the query and the decoder in the data recognition model. The multimedia sequence is recognized according to the key media features to obtain a multimedia recognition result of the multimedia sequence, which is used to reflect at least one of the multimedia attributes and the multimedia key content of the multimedia sequence. It can be seen that the present application does not use the truncation method of the front sequence part of the multimedia sequence or delete part of the sequence of the multimedia sequence to shorten the length of the multimedia sequence, but inputs the entire multimedia sequence into the data recognition model and extracts the key media features from the N media features corresponding to the multimedia sequence. In this way, the loss of key information of the multimedia sequence caused by truncation or deletion can be avoided, the data recognition model can accept the input of multimedia sequences of any sequence length, and accurate recognition of the multimedia sequence can be achieved without retraining, thereby improving the accuracy of multimedia sequence recognition.

[0128] Further, please refer to Figure 6 , Figure 6 is a flowchart of a data processing method provided by an embodiment of the present application. As shown in Figure 6 , the method can be executed by any terminal device in Figure 1 , or can be executed by the server 10 in Figure 1 , or can be executed by the terminal device and the server in Figure 1 together, and the device used to execute the data processing method in the present application can be collectively referred to as a computer device. The data processing method can include but is not limited to the following steps:

[0129] S201, when the sequence length of the multimedia sequence to be processed is greater than the limited recognition length of the data recognition model, the multimedia sequence is processed by block according to the limited recognition length, and N media sequence blocks are obtained.

[0130] S202, the N media sequence blocks are feature encoded by an encoder, and N media sequence blocks respectively corresponding media features are obtained.

[0131] S203, the key media features of the multimedia sequence are retrieved from the N media sequence blocks respectively corresponding media features by a queryer for feature retrieval and a decoder.

[0132] S204, the multimedia sequence is recognized according to the key media features, and the multimedia recognition result of the multimedia sequence is obtained.

[0133] Specifically, the contents of steps S201-S204 in the embodiments of the present application can refer to the contents of steps S101-S104 described above, and the embodiments of the present application will not be repeated here.

[0134] S205, the labeled recognition result of the multimedia sequence is obtained.

[0135] Specifically, the embodiments of the present application obtain the data recognition model by adding the queryer to the trained recognition model, without changing the codes of the encoder and the decoder in the data recognition model, and without adding additional learning weights, so that the data recognition model does not need to be retrained, and the multimedia sequence can be accurately recognized. However, if the data recognition model is further trained, the model performance of the data recognition model can be further improved. Specifically, in the process of training the data recognition model, the computer device can obtain the labeled recognition result of the multimedia sequence, which can be artificially labeled, and the labeled recognition result can be used to reflect the real media category, real media meaning, real media summary, etc. of the multimedia sequence.

[0136] S206, determine a model loss of the data recognition model according to the annotation recognition result of the multimedia sequence and the multimedia recognition result of the multimedia sequence obtained by the data recognition model.

[0137] Specifically, the computer device can determine the model loss of the data recognition model according to the annotation recognition result of the multimedia sequence and the multimedia recognition result of the multimedia sequence obtained by the data recognition model (i.e., the recognition model after adding the queryer).

[0138] S207, adjust the model parameters of the data recognition model according to the model loss function and the model loss of the data recognition model, and obtain an adjusted data recognition model.

[0139] Further, the computer device can obtain the model loss function of the data recognition model, determine the parameter adjustment gradient of the data recognition model according to the model loss function and the model loss of the data recognition model, and further adjust the model parameters of the data recognition model according to the parameter adjustment gradient to obtain an adjusted data recognition model. Optionally, the computer device can determine the parameter adjustment gradient of all network layers in the data recognition model according to the model loss function and the model loss of the data recognition model, and adjust the model parameters in all network layers according to the parameter adjustment gradient. Optionally, the computer device can determine the parameter adjustment gradient of the key network layer in the data recognition model according to the model loss function and the model loss of the data recognition model, and adjust the model parameters of the key network layer according to the parameter adjustment gradient. The key network layer can be determined from all network layers in the data recognition model. In this way, further incremental training of the data recognition model can further improve the model performance of the data recognition model.

[0140] The embodiment of the present application provides a method for expanding the input sequence length of a general data recognition model. A queryer for feature retrieval is added to a trained recognition model to obtain a data recognition model. The data recognition model comprises a decoder and an encoder. The queryer is used to retrieve key media features from media features corresponding to a multimedia sequence to be processed. The code of the encoder and the decoder in the data recognition model is not changed, and no additional learning weight is added, so that the recognition accuracy of the data recognition model for the media sequence is not affected. Specifically, when the sequence length of the multimedia sequence to be processed is greater than the limited recognition length of the data recognition model, the multimedia sequence can be divided into N media sequence blocks according to the limited recognition length. It can be understood that the sequence length of the N media sequence blocks is less than or equal to the limited recognition length, so that the N media sequence blocks can be input into the data recognition model, the N media sequence blocks are feature coded by the encoder in the data recognition model, and media features corresponding to the N media sequence blocks are obtained. Further, the key media features of the multimedia sequence are retrieved from the media features corresponding to the N media sequence blocks by the queryer and the decoder in the data recognition model. The multimedia sequence is recognized according to the key media features, and a multimedia recognition result of the multimedia sequence is obtained. It can be seen that the present application does not need to use the truncation method of the front sequence part in the multimedia sequence, or delete part of the sequence in the multimedia sequence to shorten the length of the multimedia sequence, but input the whole multimedia sequence into the data recognition model, and extract the key media features from the N media features corresponding to the multimedia sequence. In this way, the loss of key information of the multimedia sequence caused by truncation or deletion can be avoided, and the data recognition model can accept the input of the multimedia sequence with any sequence length. Meanwhile, the model performance of the data recognition model can be further improved by further incremental training of the data recognition model.

[0141] Further, please refer to Figure 7 , Figure 7 is a structural schematic diagram of a data processing apparatus provided by the embodiment of the present application. The data processing apparatus can be a computer program (including program code) running in a computer device, for example, the data processing apparatus is an application software; the data processing apparatus can be used to execute corresponding steps in the method provided by the embodiment of the present application. As shown in Figure 7 , the data processing apparatus can be any blockchain node in a blockchain network. The data processing apparatus can comprise a blocking module 11, an encoding module 12, a retrieval module 13, an identification module 14, an acquisition module 15, a determination module 16 and an adjustment module 17.

[0142] The block module 11 is configured to perform block processing on the multimedia sequence according to a limited recognition length when a sequence length of the multimedia sequence to be processed is greater than the limited recognition length of the data recognition model, to obtain N media sequence blocks; the data recognition model comprises a decoder and an encoder, and N is an integer greater than 1.

[0143] The encoding module 12 is configured to perform feature coding on the N media sequence blocks by the encoder to obtain media features corresponding to the N media sequence blocks, respectively.

[0144] The retrieval module 13 is configured to retrieve key media features of the multimedia sequence from the media features corresponding to the N media sequence blocks by a queryer for feature retrieval and the decoder.

[0145] The recognition module 14 is configured to recognize the multimedia sequence according to the key media features to obtain a multimedia recognition result of the multimedia sequence; the multimedia recognition result is used to reflect at least one of a multimedia attribute and a multimedia key content of the multimedia sequence.

[0146] The number of the decoders is M, and M is a positive integer greater than 1; the retrieval module 13 is specifically configured to:

[0147] The retrieval attention parameter of the i+1th decoder is determined according to the decoding media feature output by the ith decoder by the i+1th decoder; the decoding media feature output by the ith decoder is obtained by decoding the initial key media feature corresponding to the ith decoder by the ith decoder; i is a positive integer less than M;

[0148] The initial key media feature corresponding to the i+1th decoder is retrieved from the media features corresponding to the N media sequence blocks by the queryer for feature retrieval according to the retrieval attention parameter of the i+1th decoder;

[0149] The initial key media feature corresponding to the Mth decoder is determined as the key media feature of the multimedia sequence.

[0150] The i+1th decoder comprises a first cross-attention layer; the retrieval module 13 is further specifically configured to:

[0151] The decoding media feature output by the ith decoder is linearly transformed by the first cross-attention layer according to a query weight matrix in the first cross-attention layer to obtain a processed decoding media feature;

[0152] The processed decoding media feature is linearly transformed according to a key weight matrix in the first cross-attention layer to obtain the retrieval attention parameter of the i+1th decoder.

[0153] The retrieval module 13 is further configured to:

[0154] The queryer for feature retrieval is configured to determine the importance of each of the N media features according to the retrieval attention parameter of the i+1th decoder.

[0155] The media feature with the maximum importance among the N media features is determined as the initial key media feature corresponding to the i+1th decoder.

[0156] The retrieval module 13 is further configured to:

[0157] The queryer for feature retrieval is configured to obtain a parameter matrix reflecting the retrieval attention parameter of the i+1th encoder and a feature matrix reflecting the jth media feature among the N media features; j is a positive integer less than or equal to N.

[0158] The parameter matrix and the transpose of the feature matrix are multiplied to obtain an association degree matrix reflecting the association degree between the jth media feature and the media recognition.

[0159] The importance of the jth media feature is determined according to the association degree matrix.

[0160] The data recognition model further includes a recognition layer.

[0161] The recognition module 14 is configured to include:

[0162] The Mth decoder is configured to decode the initial key media feature corresponding to the Mth decoder to obtain a decoded media feature output by the Mth decoder.

[0163] The recognition layer is configured to recognize the decoded media feature output by the Mth decoder to obtain the multimedia recognition result of the multimedia sequence.

[0164] The Mth decoder includes a second cross-attention layer and a decoding layer.

[0165] The recognition module 14 is further configured to include:

[0166] The second cross-attention layer is configured to perform cross-attention feature extraction on the initial key media feature corresponding to the Mth decoder to obtain an effective media feature.

[0167] The decoding layer is configured to decode the effective media feature to obtain a decoded media feature output by the Mth decoder.

[0168] The recognition module 14 is configured to include:

[0169] The second cross-attention layer is used to obtain an association degree matrix reflecting the association degree between the initial key media features corresponding to the Mth decoder and media recognition.

[0170] The initial key media features corresponding to the Mth decoder are linearly transformed according to the value weight matrix in the second cross-attention layer to obtain processed initial key media features.

[0171] The second cross-attention layer is used to perform effective feature extraction on the processed initial key media features according to the association degree matrix of the initial key media features corresponding to the Mth decoder to obtain effective media features.

[0172] The encoding module 12 is specifically configured to:

[0173] The media embedding vector of the tth media sequence block in the N media sequence blocks is obtained, and the media embedding vector of the tth media sequence block is subjected to attention calculation by the encoder to obtain the attention feature of the tth multimedia sequence block; t is a positive integer less than or equal to N;

[0174] The media embedding vector and the attention feature of the tth media sequence block are fused to obtain a fused attention feature, and the fused attention feature is linearly transformed to obtain a transformed attention feature of the tth media sequence block;

[0175] The fused attention feature and the transformed attention feature of the tth media sequence block are fused to obtain the media feature corresponding to the tth media sequence block.

[0176] The block module 11 is specifically configured to:

[0177] When the sequence length of the multimedia sequence to be processed is greater than the limited recognition length of the data recognition model, the window size of the sliding block window is determined according to the limited recognition length;

[0178] The multimedia sequence is block-processed according to the sliding block window with the window size and the sliding step of the sliding block window to obtain N multimedia sequence blocks; the sliding step is less than or equal to the limited recognition length.

[0179] The multimedia sequence is a question and answer sequence data, and the question and answer sequence data includes P questions and Q answers, P and Q are positive integers;

[0180] The block module 11 is specifically configured to:

[0181] When the sequence length of the multimedia sequence to be processed is greater than the limited recognition length of the data recognition model, the P questions and the Q answers are sorted according to the generation time corresponding to the P questions and the generation time corresponding to the Q answers respectively to obtain the sorted question and answer sequence data.

[0182] According to the limited recognition length, the sorted question and answer sequence data is processed in blocks to obtain N media sequence blocks; one of the N media sequence blocks includes at least one question and at least one answer corresponding to the question.

[0183] The data processing apparatus further includes:

[0184] The acquisition module 15 is configured to acquire the labeled recognition result of the multimedia sequence.

[0185] The determination module 16 is configured to determine the model loss of the data recognition model according to the labeled recognition result of the multimedia sequence and the multimedia recognition result of the multimedia sequence obtained by the data recognition model.

[0186] The adjustment module 17 is configured to adjust the model parameters of the data recognition model according to the model loss function and the model loss of the data recognition model, to obtain an adjusted data recognition model.

[0187] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined target, and can be implemented entirely or partially by using software, hardware (such as a processing circuit or a memory) or a combination thereof. Similarly, one processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an overall module or unit that includes the functions of the module or unit. According to one embodiment of the present application, Figure 7 The various modules in the data processing apparatus shown can be combined into one or several units respectively or all, or some of the units can be further split into at least two sub-units with smaller functions, which can achieve the same operation without affecting the implementation of the technical effects of the embodiments of the present application. The above modules are divided based on logical functions, and in actual applications, the functions of one module can also be implemented by at least two units, or the functions of at least two modules can be implemented by one unit. In other embodiments of the present application, the data processing apparatus can also include other units, and in actual applications, these functions can also be assisted by other units, and can be implemented by at least two units.

[0188] According to one embodiment of the present application, a computer program (including program code) capable of executing the steps involved in the corresponding method shown in Figure 3 may be constructed by running on a general-purpose computer device such as a computer including a central processing unit (CPU), a random access storage medium (RAM), a read-only storage medium (ROM), and the like processing elements and storage elements. Figure 7The data processing apparatus shown in the above embodiments, and a data processing method for implementing the embodiments of the present application. The above computer program can be recorded on a computer readable recording medium, for example, and loaded into the above computer device through the computer readable recording medium, and run therein.

[0189] The embodiments of the present application propose a method for expanding the input sequence length of a general data recognition model. Since the role of the queryer for feature retrieval is to retrieve key media features from the media features corresponding to the multimedia sequence to be processed, the code in the media recognition model will not be changed, and no additional learning weight will be added, so the recognition accuracy of the data recognition model for the media sequence will not be affected. Specifically, when the sequence length of the multimedia sequence to be processed is greater than the limited recognition length of the data recognition model, the multimedia sequence can be blocked according to the limited recognition length to obtain N media sequence blocks. It can be understood that the sequence length of the N media sequence blocks is less than or equal to the limited recognition length, so the N media sequence blocks can be input into the data recognition model, and the encoder in the data recognition model is used to encode the N media sequence blocks to obtain the media features corresponding to the N media sequence blocks, respectively. Further, the queryer and the decoder in the data recognition model are used to retrieve the key media features of the multimedia sequence from the media features corresponding to the N media sequence blocks, respectively. According to the key media features, the multimedia sequence is recognized to obtain the multimedia recognition result of the multimedia sequence. It can be seen that the present application does not need to use the truncation method of the front sequence part in the multimedia sequence, or delete part of the sequence in the multimedia sequence to shorten the length of the multimedia sequence, but input the entire multimedia sequence into the data recognition model, and extract the key media features from the N media features corresponding to the multimedia sequence. In this way, the loss of key information of the multimedia sequence caused by truncation or deletion can be avoided, and the data recognition model can accept the input of multimedia sequences of any sequence length. At the same time, further incremental training of the data recognition model can further improve the model performance of the data recognition model.

[0190] Further, please refer to Figure 8 , Figure 8 is a schematic diagram of a computer device provided by the embodiments of the present application. As Figure 8 shown, the computer device 3000 can be the above Figure 2Corresponding to the terminal device or the server in the embodiment, the computer device 3000 can include at least one processor 3001 such as a CPU, at least one network interface 3004, a user interface 3003, a memory 3005, and at least one communication bus 3002. The communication bus 3002 is used to realize the connection and communication between the components. The user interface 3003 can include a display, a keyboard, and the network interface 3004 can optionally include a standard wired interface and a wireless interface such as a WI-FI interface. The memory 3005 can be a high-speed RAM memory or a non-volatile memory such as at least one disk memory. The storage 3005 can also be at least one storage device located away from the aforementioned processor 3001. As shown in the figure, the memory 3005 as a computer storage medium can include an operating system, a network communication module, a user interface module, and a computer program control application. Figure 8

[0191] In the computer device 3000 shown in the figure, the network interface 3004 is mainly used for the network communication between the second node device and the target relay server and the target oracle server; the user interface 3003 is mainly used to provide an input interface for the user; and the processor 3001 can be used to call the computer program control application stored in the memory 3005 to realize the following functions. Figure 8

[0192] When the sequence length of the multimedia sequence to be processed is greater than the limited recognition length of the data recognition model, the multimedia sequence is processed by block division according to the limited recognition length, to obtain N media sequence blocks; the data recognition model includes a decoder and an encoder; N is an integer greater than 1;

[0193] The N media sequence blocks are feature coded by the encoder to obtain media features corresponding to the N media sequence blocks, respectively;

[0194] The key media features of the multimedia sequence are searched from the media features corresponding to the N media sequence blocks, respectively, by a queryer for feature search and the decoder;

[0195] The multimedia sequence is recognized according to the key media features to obtain a media recognition result of the multimedia sequence; the media recognition result is used to reflect at least one of a media attribute and a media key content of the multimedia sequence.

[0196] It should be understood that the computer device 3000 described in the embodiments of the present application can also perform the functions of the computer device 3000 described in the foregoing embodiments. Figure 6 ​​Corresponding to the description of a data processing method in the embodiment, the computer device 3000 described in the embodiment can also execute the foregoing Figure 7 Corresponding to the description of a data processing device in the embodiment, details are not repeated here. In addition, the beneficial effects of using the same method are not repeated.

[0197] In addition, it should be noted here that the embodiment of the present application also provides a computer readable storage medium, and the computer readable storage medium stores the computer program executed by the data processing device mentioned above, and the computer program includes program instructions, which can execute the foregoing Figure 3 Or Figure 6 Corresponding to the description of the data processing method in the embodiment, therefore, details will not be repeated here. In addition, the beneficial effects of using the same method are not repeated. For technical details not disclosed in the computer readable storage medium embodiment of the present application, please refer to the description of the method embodiment of the present application. As an example, the program instructions can be deployed to execute on one computing device, or on multiple computing devices located in one place, or on multiple computing devices distributed in multiple places and interconnected through a communication network, which can constitute a blockchain system.

[0198] The present application provides a computer program product or computer program, which includes computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device can execute the foregoing Figure 3 Or Figure 6 Corresponding to the description of a data processing method in the embodiment, details are not repeated here. In addition, the beneficial effects of using the same method are not repeated.

[0199] It should be noted that the collection and processing of related data in the present application should strictly comply with the requirements of relevant national laws and regulations, obtain the informed consent or separate consent of the personal information subject (or have a legal basis), and carry out subsequent data use and processing behavior within the scope of authorization of laws and regulations and the personal information subject. For example, the collection of multimedia sequences in the present application will obtain the informed consent or separate consent of the personal information subject (or have a legal basis).

[0200] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware. The program can be stored in a computer readable storage medium, and when the program is executed, the processes of the above-mentioned embodiment methods can be included. The storage medium can be a magnetic disc, an optical disc, a read-only memory (ROM), a random access memory (RAM), or the like.

[0201] The above disclosure is only the preferred embodiments of the present application, and of course cannot limit the scope of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope of the present application.

Claims

1. A data processing method, characterized by, The method comprises the following steps: when the sequence length of a multimedia sequence to be processed is greater than a limited recognition length of a data recognition model, the multimedia sequence is processed in blocks according to the limited recognition length, and N media sequence blocks are obtained; the data recognition model comprises an encoder and a decoder, and N is an integer greater than 1; media features corresponding to the N media sequence blocks are obtained by performing feature coding on the N media sequence blocks through the encoder; key media features of the multimedia sequence are retrieved from the media features corresponding to the N media sequence blocks through a queryer for feature retrieval and the decoder; a multimedia recognition result of the multimedia sequence is obtained by recognizing the multimedia sequence according to the key media features; the multimedia recognition result is used to reflect at least one of a multimedia attribute and a multimedia key content of the multimedia sequence.

2. The method of claim 1, wherein, The number of the decoders is M, and M is a positive integer greater than 1; the step of retrieving the key media features of the multimedia sequence from the media features corresponding to the N media sequence blocks through the queryer for feature retrieval and the decoder comprises the following steps: a retrieval attention parameter of an (i+1)th decoder is determined according to a decoded media feature output by an ith decoder through the (i+1)th decoder; the decoded media feature output by the ith decoder is obtained by decoding an initial key media feature corresponding to the ith decoder through the ith decoder; i is a positive integer less than M; an initial key media feature corresponding to the (i+1)th decoder is retrieved from the media features corresponding to the N media sequence blocks through the queryer for feature retrieval according to the retrieval attention parameter of the (i+1)th decoder; an initial key media feature corresponding to an Mth decoder is determined as the key media features of the multimedia sequence.

3. The method of claim 2, wherein, the (i+1)th decoder comprises a first cross-attention layer; the step of determining the retrieval attention parameter of the (i+1)th decoder according to the decoded media feature output by the ith decoder through the (i+1)th decoder comprises the following steps: a processed decoded media feature is obtained by performing linear transformation on the decoded media feature output by the ith decoder according to a query weight matrix in the first cross-attention layer through the first cross-attention layer; the retrieval attention parameter of the (i+1)th decoder is obtained by performing linear transformation on the processed decoded media feature according to a key weight matrix in the first cross-attention layer.

4. The method of claim 2, wherein, the step of retrieving the initial key media feature corresponding to the (i+1)th decoder from the media features corresponding to the N media sequence blocks through the queryer for feature retrieval according to the retrieval attention parameter of the (i+1)th decoder comprises the following steps: the importance degrees of the N media features are determined according to the retrieval attention parameter of the (i+1)th decoder through the queryer for feature retrieval. The media feature corresponding to the maximum importance degree among the N media features is determined as the initial key media feature corresponding to the i+1th decoder.

5. The method of claim 4, wherein, The queryer for feature retrieval determines the importance degrees of the N media features corresponding to the i+1th decoder. The queryer for feature retrieval obtains a parameter matrix reflecting the retrieval attention parameters of the i+1th encoder and a feature matrix reflecting the jth media feature among the N media features; j is a positive integer less than or equal to N; The parameter matrix and the transpose of the feature matrix are multiplied to obtain an association degree matrix reflecting the association degree between the jth media feature and media recognition. The importance degree of the jth media feature is determined according to the association degree matrix.

6. The method of claim 2, wherein, The data recognition model further comprises an identification layer: The multimedia sequence is identified according to the key media feature to obtain a multimedia recognition result of the multimedia sequence. The initial key media feature corresponding to the Mth decoder is decoded by the Mth decoder to obtain a decoded media feature output by the Mth decoder. The decoded media feature output by the Mth decoder is identified by the identification layer to obtain a multimedia recognition result of the multimedia sequence.

7. The method of claim 6, wherein, The Mth decoder comprises a second cross-attention layer and a decoding layer. The initial key media feature corresponding to the Mth decoder is decoded by the Mth decoder to obtain a decoded media feature output by the Mth decoder, comprising: The initial key media feature corresponding to the Mth decoder is cross-attention feature extracted by the second cross-attention layer to obtain an effective media feature. The effective media feature is decoded by the decoding layer to obtain a decoded media feature output by the Mth decoder.

8. The method of claim 7, wherein, The initial key media feature corresponding to the Mth decoder is cross-attention feature extracted by the second cross-attention layer to obtain an effective media feature, comprising: The second cross-attention layer obtains an association degree matrix reflecting the association degree between the initial key media feature corresponding to the Mth decoder and media recognition; The initial key media feature corresponding to the Mth decoder is linearly transformed according to the value weight matrix in the second cross-attention layer to obtain a processed initial key media feature; The second cross-attention layer extracts an effective media feature from the processed initial key media feature according to the association degree matrix of the initial key media feature corresponding to the Mth decoder.

9. The method of claim 1, wherein, The N media sequence blocks are feature encoded by the encoder to obtain media features corresponding to the N media sequence blocks, respectively, comprising: obtaining a media embedding vector of a tth media sequence block in the N media sequence blocks, performing attention calculation on the media embedding vector of the tth media sequence block by the encoder to obtain an attention feature of the tth multimedia sequence block; t is a positive integer less than or equal to N; fusing the media embedding vector and the attention feature of the tth media sequence block to obtain a fused attention feature, and performing linear transformation on the fused attention feature to obtain a transformed attention feature of the tth media sequence block; fusing the fused attention feature and the transformed attention feature of the tth media sequence block to obtain a media feature corresponding to the tth media sequence block.

10. The method of claim 1, wherein, When the sequence length of the multimedia sequence to be processed is greater than the limited recognition length of the data recognition model, the multimedia sequence is processed in blocks according to the limited recognition length to obtain N media sequence blocks, including: When the sequence length of the multimedia sequence to be processed is greater than the limited recognition length of the data recognition model, the window size of the sliding block window is determined according to the limited recognition length; According to the sliding block window with the window size and the sliding step of the sliding block window, the multimedia sequence is processed in blocks to obtain N multimedia sequence blocks; the sliding step is less than or equal to the limited recognition length.

11. The method of claim 1, wherein, The multimedia sequence is question and answer sequence data, and the question and answer sequence data includes P questions and Q answers, P and Q are positive integers; When the sequence length of the multimedia sequence to be processed is greater than the limited recognition length of the data recognition model, the multimedia sequence is processed in blocks according to the limited recognition length to obtain N media sequence blocks, including: When the sequence length of the multimedia sequence to be processed is greater than the limited recognition length of the data recognition model, the P questions and the Q answers are sorted according to the generation time corresponding to the P questions and the generation time corresponding to the Q answers respectively to obtain sorted question and answer sequence data; According to the limited recognition length, the sorted question and answer sequence data are processed in blocks to obtain N media sequence blocks; one of the N media sequence blocks includes at least one question and an answer corresponding to the at least one question.

12. The method of claim 1, wherein, The data recognition model is obtained by adding the queryer to the trained recognition model; the method further includes: obtaining a labeled recognition result of the multimedia sequence; determining a model loss of the data recognition model according to the labeled recognition result of the multimedia sequence and a multimedia recognition result of the multimedia sequence obtained by the data recognition model; adjusting the model parameters of the data recognition model according to the model loss function of the data recognition model and the model loss to obtain an adjusted data recognition model.

13. A data processing apparatus, characterized by: including: The block module is configured to perform block processing on the multimedia sequence according to a limited recognition length of a data recognition model when a sequence length of the multimedia sequence to be processed is greater than the limited recognition length, to obtain N media sequence blocks; the data recognition model comprises an encoder and a decoder, and N is an integer greater than 1. The encoding module is configured to perform feature encoding on the N media sequence blocks by using the encoder, to obtain media features corresponding to the N media sequence blocks, respectively. The searching module is configured to search for key media features of the multimedia sequence from the media features corresponding to the N media sequence blocks, respectively, by using a queryer for feature searching and the decoder. The recognition module is configured to recognize the multimedia sequence according to the key media features, to obtain a multimedia recognition result of the multimedia sequence; and the multimedia recognition result is used to reflect at least one of multimedia attributes and multimedia key contents of the multimedia sequence.

14. A computer device, comprising: The computer device comprises: a processor and a memory; the memory is connected to the processor, and is configured to store a computer program; and the processor is configured to call the computer program, so that the computer device executes the method in any one of claims 1-12.

15. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is suitable for being loaded and executed by the processor, so that the computer device with the processor executes the method in any one of claims 1-12.

16. A computer program product or computer program, characterised in that, The computer program product or the computer program comprises computer instructions stored in the computer readable storage medium, and the computer instructions are suitable for being read and executed by the processor, so that the computer device with the processor executes the method in any one of claims 1-12.

Images