Data processing method and apparatus, device, and readable storage medium
By introducing single-modal and cross-modal routing layers into the modal characterization model, combining attention interaction processing, optimizing the routing layer to retain the feature information of each modal, the reduction in the accuracy of single-modal data understanding in the existing technology is solved. The problem has improved the representation accuracy and model universality of multimodal data.
Patent Information
- Application Number
- PCT/CN2024/083332
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-19
- Filing Date
- 2024-03-22
- Publication Date
- 2024-11-28
AI Technical Summary
In the multimodal understanding task of media data, in order to improve the accuracy of multimodal information, the individual information of singlemodal information will be eliminated, resulting in the model's understanding of singlemodal data decreases, affecting the subsequent tasks. Processing results.
A data processing method is proposed. By introducing a single-modal routing layer and a cross-modal routing layer in the modal representation model, combining attention interaction processing and feature prediction processing, the routing layer is optimized to retain feature information of each modal type. Improve the universality and accuracy of modal representation models.
Effectively improve the accuracy of the modal representation model's understanding of single-modal and cross-modal data, maintain the performance of each modal type, enhance the universality and flexibility of the model, and can output cross-modal and single-modal simultaneously Characterization vector of type.
Smart Images

Figure CN2024083332_28112024_PF_FP_ABST
Abstract
Description
Data processing method, device, equipment and readable storage medium
[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on May 19, 2023, with application number 2023105720436 and application name “A data processing method, device, equipment and readable storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field
[0002] The present application relates to the field of computer technology, and in particular to a data processing method, apparatus, device, and readable storage medium. Background Art
[0003] With the rapid development of multimedia technology, media data (such as pictures, text, video, audio, etc.) are produced in large quantities. With the generation of massive media data, the task of understanding media data (such as video understanding task) becomes particularly important. The task of understanding media data can provide media data with many rich and diverse media tags (such as dance tags, singing tags, competition tags, game tags, etc.). The media tags of media data can facilitate various media processing of media data. For example, it can facilitate media retrieval, classification, archiving, media recommendation, media secondary editing and other processing of media data.
[0004] In media retrieval tasks, media tags can be used to easily retrieve relevant media segments, such as text tags and title information of media data. Media retrieval is of great significance for media recommendation and media processing. As the demand for media retrieval grows, the requirements for media retrieval capabilities are also becoming higher and higher. For example, after inputting media data with a modality type of text, the demand for retrieval results is not limited to media data with a modality type of text, but also requires the retrieval of media data with a modality type of non-text (such as video type, audio type, image type, etc.). In other words, in the media data retrieval business, the requirements for multi-modal retrieval performance are becoming increasingly greater.
[0005] However, in related technologies, for the task of understanding the multimodality of media data, a unified model is used to perform understanding and reasoning. Specifically, the features of each modality are extracted for each modality, and then the multimodal information of the features of all modalities is fused to finally obtain multimodal information. However, when training the model, related technologies will eliminate the individual information of a single modality and extract the common information between different modalities in order to improve the accuracy of multimodal information. This will seriously affect the accuracy of the model's understanding of single-modal data, and thus seriously affect the accuracy of the processing results of single-modal data in subsequent tasks (such as retrieval tasks).
[0006] Summary of the Invention
[0007] The embodiments of the present application provide a data processing method, apparatus, device, and readable storage medium, which can improve the versatility of the modal characterization model and the accuracy of the modal characterization results in the modal characterization business.
[0008] On the one hand, an embodiment of the present application provides a data processing method, including:
[0009] Get the training task group set; the training task group set includes the training task group S i , i is a positive integer; the training task group S i The modality type is a single modality type or a cross-modality type; in the training task group S i When the modality type is the single modality type, the training task group S i Contains a sample media data; in the training task group S i When the modality type is the cross-modality type, the training task group S i Contains sample media data of at least two different modal types;
[0010] In the modal representation model, the training task group S i Perform attention interaction processing to obtain the training task group S i The corresponding attention representation vector; attention interaction processing is used to group the training task S i The different elements in the game fully interact with each other;
[0011] In the single-modal routing layer and cross-modal routing layer included in the modal representation model, the task group S is determined and trained. i The target routing layer that matches the modality type belongs to, and the training task group S is based on the target routing layer i The corresponding attention representation vector is processed for feature prediction to obtain the training task group S i The corresponding predicted modal representation vector;
[0012] When the predicted modal representation vector corresponding to each training task group in the training task group set is determined, the single-modal routing layer is optimized based on the predicted modal representation vector corresponding to the training task group whose modality type is a single-modal type, and the cross-modal routing layer is optimized based on the predicted modal representation vector corresponding to the training task group whose modality type is a cross-modal type; the optimized single-modal routing layer is used to perform feature prediction processing on the task group whose modality type is a single-modal type, and the optimized cross-modal routing layer is used to perform feature prediction processing on the task group whose modality type is a cross-modal type.
[0013] In one aspect, an embodiment of the present application provides a data processing device, including:
[0014] The set acquisition module is used to obtain the training task group set; the training task group set includes the training task group S i , i is a positive integer; the training task group S i The modality type is a single modality type or a cross-modality type; in the training task group S i When the modality type is the single modality type, the training task group S i Contains a sample media data; in the training task group S i When the modality type is the cross-modality type, the training task group S i Contains sample media data of at least two different modal types;
[0015] Attention processing module, used to train the task group S in the modality representation model i Perform attention interaction processing to obtain the training task group S i The corresponding attention representation vector; attention interaction processing is used to group the training task S i The different elements in the game fully interact with each other;
[0016] The routing layer determination module is used to determine and train the task group S in the single-modal routing layer and cross-modal routing layer included in the modal representation model. i The target routing layer that matches the modality type;
[0017] Prediction module, used to train task group S based on target routing layer i The corresponding attention representation vector is processed for feature prediction to obtain the training task group S i The corresponding predicted modal representation vector;
[0018] A first optimization module is configured to optimize the unimodal routing layer based on the predicted modal representation vector corresponding to the training task group whose modality type is a unimodal type when the predicted modal representation vector corresponding to each training task group in the training task group set is determined;
[0019] The second optimization module is used to optimize the cross-modal routing layer based on the predicted modal representation vector corresponding to the training task group whose modality type is a cross-modal type; the optimized single-modal routing layer is used to perform feature prediction processing on the task group whose modality type is a single-modal type, and the optimized cross-modal routing layer is used to perform feature prediction processing on the task group whose modality type is a cross-modal type.
[0020] In one embodiment, the specific implementation method of the set acquisition module acquiring the training task group set includes:
[0021] Obtain N sample media data; N is a positive integer, and a modality type of one sample media data in the N media data is any one of a first modality type and a second modality type; the first modality type and the second modality type are different;
[0022] Based on the modality type to which each sample media data belongs in the N sample media data, a task group construction process is performed on the N sample media data to obtain a training task group set.
[0023] In one embodiment, the set acquisition module performs task group construction processing on the N sample media data based on the modality type to which each sample media data belongs, and obtains the specific implementation method of the training task group set, including:
[0024] Combining first sample media data of the N sample media data with a first identifier representing a first modality type to obtain a first training task group whose modality type is the first modality type; the first sample media data refers to any sample media data of the N sample media data whose modality type is the first modality type;
[0025] Combining second sample media data from the N sample media data with a second identifier representing the second modality type to obtain a second training task group whose modality type is the second modality type; the second sample media data refers to any sample media data from the N sample media data whose modality type is the second modality type;
[0026] Based on the media source channels corresponding to the first sample media data and the second sample media data, respectively, cross-modal combination processing is performed on the first sample media data and the second sample media data to obtain a training task group whose modality type is a cross-modal type;
[0027] The first training task group and the second training task group are both determined as training task groups whose modality type is a single modality type, and the set consisting of the training task group whose modality type is the cross-modality type and the training task group whose modality type is the single modality type is determined as a training task group set.
[0028] In one embodiment, the number of the first sample media data and the second sample media data is one or more; the one or more first sample media data include the first sample media data M j , j is a positive integer; the modality type is a cross-modal training task group, including the first sample media data M j corresponding training task group;
[0029] The specific implementation method of the set acquisition module performing cross-modal combination processing on the first sample media data and the second sample media data based on the media source channels corresponding to the first sample media data and the second sample media data to obtain a training task group whose modality type is a cross-modal type includes:
[0030] The first sample media data M j The corresponding media source channel is determined as the target media source channel;
[0031] The second sample media data whose media source channel is the target media source channel in one or more second sample media data is determined as the first sample media data M j Corresponding associated sample media data;
[0032] The first sample media data, the associated sample media data, the first identifier and the second identifier are combined to obtain the first sample media data M j The corresponding training task group.
[0033] In one embodiment, the attention processing module performs attention processing on the training task group S in the modality representation model. i Perform attention interaction processing to obtain the training task group S i The specific implementation of the corresponding attention representation vector includes:
[0034] In the modal representation model, the training task group S is trained through the feature extraction network layer. i The sample media data contained therein is processed for feature extraction to obtain the training task group S i Corresponding media characteristics;
[0035] Through the multi-head self-attention network layer in the modality representation model, the training task group S i The corresponding media features are processed by multi-head self-attention to obtain the training task group S i The corresponding attention representation vector.
[0036] In one embodiment, the multi-head self-attention network layer in the modality representation model includes Q self-attention sub-network layers; Q is a positive integer; the Q self-attention sub-network layers include the self-attention sub-network layer V k , k is a positive integer;
[0037] The attention processing module uses the multi-head self-attention network layer in the modality representation model to train the task group S i The corresponding media features are processed by multi-head self-attention to obtain the training task group S i The specific implementation of the corresponding attention representation vector includes:
[0038] Get the self-attention sub-network layer Vk The included attention parameter matrix will be the self-attention sub-network layer V k The included attention parameter matrix is related to the training task group S i The corresponding media features are processed to obtain the linear transformation matrix corresponding to the attention parameter matrix;
[0039] Through the self-attention sub-network layer V k The fully connected components in the linear transformation matrix perform feature integration processing to obtain the self-attention sub-network layer V k The corresponding attention representation sub-vector;
[0040] When the attention representation sub-vectors corresponding to the Q self-attention sub-network layers are determined, the Q attention representation sub-vectors are fused to obtain the training task group S i The corresponding attention representation vector.
[0041] In one embodiment, the single modality type includes a first modality type and a second modality type; the single modality routing layer includes a first modality sub-routing layer corresponding to the first modality type and a second modality sub-routing layer corresponding to the second modality type;
[0042] The routing layer determination module determines and trains the task group S in the single-modal routing layer and the cross-modal routing layer included in the modal representation model. i The specific implementation of the target routing layer that matches the modality type includes:
[0043] If the training task group S i If the modality type is a cross-modal type, the cross-modal routing layer included in the modality representation model is determined to be the same as the training task group S i The target routing layer that matches the modality type;
[0044] If the training task group S i If the modality type is the first modality type, the first modality sub-routing layer corresponding to the first modality type is determined to be the same as the training task group S i The target routing layer that matches the modality type;
[0045] If the training task group S i If the modality type is the second modality type, the second modality sub-routing layer corresponding to the second modality type is determined to be the same as the training task group S i The target routing layer that matches the modality type.
[0046] In one embodiment, the single modality type includes a first modality type and a second modality type; the single modality routing layer includes a first modality sub-routing layer corresponding to the first modality type and a second modality sub-routing layer corresponding to the second modality type;
[0047] The specific implementation method of the first optimization module optimizing the unimodal routing layer based on the predicted modal representation vector corresponding to the training task group of the unimodal type includes:
[0048] Determine the training task group whose modality type is the first modality type as the first modality training task group, and determine the training task group whose modality type is the second modality type as the second modality training task group;
[0049] Obtaining a true modal representation vector corresponding to the first modal training task group and a true modal representation vector corresponding to the second modal training task group;
[0050] Performing error calculation processing on the predicted modal representation vector corresponding to the first modal training task group and the true modal representation vector corresponding to the first modal training task group to obtain a first loss value corresponding to the first modal training task group;
[0051] Performing error calculation processing on the predicted modal representation vector corresponding to the second modal training task group and the true modal representation vector corresponding to the second modal training task group to obtain a second loss value corresponding to the second modal training task group;
[0052] The first modality sub-routing layer corresponding to the first modality type is optimized based on the first loss value, and the second modality sub-routing layer corresponding to the second modality type is optimized based on the second loss value.
[0053] In one embodiment, the first modality type is a text modality type, and the first modality training task group is a text word sequence; the predicted modality representation vector corresponding to the first modality training task group is composed of the predicted representation features corresponding to each text word in the text word sequence; the real modality representation vector corresponding to the first modality training task group is composed of the real representation features corresponding to each text word in the text word sequence;
[0054] The first optimization module performs error calculation processing on the predicted modal representation vector corresponding to the first modal training task group and the true modal representation vector corresponding to the first modal training task group to obtain a first loss value corresponding to the first modal training task group. The specific implementation method includes:
[0055] Obtaining a masked text word from the text words included in the text word sequence; the masked text word refers to a text word that has been masked;
[0056] Obtaining the predicted representation features corresponding to the masked text words in the predicted modal representation vector corresponding to the first modal training task group, and obtaining the real representation features corresponding to the masked text words in the real modal representation vector corresponding to the first modal training task group;
[0057] Determining a first feature similarity between a predicted representation feature corresponding to the masked text word and a true representation feature corresponding to the masked text word;
[0058] The first feature similarity is determined as a first loss value corresponding to the first modality training task group.
[0059] In one embodiment, the training task group set includes at least two training task groups whose modality type is cross-modality type; at least two training task groups whose modality type is cross-modality type, including training task group S j , j is a positive integer; training task group S j The training task group S includes first sample media data of a first modality type and second sample media data of a second modality type. j The corresponding predicted modal representation vector includes the predicted representation features corresponding to the first sample media data and the predicted representation features corresponding to the second sample media data;
[0060] The second optimization module optimizes the cross-modal routing layer based on the predicted modal representation vector corresponding to the training task group whose modality type is cross-modal, including:
[0061] Determining a second feature similarity between a predicted characterizing feature corresponding to the first sample media data and a predicted characterizing feature corresponding to the second sample media data;
[0062] Each training task group whose modality type is a cross-modality type is determined as a cross-modality training task group, and the training task group S is determined. j The third feature similarity between the remaining cross-modal training task groups in at least two cross-modal training task groups; the remaining cross-modal training task groups refer to the at least two cross-modal training task groups, except the training task group S j Cross-modal training task force other than
[0063] Based on the second feature similarity and the third feature similarity, the cross-modal routing layer is optimized.
[0064] In one embodiment, the data processing device may further include:
[0065] A task group acquisition module is used to acquire a target task group for describing the media data to be classified; the target task group includes at least two media data of different modal types;
[0066] An interaction processing module is used to perform attention interaction processing on the target task group in the modality representation model to obtain the attention representation vector corresponding to the target task group;
[0067] A feature processing module is used to perform feature prediction processing on the attention representation vector corresponding to the target task group in the optimized cross-modal routing layer to obtain a predicted modality representation vector corresponding to the target task group;
[0068] The category recognition module is used to perform category recognition processing on the predicted modal representation vector corresponding to the target task group to obtain the media category to which the media data to be classified belongs; the media category to which the media data to be classified belongs is used to archive the media data to be classified.
[0069] In one aspect, an embodiment of the present application provides a computer device, including: a processor and a memory;
[0070] The memory stores a computer program, and when the computer program is executed by the processor, the processor executes the method in the embodiment of the present application.
[0071] On one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. The computer program includes program instructions. When the program instructions are executed by a processor, the method in the embodiment of the present application is executed.
[0072] In one aspect of the present application, a computer program product is provided, comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the method provided in one aspect of the embodiments of the present application.
[0073] In an embodiment of the present application, a general modal representation model is provided, which can serve both data of a unimodal type and data of a cross-modal type. Specifically, the present application can use a set of training task groups including unimodal and cross-modal types to train the modal representation model, wherein the modal representation model includes a unimodal routing layer and a cross-modal routing layer. The unimodal routing layer can perform feature prediction processing on data (attention representation vector) of a unimodal type, and the cross-modal routing layer can perform feature prediction processing on data (attention representation vector) of a cross-modal type. During training optimization, the unimodal routing layer can be optimized based on the predicted modal representation vector corresponding to the training task group of the unimodal type, and the cross-modal routing layer can be optimized by the predicted modal representation vector corresponding to the training task group of the cross-modal type. Based on this, the single-modal routing layer and the cross-modal routing layer in the modal representation model can be trained and optimized at the same time, and during the training optimization, the training optimization is carried out separately based on data of different modal types, without losing any modal type of data, and the performance of the single-modal routing layer and the cross-modal routing layer can be maintained at the same time, thereby improving the versatility and flexibility of the modal representation model for single-modal type and cross-modal type. In summary, the present application can set a routing distribution mechanism in the modal representation model, divide the routing layer into a single-modal routing layer and a cross-modal routing layer, and different routing layers can process data of different modal types. During training, the data of different modal types are also trained and optimized separately. Each routing layer will only retain the relevant information of the current modal type, so there is no need to discard any modal type of data, and the representation vectors of cross-modal type and single-modal type can be output at the same time, which can greatly improve the accuracy of the representation results of data of various modal types. In summary, the present application can improve the versatility of the modal representation model and the accuracy of the modal representation results in the modal representation business. BRIEF DESCRIPTION OF THE DRAWINGS
[0074] FIG1 is a network architecture diagram of a data processing system provided in an embodiment of the present application;
[0075] FIG2 is a flow chart of a data processing method provided in an embodiment of the present application;
[0076] FIG3 is a schematic diagram of a transformer architecture provided in an embodiment of the present application;
[0077] FIG4 is a schematic diagram of an architecture of a modal routing distribution provided in an embodiment of the present application;
[0078] FIG5 is a schematic diagram of a scenario of media data retrieval provided in an embodiment of the present application;
[0079] FIG6 is a schematic diagram of a process for optimizing a single-mode routing layer according to an embodiment of the present application;
[0080] FIG7 is a schematic diagram of a process for optimizing a cross-modal routing layer according to an embodiment of the present application;
[0081] FIG8 is a schematic structural diagram of a data processing device provided in an embodiment of the present application;
[0082] FIG9 is a schematic structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0083] This application involves artificial intelligence and related concepts. For ease of understanding, the following briefly explains artificial intelligence and related concepts:
[0084] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also involves studying the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.
[0085] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.
[0086] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, autonomous driving, drones, robots, smart medical care, smart customer service, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.
[0087] The solution provided in the embodiment of the present application belongs to the machine learning (ML) and natural language processing (NLP) technologies under the field of artificial intelligence.
[0088] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.
[0089] Natural language processing (NLP) is a key area of research in computer science and artificial intelligence. It studies the theories and methods that enable effective communication between humans and computers using natural language. Natural language processing (NLP) integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language we use in everyday life—and is closely linked to the study of linguistics. Natural language processing technologies typically include text processing, semantic understanding, machine translation, robotic question answering, and knowledge graphs.
[0090] The embodiments of the present application can specifically use natural language processing technology to perform semantic analysis on relevant text data (such as title data, description data) of media data (such as text data, video data) to obtain semantic analysis results of the media data, so as to better understand the media data; at the same time, the embodiments of the present application can use machine learning technology to train and optimize relevant models (such as the modal representation model mentioned later) to improve the accuracy of the model's output results.
[0091] For ease of understanding, please refer to Figure 1, which is a network architecture diagram of a data processing system provided in an embodiment of the present application. As shown in Figure 1, the network architecture may include a business server 1000 and a terminal device cluster. The terminal device cluster may include one or more terminal devices, and the number of terminal devices will not be limited here. As shown in Figure 1, multiple terminal devices may include terminal device 100a, terminal device 100b, terminal device 100c, ..., terminal device 100n; as shown in Figure 1, terminal device 100a, terminal device 100b, terminal device 100c, ..., terminal device 100n can be respectively connected to the business server 1000 through a network, so that each terminal device can exchange data with the business server 1000 through the network connection. In addition, any terminal device in the terminal device cluster 100 may refer to an intelligent device running an operating system, and the embodiment of the present application does not specifically limit the operating system of the terminal device.
[0092] The terminal device in the data processing system shown in FIG1 may be a smart phone, a tablet computer, a laptop computer, a PDA, a desktop computer, a mobile internet device (MID), a POS (Point of Sales) machine, a smart speaker, a smart TV, a smart watch, a smart car terminal, a virtual reality (VR) device, an augmented reality (AR) device, etc., but is not limited thereto. The terminal device is often equipped with a display device, which may be a display, a display screen, a touch screen, etc., and the touch screen may be a touch screen, a touch panel, etc.
[0093] The business server in the data processing system shown in Figure 1 can be a single physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal device and the business server can be connected directly or indirectly via wired or wireless communication, which is not limited in this application.
[0094] In one possible implementation, applications such as video applications, browser applications, game applications, educational applications, etc. are running in a terminal device (such as terminal device 100a), and examples of the applications will not be given one by one here. In the various embodiments of the present application, video applications are used as an example for illustration. An object can run a video application in a terminal device, and the video application can provide media data (such as text data, video data, audio data, image data, etc.) to the object in the form of a feeds stream (feed can refer to a message source, and feed can also be translated as source material, feed, information provision, feed, summary, source, news subscription. It is a data format through which the application transmits the latest information to the object, usually arranged in a timeline manner), and the object can browse these media data in the video application. For video applications, a retrieval function can also be provided to the object, and the object can retrieve media data (such as text data, image data, audio data, video data, etc.) associated with the input information by inputting certain information. For example, after a subject inputs some text data, the video application can output other text data associated with the text data, or it can also output image data, video data, audio data, etc. associated with the text data. Alternatively, after a subject inputs some image data, the video application can output other image data associated with the image data, or it can also output video data, text data, audio data, etc. associated with the image data. Image data, text data, audio data, and video data can be considered as media data of different modal types. For example, image data can be understood as media data of the image modal type, and video data can be understood as media data of the video modal type. That is to say, video applications can provide retrieval functions for multimodal media data. In order to improve the accuracy of multimodal retrieval results, the present application provides a multimodal joint training framework based on a modal routing distribution mechanism. On the basis of training only one modal representation model, it simultaneously completes the tasks of determining the modal representation features of cross-modality (cross-modality refers to a modal combination of two or more modalities) and single modality. Based on the modal representation features of various modalities (which may include cross-modal modal representation features and single modal modal representation features), media data of different modal types can be retrieved, thereby reducing problems such as large model storage, time-consuming retrieval, and repeated calculations, so that single-modality and cross-modal tasks can promote each other, maintain good performance respectively, and improve versatility and accuracy.
[0095] Specifically, a modal representation model can be deployed in the business server 1000, and the business server 1000 can collect the information input by the object in the video application (such as input text data). Based on the information input by the object in the video application, the business server 1000 can perform inference analysis on the information input by the object through the modal representation model to obtain a modal representation vector of the input information. Based on the modal representation vector of the input information, the business server 1000 can determine the media data associated with the input information (such as text data, image data, video data, audio data, etc.). It can be understood that the modal representation model deployed in the business server 1000 can include different modal routing layers. For example, it can specifically include a text modal routing layer for routing text data, an image modal routing layer for routing image data, an audio modal routing layer for routing audio data, and a cross-modal routing layer for routing multimodal media data. Then, for input information of different modal types, corresponding routing processing can be performed by the corresponding modal routing layer (the routing layer here can refer to the fully connected layer, and routing processing can refer to linear transformation processing through the fully connected layer. Since linear transformation processing through the fully connected layer actually refers to feature integration prediction processing through the fully connected layer, the routing processing here can also be understood as feature prediction processing) to obtain the modal representation vector of the input information. For example, when the input information is text data, it can be routed by the text modal routing layer, and when the input information is image data, it can be routed by the image modal routing layer.
[0096] That is to say, the present application sets up a routing distribution mechanism in the modal representation model, and sets up different modal routing layers for different modal types. Input information of different modal types can be distributed to the corresponding modal routing layer for routing processing. Compared with the traditional technology in which a unified routing layer performs unified processing on input data of different modal types, setting up a routing distribution mechanism can improve the targetedness of the routing layer. Input data of one modal type can be targetedly routed by the corresponding modal routing layer. Based on different modal routing layers, it can adapt to input data of various modal types. This can greatly improve the versatility of the modal representation model for different modal types. Moreover, since the input data of different modal types are targetedly processed by the corresponding modal routing layer, the processing process is in line with the characteristics of the modal type itself, and the obtained processing results (that is, the representation results of the input data) can also be more accurate, that is: the accuracy of the representation results of data of different modal types can be improved.
[0097] In this application, in order to improve the accuracy of the output results of the modal representation model, the modal representation model can be trained and optimized. During the training process of the modal representation model, since different modal routing layers are set, for different modal routing layers, training samples of different modal types can be used for targeted training. For example, for the text modal routing layer, text sample data can be used for training, for the image modal routing layer, image sample data can be used for training, and for the cross-modal routing layer, cross-modal sample data (such as sample data containing both image sample data and text sample data) can be used for training. For routing layers such as the text modal routing layer and the image modal routing layer that are only for one modal type, this application can be referred to as a single modal routing layer, and for routing layers for multiple modal types, this application can be referred to as a cross-modal routing layer, and the cross-modal routing layer can process data of multiple modal types. For the unimodal routing layer, it can be trained by training data (training task group) whose modality type is unimodal (such as text modality type, image modality type, audio modality type, etc.); for the cross-modal routing layer, it can be trained by training data (training task group) whose modality type is cross-modal.
[0098] Specifically, the present application can configure a set of training task groups. For example, a sample media data and its corresponding modal type can be combined into a training task group, or two sample media data of different modal types and their corresponding modal types can be combined into a training task group. Based on this, by combining the sample media data and the modal type, a set of training task groups containing single modal types and cross-modal types can be obtained. For example, when the sample media data is sample text data 1, its modal type is text modal type, then after combining them, a training task group of <sample text data 1, text modal type> can be obtained. The training task group only contains one text data, and the modal type of the training task group is text modal type (single modal type); for another example, when the sample media data is sample image data 1, its modal type is image modal type, then after combining them, a training task group of <sample image data 1, image modal type> can be obtained. The training task group contains only one text data, and the modality type of the training task group is an image modality type (single modality type); for another example, when the sample media data is sample text data 1 and sample image data 1, its modality type includes text modality type and image modality type, then after combining them, a training task group can be obtained as <sample text data 1-sample image data 1, text modality type-image modality type>, which contains both a text data and an image data, and the modality type of the training task group is a cross-modality type. In other words, the training task group in this application can include a single-modality type training task group and a cross-modality type training task group. For a single-modality type training task group, it will only include sample media data of one modality type, which can be a text modality type, an image modality type, an audio modality type, etc.; for a cross-modality type training task group, it needs to include sample media data of two or more different modality types.
[0099] Furthermore, for each training task group, the modal representation model can be called to perform attention interaction processing on it, where the attention interaction processing here can refer to multi-head self-attention processing based on the multi-head self-attention mechanism. Since the multi-head self-attention processing can make each element in the input data fully interact with other elements, this application calls the multi-head self-attention processing attention interaction processing, and the result obtained through the attention interaction processing is called the attention representation vector corresponding to the training task group. Subsequently, for each attention representation vector, based on the modality type to which the training task group belongs, a target routing layer that matches it can be determined in the single-modal routing layer and the cross-modal routing layer contained in the modality representation model. Through the target routing layer, the attention representation vector of the training task group can be routed (feature prediction processing), and finally the modality representation vector corresponding to the training task group can be obtained (the modality representation vector corresponding to the training task group refers to the feature vector output by the target routing layer for representing the modality type. Since the modality representation model needs to be trained in the training phase, the modality representation vector output by the target routing layer can be understood as a predicted value, and the output modality representation vector can be called a predicted modality representation vector. The target routing layer needs to be trained and optimized based on the predicted modality representation vector, so that the modality representation vector output by the modality representation model becomes more and more accurate). For example, assuming that the modality type of a training task group is the text modality type in the unimodal type, then after obtaining the attention representation vector of the training task group, the text modality routing layer can be determined as the target routing layer of the training task group in the unimodal routing layer contained in the modality representation model. Based on the text modality routing layer, the predicted modality representation vector corresponding to the training task group can be output, and based on the predicted modality representation vector corresponding to the training task group, the text modality routing layer can be trained and optimized accordingly.
[0100] That is to say, when the predicted modal representation vector corresponding to each training task group in the training task group set is determined, the corresponding single modal routing layer (such as text modal routing layer, image modal routing layer, audio modal routing layer or video modal routing layer) can be optimized based on the predicted modal representation vector corresponding to the training task group whose modal type is a single modal type (such as text modal type, image modal type, audio modal type or video modal type), and the optimized single modal routing layer can perform routing processing on single modal type data; similarly, based on the predicted modal representation vector corresponding to the training task group whose modal type is a cross-modal type, the cross-modal routing layer can be optimized, and the optimized cross-modal routing layer can perform routing processing on cross-modal type data.
[0101] It should be understood that the embodiment of the present application provides a multimodal joint training framework based on a modal routing distribution mechanism. On the basis of training only one modal representation model, it can simultaneously complete the tasks of outputting modal representation vectors of cross-modality and unimodality. Based on the modal representation features of each modality, media data of different modal types can be retrieved, thereby reducing problems such as large model storage, time-consuming retrieval, and repeated calculations, so that unimodal and cross-modal tasks can promote each other, maintain good performance respectively, and improve versatility and accuracy.
[0102] It is understandable that the method provided in the embodiment of the present application can be executed by a computer device, which includes but is not limited to the terminal device or business server mentioned in Figure 1.
[0103] It should be noted that, in the specific implementation of this application, data related to user information, user data (such as the information input by the above-mentioned objects) must be manually authorized by the user (i.e., with the user's consent) before they can be obtained. In other words, when the above embodiments of this application are applied to specific products or technologies, the methods and related functions provided by the embodiments of this application are operated with the user's permission or consent (the user can actively enable the functions provided by the embodiments of this application), and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant regions.
[0104] For ease of understanding, the data processing method provided by the embodiment of the present application will be described in detail with reference to the accompanying drawings. Please refer to Figure 2, which is a flow chart of a data processing method provided by an embodiment of the present application. Among them, the method can be executed by a computer device, which can be a terminal device (for example, any terminal device in the terminal device cluster shown in Figure 1 above, such as terminal device 100a), and the computer device can also be a server (such as the business server 1000 in the embodiment corresponding to Figure 1 above), and the computer device can also be a terminal device and a server. For ease of understanding, this embodiment is described by taking the method being executed by a terminal device as an example. As shown in Figure 2, the data processing method can at least include the following steps S101-step S104:
[0105] Step S101, obtain a training task group set; the training task group set includes a training task group S i , i is a positive integer; training task group S i The modality type is single modality type or cross modality type; in the training task group S i When the modality type is single modality type, the training task group S i Contains a sample media data; in the training task group S i When the modality type is cross-modality, the training task group Si Contains sample media data of at least two different modal types.
[0106] In the present application, a single modal type may refer to the modal type to which media data belongs, which may specifically include a text modal type, an image modal type, an audio modal type, and a video modal type. The modal type to which a certain training task group belongs may be determined based on the sample media data contained in the training task group. For example, if the training task group only contains one sample media data, and the modal type to which the sample media data belongs is a text modal type (i.e., the sample media data is sample text data), then the modal type to which the training task group belongs may be a single modal type, and may specifically be a text modal type. For another example, if the training task group only contains one sample media data, and the modal type to which the sample media data belongs is an image modal type (i.e., the sample media data is sample image data), then the modality to which the training task group belongs may be a single modal type, and may specifically be an image modal type. In other words, when a certain training task group only contains sample media data of one modal type, the modal type to which the training task group belongs may be a single modal type, and the specific modal type is determined based on the modal type of the sample media data contained therein. Similarly, it can be deduced that when a training task group contains two or more sample media data, and these sample media data belong to different modal types, it can be determined that the modal type to which the training task group belongs is a cross-modal type (or multi-modal type). For example, if a training task group contains both sample text data (sample media data with a modal type of text modality) and sample video data (sample media data with a modal type of video modality), then it can be determined that the modal type to which the training task group belongs is a cross-modal type.
[0107] That is to say, the sample media data in this application may refer to media data serving as training samples, which may specifically include sample text data, sample image data, sample audio data or sample video data. When a training task group is composed of sample media data of one modal type, it can be determined that the modal type to which the training task group belongs is a single modal type, and the modal type to which the sample media data belongs can be determined as the modal type to which the training task group belongs; and when a training task group is composed of sample media data of multiple (two or more) modal types, it can be determined that the modal type to which the training task group belongs is a cross-modal type or a multi-modal type.
[0108] Specifically, the specific implementation method for obtaining the training task group set can be: N (N is a positive integer) sample media data can be obtained; wherein, the modal type to which one sample media data among the N media data belongs is any one of the first modal type and the second modal type; and the first modal type is different from the second modal type; based on the modal type to which each sample media data among the N sample media data belongs, the N sample media data can be subjected to task group construction processing, thereby obtaining the training task group set.
[0109] It can be understood that the first modality type and the second modality type here may refer to different modality types of media data, and both the first modality type and the second modality type may refer to text modality type, image modality type, audio modality type or video modality type, and the first modality type is different from the second modality type. For example, when the first modality type is a text modality type, the second modality type may refer to any one or more of image modality type, audio modality type or video modality type. When the first modality type is an image modality type, the second modality type may refer to any one or more of text modality type, audio modality type or video modality type. In the present application, a plurality of sample media data of different modality types are obtained in advance as samples, and then the task group construction process is performed based on the modality type to which each sample media data belongs, thereby obtaining a training task group containing different sample media data, and then obtaining a training task group set.
[0110] Specifically, for N sample media data, based on the modal type to which each sample media data belongs, the N sample media data are subjected to task group construction processing to obtain a specific implementation method of a training task group set. The first sample media data of the N sample media data can be combined with a first identifier for representing the first modal type, thereby obtaining a first training task group whose modal type is the first modal type; wherein the first sample media data refers to any sample media data of the N sample media data whose modal type is the first modal type; it should be understood that the N sample media data contain media data of the first modal type and a first identifier for representing the first modal type. For media data of the second modal type, any media data whose modal type is the first modal type can be referred to as first sample media data. Then, for any first sample media data, it can be combined with the identifier of the first modal type, thereby obtaining a training task group (this training task group can be referred to as the first training task group). Since the number of first sample media data may not be one, the number of first training task groups may also be multiple. Similarly, the second sample media data in N sample media data can be combined with the second identifier used to characterize the second modal type, thereby obtaining a training task group whose modal type is the first. A second training task group of two modal types; wherein the second sample media data refers to any sample media data of the N sample media data whose modal type is the second modal type; it should be understood that the N sample media data include media data of the first modal type and media data of the second modal type, and any media data whose modal type is the second modal type can be referred to as second sample media data. Then, for any second sample media data, it can be combined with the identifier of the second modal type, thereby obtaining a training task group (this training task group can be referred to as the second training task group). Since the second sample media data The number of data may not be one, so the number of second training task groups may also be multiple; based on the media source channels corresponding to the first sample media data and the second sample media data respectively, the first sample media data and the second sample media data can be cross-modally combined to obtain a training task group whose modality type is a cross-modal type; further, the first training task group and the second training task group can both be determined as training task groups whose modality type is a single modality type, and the set consisting of the training task group whose modality type is the cross-modal type and the training task group whose modality type is the single modality type can be determined as a training task group set.
[0111] It can be understood that, taking the first modality type as a text modality type and the second modality type including an image modality type and a video modality type as an example, N sample media data may include sample text data 1 (modality type is a text modality type), sample text data 2 (modality type is a text modality type), sample image data 1 (modality type is an image modality type), sample video data 1 (modality type is a video modality type), and sample video data 2 (modality type is a video modality type). Among them, the identifier used to characterize the text modality type can be "text", the identifier used to characterize the image modality type can be "vision", and the identifier used to characterize the video modality type can be "video". Then the sample text data 1 and the identifier "text" used to characterize the text modality type can be combined to obtain a training task group of <sample text data 1, text>, or the sample text data 1 and the identifier "text" used to characterize the text modality type can be combined to obtain a training task group of <sample text data 2, text>. It should be understood that since the training task groups <sample text data 1, text> and <sample text data 2, text> only contain sample media data of the text modality type, the modality type to which these two training task groups belong can be a single modality type, and the modality type to which they belong can specifically be a text modality. mode type; similarly, the sample image data 1 can be combined with the identifier "vision" for characterizing the image modality type, thereby obtaining a training task group of <sample image data 1, vision>, which only contains sample media data of the image modality type. The modality type to which this training task group belongs can be a single modality type, and the modality type to which it belongs can specifically be an image modality type; similarly, the sample video data 1 and the sample video data 2 can be respectively combined with the identifier "video" for characterizing the video modality type, thereby obtaining training task groups of <sample video data 1, video> and <sample video data 2, video>. Both of these training task groups only contain sample media data of the video modality type. The modality type to which this training task group belongs can be a single modality type, and the modality type to which it belongs can specifically be a video modality type. That is, for sample media data, the sample media data of a certain modality type can be combined with the identifier of its corresponding modality type to obtain a training task group with a unimodal modality type. Similarly, sample media data of different modality types can be cross-modally combined to obtain a training task group with a multimodal modality type (or cross-modal modality type). In other words, a training task group with a unimodal modality type should contain a sample media data and an identifier of the modality type to which the sample media data belongs.
[0112] Specifically, sample media data of different modal types with the same media source channel (the media source channel may refer to the method or location for obtaining the sample media data) may be combined, so that the number of the first sample media data and the number of the second sample media data are both one or more, and the one or more first sample media data include the first sample media data M j (j is a positive integer), the modality type is a cross-modal training task group, including the first sample media data M j For example, for the media source channels corresponding to the first sample media data and the second sample media data, the first sample media data and the second sample media data are subjected to cross-modal combination processing to obtain a training task group of a cross-modal type. The specific implementation method can be as follows: the first sample media data M j The corresponding media source channel is determined as the target media source channel; then, the second sample media data whose media source channel is the target media source channel in one or more second sample media data can be determined as the first sample media data M j Finally, the first sample media data, the associated sample media data, the first identifier and the second identifier can be combined to obtain the first sample media data M j The corresponding training task group.
[0113] For example, assuming that N sample media data include sample text data a, sample text data b, sample image data a, sample video data h1, and sample video data h2, and the sample text data a and the sample image data a both come from the sample video data h1 (the sample text data a is the video description information of the sample video data h1, and the sample image data a is a certain picture frame of the sample video data h1), then the sample text data a and the sample image data a can be determined to be mutually related (the sample text data a is the associated media data of the sample image data a, and similarly, the sample image data a is also the associated media data of the sample text data a), the sample text data a and the sample video data h1 are also mutually related, and the sample image data a and the sample video data h1 are also mutually related. Based on this, the sample text data a and the sample image data a and their corresponding identifiers (used to characterize the text modality) can be identified. The identifier "text" of the type is combined with the identifier "vision" used to characterize the image modality type to obtain a training task group of <sample text data a-sample image data a, text-vision>. The sample text data a can also be combined with the sample video data h1 and its corresponding identifier (the identifier "text" used to characterize the text modality type and the identifier "video" used to characterize the video modality type) to obtain a training task group of <sample text data a-sample video data h1, text-video>. The sample image data a can also be combined with the sample video data h1 and its corresponding identifier (the identifier "vision" used to characterize the image modality type and the identifier "video" used to characterize the video modality type) to obtain a training task group of <sample image data a-sample video data h1, vision-video>. It can be seen that a training task group whose modality type is a cross-modal type should contain two or more sample media data and identifiers of the modality type to which each sample media data belongs.
[0114] From the above, it can be seen that the task group construction process in the present application refers to combining the sample media data and the identifier of the modal type according to the modal type to which each sample media data belongs and the identifier of the modal type to which each sample media data belongs, so as to obtain a training task group whose modal type is a single modal type and a training task group whose modal type is a cross-modal type. When the present application combines the sample media data to obtain the training task group, the format of each training task group is unified. Whether it is a single modal type training task group or a multi-modal type training task group, it is obtained by combining the sample media data and its corresponding identifier for characterizing the modal type. Through the identifier for characterizing the modal type, it can be clearly indicated whether the modal type contained in the training task group is a single modal type or a cross-modal type, and thus it can be reflected whether the modal type to which the training task group belongs is a single modal type or a cross-modal type. In addition, the training task group S in the present application i It can refer to any training task group in the training task group set. For a training task group of a single-modality type in the training task group set, it is mainly composed of sample media data of one modality type (such as any one of a text modality type, an image modality type, a video modality type, or an audio modality type), and the number of the sample media data is one. Then, for a training task group in the training task group set (such as training task group S i ), when the training task group is of a single-modal type, the training task group may only contain one sample media data; and for a training task group in the training task group set whose modality type is a cross-modal type, it needs to be composed of sample media data of at least two different modal types, and the number of sample media data under each modality type needs to be at least one, then for a training task group in the training task group set (such as training task group S i ), when the training task group is of a cross-modal type, the training task group should contain at least two sample media data of different modal types.
[0115] Step S102: training task group S in the modality representation model i Perform attention interaction processing to obtain the training task group S i The corresponding attention representation vector; attention interaction processing is used to group the training task S i The different elements in the game interact fully.
[0116] In the present application, the modal representation model may refer to a model based on the transformer architecture, through which feature extraction modeling processing can be performed on each training task group. The feature extraction modeling processing can make the elements of each feature level in the training task group fully interact, so that each element can pay more attention to the elements that are more closely related to itself, so the feature extraction modeling processing can also be called attention interaction processing. Specifically, the entire network structure of the transformer architecture is entirely composed of an attention mechanism (Attention mechanism). More precisely, the transformer architecture consists of and only consists of self-Attenion and a feedforward neural network (FFN). In the present application, a trainable neural network based on the transformer (i.e., a modal representation model) can be built in the form of stacked transformers. The use of the Attention mechanism can solve the computational context forgetting problem of traditional RNNs (or LSTMs, GRUs, etc.). To facilitate understanding of the transformer architecture, please refer to Figure 3, which is an architectural schematic diagram of a transformer architecture provided in an embodiment of the present application. As shown in Figure 3, the transformer architecture includes an encoder and a decoder. The encoder may include a multi-head attention layer and a routing layer. The multi-head attention layer can be used to process the input features with multi-head attention, and the routing layer in the encoder can be used to route the content processed by the multi-head attention layer. The routing layer here can refer to an FFN, specifically a multi-layer fully connected layer. The routing layer can be used to fully connect the content processed by the multi-head attention layer (feature integration processing, also called routing processing). Furthermore, the features encoded by the encoder (i.e., the content output by the routing layer) can be input into the decoder, and the decoder can decode the features encoded by the encoder. Finally, the content decoded and output by the decoder can be used as the modal representation vector corresponding to the input features.
[0117] Based on the above, the modal representation model can include a multi-head attention layer, which can be determined as a multi-head self-attention network layer in this application. Here, in the modal representation model, the training task group S i Perform attention interaction processing, which can be regarded as the training task group S in the multi-head self-attention network layer i The specific implementation method is as follows: In the modality representation model, the training task group S can be processed by the feature extraction network layer. i The sample media data contained therein is subjected to feature extraction processing, thereby obtaining the training task group S iCorresponding media features; further, the training task group S can be i The corresponding media features are processed by multi-head self-attention, and the training task group S is obtained. i The corresponding attention representation vector.
[0118] It can be understood that the feature extraction network layer here can refer to the embedding layer, through which the features of the sample media data in the training task group can be extracted to obtain media features. Among them, the media features here can be determined based on the modal type to which the sample media data belongs. When the sample media data is a text modal type, the media features of the sample media data can refer to the word features corresponding to each text word; when the sample media data is an image modal type, the media features of the sample media data can refer to pixel features (an image can be divided into multiple pixel networks, and the media features can be composed of the pixel features corresponding to each pixel grid); when the sample media data is an audio modal type, the media features of the sample media data can refer to phoneme features; when the sample media data is a video modal type, the media features of the sample media data can be composed of frame features corresponding to the video frame. When the sample media data is a cross-modal type, the media features of the training task group can be composed of the media features of sample media data of different modal types, that is, it can be composed of two or more features among word features, pixel features, phoneme features, and frame features.
[0119] The multi-head self-attention network layer can include Q (Q is a positive integer) self-attention sub-network layers. The output result of the multi-head self-attention network layer can be obtained by fusing the output results of these self-attention sub-network layers. Specifically, the Q self-attention sub-network layers include the self-attention sub-network layer V k (k is a positive integer) as an example, for the multi-head self-attention network layer in the modal representation model, for the training task group S i The corresponding media features are processed by multi-head self-attention to obtain the training task group S i The specific implementation of the corresponding attention representation vector can be: the self-attention sub-network layer V k The included attention parameter matrix can then be used to self-attention sub-network layer V k The included attention parameter matrix is related to the training task group S i The corresponding media features are processed, and the linear transformation matrix corresponding to the attention parameter matrix can be obtained; further, the self-attention sub-network layer V k The fully connected components in the linear transformation matrix perform feature integration processing, thereby obtaining the self-attention sub-network layer Vk The corresponding attention representation sub-vector; when the attention representation sub-vectors corresponding to the Q self-attention sub-network layers are determined, the Q attention representation sub-vectors can be fused to obtain the training task group S i The corresponding attention representation vector.
[0120] It is understandable that in a certain self-attention sub-network layer, the attention parameter matrix included may include W Q 、W K and W V Matrix, the training task group S i The corresponding media features are respectively Q 、W K and W V By multiplying, we can get W Q The linear transformation matrix corresponding to the matrix, W K The linear transformation matrix corresponding to the matrix and W V The linear transformation matrix corresponding to the matrix. For the linear transformation matrix W Q With the linear transformation matrix W K , they can be multiplied, and the result of the multiplication of the two can be input into the fully connected layer (such as the Softmax layer) for full connection processing (that is, feature integration processing). The result of the full connection processing can be compared with the linear transformation matrix W V Perform multiplication operation again and linear transformation matrix W V The result of the multiplication operation can be used as the output result of a self-attention sub-network layer (i.e., the attention representation sub-vector). Each self-attention sub-network layer contains different W Q 、W K and W V Matrix, using the same principle, different attention representation sub-vectors can be obtained. Finally, these different attention representation sub-vectors can be fused to obtain the final attention representation vector.
[0121] For the attention representation sub-vectors based on each self-attention sub-network layer, the specific implementation method of obtaining the final attention representation vector can be shown as formula (1):
[0122] Among them, E attn Can be used to represent the final attention representation vector; It can be used to represent the attention representation sub-vector of the self-attention sub-network layer Hn; Concat() can be used to represent the splicing process. In other words, the attention representation sub-vectors can be fused (spliced) to obtain the attention representation vector after fusion (splicing).
[0123] Step S103: Determine the training task group S in the single-modal routing layer and the cross-modal routing layer included in the modality representation model. i The target routing layer that matches the modality type belongs to, and the training task group S is based on the target routing layer i The corresponding attention representation vector is processed for feature prediction to obtain the training task group S i The corresponding predicted modal representation vector.
[0124] In this application, based on the above, it can be seen that the encoder and decoder included in the modal representation model will include a multi-head attention layer and a routing layer. The routing layer can perform feature integration processing (i.e. routing processing, referred to as feature prediction processing in this application) on the attention representation vector output by the multi-head attention layer to obtain the final modal representation vector. The modal representation vector refers to the features obtained after reasoning and understanding the training task group. In order to improve the accuracy of understanding data of different modal types, this application can set the routing layer in the modal representation model to different modal routing layers, and connect each modal routing layer to different modal types. Type corresponding to, for example, the routing layer can be set to text modality routing layer, image modality routing layer, audio modality routing layer, video modality routing layer and cross-modality routing layer, the text modality routing layer can be used to process data with a modality type of text modality type, the image modality routing layer can be used to process data with a modality type of image modality type, the audio modality routing layer can be used to process data with a modality type of audio modality type, the video modality routing layer can be used to process data with a modality type of video modality type, and the cross-modality routing layer can process data with a modality type of cross-modality type. The text modality routing layer, image modality routing layer, audio modality routing layer and video modality routing layer can all be understood as a single modality routing layer, that is to say, in the modal representation model, a single modality routing layer and a cross-modality routing layer can be included at the same time, then based on this, for the training task group S i In terms of determining the training task group S i After the corresponding attention representation vector is obtained, the attention representation vector corresponding to the training task group S can be determined in the single-modal routing layer and cross-modal routing layer contained in the modality representation model. i The modal routing layer that matches the modality type is used as the target routing layer, and then the training task group S is trained based on the target routing layer. i The corresponding attention representation vector is processed for feature prediction to obtain the training task group Si The corresponding modal representation vector (predicted modal representation vector).
[0125] Taking the example of a single modality type including the first modality type and the second modality type, and the single modality routing layer including the first modality sub-routing layer corresponding to the first modality type and the second modality sub-routing layer corresponding to the second modality type, for the single modality routing layer and the cross-modality routing layer included in the modality representation model, determine the training task group S i The specific implementation of the target routing layer that matches the modality type can be as follows: if the training task group S i If the modality type is a cross-modal type, the cross-modal routing layer included in the modality representation model can be determined as the same as the training task group S i The target routing layer that matches the modality type belongs to; if the training task group S i If the modality type is the first modality type, the first modality sub-routing layer corresponding to the first modality type can be determined as the first modality sub-routing layer corresponding to the training task group S i The target routing layer that matches the modality type belongs to; and if the training task group S i If the modality type is the second modality type, the second modality sub-routing layer corresponding to the second modality type can be determined as the training task group S i The target routing layer that matches the modality type.
[0126] It is understandable that the first modality type and the second modality type may refer to different modality types. For example, the first modality type may refer to one or more of a text modality type, an image modality type, an audio modality type, and a video modality type, while the second modality type may refer to the remaining modality types among the text modality type, the image modality type, the audio modality type, and the video modality type except the first modality type. Then, in the training task group S i When the modality type is text modality type, the text modality routing layer can be determined as the training task group S i The corresponding target routing layer; in the training task group S i When the modality type is an image modality type, the image modality routing layer can be determined as the training task group S i The corresponding target routing layer; in the training task group S i When the modality type is an audio modality type, the audio modality routing layer can be determined as the training task group S i The corresponding target routing layer; in the training task group S i When the modality type is a video modality type, the video modality routing layer can be determined as the training task group S i The corresponding target routing layer; in the training task group S iWhen the modality type is cross-modal, the cross-modal routing layer can be determined as the training task group S i The corresponding target routing layer.
[0127] To facilitate understanding of the routing distribution mechanism, please refer to Figure 4, which is a schematic diagram of the architecture of a modal routing distribution provided in an embodiment of the present application. As shown in Figure 4, the single-modal routing layer can specifically be a text modal routing layer, an image modal routing layer, an audio modal routing layer, or a video modal routing layer, which can be used to route data of the corresponding modal type. When the input feature is a media feature corresponding to the text modality type, after the attention representation vector is output by the multi-head attention layer, it can be distributed to the text modality routing layer, which will perform routing processing on it. When the input feature is a media feature corresponding to the image modality type, after the attention representation vector is output by the multi-head attention layer, it can be distributed to the image modality routing layer, which will perform routing processing on it. When the input feature is a media feature corresponding to the audio modality type, after the attention representation vector is output by the multi-head attention layer, it can be distributed to the audio modality routing layer, which will perform routing processing on it. When the input feature is a media feature corresponding to the video modality type, after the attention representation vector is output by the multi-head attention layer, it can be distributed to the video modality routing layer, which will perform routing processing on it. When the input feature is a media feature corresponding to the multi-modality type, after the attention representation vector is output by the multi-head attention layer, it can be distributed to the cross-modal routing layer, which will perform routing processing on it. Each modality routing layer can refer to a multi-layer fully connected layer.
[0128] Step S104: When the predicted modal representation vector corresponding to each training task group in the training task group set is determined, the single-modal routing layer is optimized based on the predicted modal representation vector corresponding to the training task group whose modality type is a single-modal type, and the cross-modal routing layer is optimized based on the predicted modal representation vector corresponding to the training task group whose modality type is a cross-modal type; the optimized single-modal routing layer is used to perform feature prediction processing on the task group whose modality type is a single-modal type, and the optimized cross-modal routing layer is used to perform feature prediction processing on the task group whose modality type is a cross-modal type.
[0129] In the present application, when the predicted modal characterization vector corresponding to each training task group in the training task group set is determined, the corresponding modal routing layer can also be trained and optimized based on the predicted modal characterization vector corresponding to the training task group of the corresponding modal type. For example, the single-modal routing layer can be optimized based on the predicted modal characterization vector corresponding to the training task group whose modal type is a single modal type, and the cross-modal routing layer can be optimized based on the predicted modal characterization vector corresponding to the training task group whose modal type is a cross-modal type. The optimized single-modal routing layer can be used to perform feature prediction processing on the task group whose modal type is a single modal type; the optimized cross-modal routing layer can be used to perform feature prediction processing on the task group whose modal type is a cross-modal type. Specifically, based on the predicted modal representation vector corresponding to the training task group whose modality type is text modality type, the text modality routing layer can be trained and optimized; based on the predicted modal representation vector corresponding to the training task group whose modality type is image modality type, the image modality routing layer can be trained and optimized; based on the predicted modal representation vector corresponding to the training task group whose modality type is audio modality type, the audio modality routing layer can be trained and optimized; based on the predicted modal representation vector corresponding to the training task group whose modality type is video modality type, the video modality routing layer can be trained and optimized. The text modality routing layer after training and optimization can perform feature prediction processing on text data; the image modality routing layer after training and optimization can perform feature prediction processing on image data; the audio modality routing layer after training and optimization can perform feature prediction processing on audio data; and the video modality routing layer after training and optimization can perform feature prediction processing on video data.
[0130] Taking the cross-modal target task group as an example, it can be processed based on the optimized cross-modal routing layer. Specifically, a target task group for describing the media data to be classified can be obtained; wherein the target task group includes media data of at least two different modal types; then, the target task group can be subjected to attention interaction processing in the modal representation model, thereby obtaining the attention representation vector corresponding to the target task group; further, the attention representation vector corresponding to the target task group can be subjected to feature prediction processing in the optimized cross-modal routing layer, thereby obtaining the predicted modal representation vector corresponding to the target task group; then, the predicted modal representation vector corresponding to the target task group can be subjected to category recognition processing, thereby obtaining the media category to which the media data to be classified belongs. After determining the media category to which the media data to be classified belongs, the media data to be classified can be archived according to the media category to which it belongs.
[0131] It can be understood that the scenario of archiving media data is taken as an example here to illustrate the scenario in which the embodiments of the present application can be applied. Assuming that the media data to be classified is a certain video data, the text data and image data from the video data can be used as a task group, and then the cross-modal routing layer after the training optimization is used to reason and understand it to obtain an accurate modal representation vector of the video data. Based on the modal representation vector of the video data, the media category to which the video data belongs (such as dance category, competition category, gymnastics category, etc.) can be determined. Through the media category to which the video data belongs, the video data can be accurately archived.
[0132] Of course, the application scenario of the method provided in the embodiment of the present application is not limited to this. For example, it can also be applied to the retrieval scenario of multimodal content, the scenario of media data recommendation, etc. For ease of understanding, please refer to Figure 5, which is a scene diagram of media data retrieval provided in the embodiment of the present application. As shown in Figure 5, in the video display interface 3001, the video data currently exposed to object a is video data 300a (the creator of the video data 300a is an object named "La La Loves Dancing", and the video copy of the video data 300a is "Teach you to quickly learn a dance in three minutes"). The video display interface 3001 displays a like control, a comment control, and a share control, which can be used by the object to like, comment, and share the video data. In the video display interface 3001, a search control 30a is also included, which can be used by object a to search for related content. As shown in FIG3 , after object a triggers the search control 30a, the terminal device may respond to the trigger by displaying an input box in the video display interface 3001 , and object a may input desired content into the input box.
[0133] As shown in Figure 3, object a enters the text data "Braised Crucian Carp Recipe" in the input box, and the terminal device can obtain the text data "Braised Crucian Carp Recipe" entered by object a. The terminal device can send the text data "Braised Crucian Carp Recipe" to the business server, and the business server can infer and understand the text data "Braised Crucian Carp Recipe" by training the optimized modal representation model. Specifically, the multi-head self-attention network layer in the modal representation model can output the attention representation vector corresponding to the text data "Braised Crucian Carp Recipe", and the attention representation vector can be distributed to the text modal routing layer. Through the text modal routing layer, the modal representation vector corresponding to the text data "Braised Crucian Carp Recipe" can be output. Based on the modal representation vector corresponding to the text data, the related video data can be found. As shown in Figure 3, assuming that the related video data is video data 300b (the creator of the video data 300b is an object named "Meimei makes delicious food", and the video text of the video data 300b is "Buy crucian carp and cook it like this, it smells delicious!"), the business server can return the video data 300b to the terminal device, and the terminal device can display the video data 300b in the video display interface 3001.
[0134] It should be noted that after determining the modal representation vector corresponding to the text data, the business server can determine whether the two data are associated based on the similarity between different vectors. For example, for the modal representation vector corresponding to the above-mentioned text data "Braised Crucian Carp Recipe", the modal representation vector of the candidate video data can be obtained first, and then the modal representation vector corresponding to the text data "Braised Crucian Carp Recipe" can be calculated with the modal representation vector of each candidate video data. When the similarity is greater than the similarity threshold, the candidate video data can be determined to be the associated video data of the text data "Braised Crucian Carp Recipe".
[0135] In an embodiment of the present application, a general modal representation model based on a modal routing distribution mechanism is provided, which can provide corresponding data processing functions for data whose modal type is a single modal type and data whose modal type is a cross-modal type. Specifically, the present application can use a set of training task groups including single modal type and cross-modal type to train the modal representation model, wherein the modal representation model includes a single modal routing layer and a cross-modal routing layer, the single modal routing layer can perform feature prediction processing on data (attention representation vector) whose modal type is a single modal type, and the cross-modal routing layer can perform feature prediction processing on data (attention representation vector) whose modal type is a cross-modal type. During training optimization, the single modal routing layer can be optimized based on the predicted modal representation vector corresponding to the training task group whose modal type is the single modal type, and the cross-modal routing layer can be optimized by the predicted modal representation vector corresponding to the training task group whose modal type is the cross-modal type. Based on this, the single-modal routing layer and the cross-modal routing layer in the modal representation model can be trained and optimized at the same time, and during the training optimization, the training optimization is carried out separately based on data of different modal types, without losing any modal type of data, and the performance of the single-modal routing layer and the cross-modal routing layer can be maintained at the same time, thereby improving the versatility and flexibility of the modal representation model for single-modal type and cross-modal type. In summary, the present application can set a routing distribution mechanism in the modal representation model, divide the routing layer into a single-modal routing layer and a cross-modal routing layer, and different routing layers can process data of different modal types. During training, the data of different modal types are also trained and optimized separately. Each routing layer will only retain the relevant information of the current modal type, so there is no need to discard any modal type of data, and the representation vectors of cross-modal type and single-modal type can be output at the same time, which can greatly improve the accuracy of the representation results of data of various modal types. In summary, the present application can improve the versatility of the modal representation model and the accuracy of the modal representation results in the modal representation business.
[0136] Further, please refer to Figure 6, which is a schematic diagram of a process for optimizing a single-modal routing layer provided in an embodiment of the present application. Among them, the process may correspond to the embodiment corresponding to Figure 2 above, which is a process for optimizing the single-modal routing layer for the predicted modal representation vector corresponding to the training task group whose modality type is a single-modal type. The process is a process described as an example in which the single-modal type includes a first modal type and a second modal type, and the single-modal routing layer includes a first modal sub-routing layer corresponding to the first modal type and a second modal sub-routing layer corresponding to the second modal type. As shown in Figure 6, the process may include at least the following steps S601-S605:
[0137] In step S601, a training task group whose modality type is the first modality type is determined as the first modality training task group, and a training task group whose modality type is the second modality type is determined as the second modality training task group.
[0138] Specifically, for ease of distinction, the training task group whose modality type is the first modality type can be named the first modality training task group, and the training task group whose modality type is the second modality type can be named the second modality training task group.
[0139] Step S602: Obtain the true modal representation vector corresponding to the first modal training task group and the true modal representation vector corresponding to the second modal training task group.
[0140] Specifically, the real modal representation vector corresponding to the first modal training task group and the real modal representation vector corresponding to the second modal training task group can be obtained. The real modal representation vector can refer to the real modal representation vector used as the training label.
[0141] Step S603: Perform error calculation on the predicted modal representation vector corresponding to the first modal training task group and the true modal representation vector corresponding to the first modal training task group to obtain a first loss value corresponding to the first modal training task group.
[0142] Specifically, the predicted modal representation vector corresponding to the first modal training task group and the true modal representation vector corresponding to the first modal training task group can be subjected to error calculation processing (that is, the error value between the predicted modal representation vector and the true modal representation vector is calculated), thereby obtaining the first loss value corresponding to the first modal training task group. Taking the first modal type as the text modal type as an example, the first modal training task group can be a text word sequence, and the predicted modal representation vector corresponding to the first modal training task group is composed of the predicted representation features corresponding to each text word in the text word sequence, and the true modal representation vector corresponding to the first modal training task group is composed of the true representation features corresponding to each text word in the text word sequence. At this time, the predicted modal representation vector corresponding to the first modal training task group and the true modal representation vector corresponding to the first modal training task group are subjected to error calculation processing to obtain the specific first loss value corresponding to the first modal training task group. The implementation method may be as follows: a masked text word may be obtained from the text words contained in the text word sequence; wherein the masked text word refers to a text word that has been masked; subsequently, the predicted representation feature corresponding to the masked text word may be obtained in the predicted modal representation vector corresponding to the first modal training task group, and the real representation feature corresponding to the masked text word may be obtained in the real modal representation vector corresponding to the first modal training task group; a first feature similarity between the predicted representation feature corresponding to the masked text word and the real representation feature corresponding to the masked text word may be determined; subsequently, the first feature similarity may be determined as the first loss value corresponding to the first modal training task group.
[0143] It can be understood that for text word sequences, in this application, a masking strategy can be used to mask a certain text word. For example, the text words in the text word sequence that are noun attributes, verb attributes or position attributes can be masked, and the predicted modal representation vector output by the modal representation model includes the predicted representation features corresponding to the masked text word, and the real modal representation vector also includes the real representation features corresponding to the masked text word. The similarity between the predicted representation features and the real representation features can be calculated, and the similarity between the predicted representation features and the real representation features can be used as the first loss value corresponding to the first modal training task group.
[0144] Similarly, when the first modality type is an image modality type or an audio modality type, some elements in the first modality training task group can also be masked. For example, when the first modality type is an image modality type, the first modality training task group can be image data, and certain pixel areas in the image data can be masked (specifically, pixel areas with a higher mutual intersection ratio can be masked). The predicted modality representation vector output by the modality representation model includes the predicted representation feature corresponding to the masked pixel area, and the real modality representation vector also includes the real representation feature corresponding to the masked pixel area. The similarity between the predicted representation feature and the real representation feature can also be calculated. The similarity between the predicted representation feature and the real representation feature can be used as the first loss value corresponding to the first modality training task group.
[0145] It should be noted that for the first modal training task group, masking processing may not be performed, and the predicted modal representation vector and the true modal representation vector may be directly subjected to similarity calculation processing, and the resulting overall similarity may be used as the final first loss value. It should also be noted that in addition to obtaining the first loss value by calculating the similarity, the loss value between the predicted value and the true value may also be calculated using feature regression or regional classification targets. This application does not limit the method of calculating the loss value based on the predicted value and the true value.
[0146] Step S604: Perform error calculation on the predicted modal representation vector corresponding to the second modal training task group and the true modal representation vector corresponding to the second modal training task group to obtain a second loss value corresponding to the second modal training task group.
[0147] Specifically, error calculation is performed on the predicted modal representation vector corresponding to the second modal training task group and the true modal representation vector corresponding to the second modal training task group to obtain the second loss value corresponding to the second modal training task group. The specific method can refer to the description of determining the first loss value above. The principles of the two are the same and will not be repeated here.
[0148] Step S605: Optimize the first modality sub-routing layer corresponding to the first modality type based on the first loss value, and optimize the second modality sub-routing layer corresponding to the second modality type based on the second loss value.
[0149] Specifically, the first modal sub-routing layer corresponding to the first modal type can be optimized based on the first loss value, and the second modal sub-routing layer corresponding to the second modal type can be optimized based on the second loss value. In other words, for modal sub-routing layers of different modal types, the loss values obtained from the predicted values of the corresponding modal types can be used for optimization. Specifically, the parameters of each modal sub-routing layer can be adjusted based on the loss value until the loss value meets the convergence condition. Among them, the convergence condition here can refer to the number of iterations reaching the iteration threshold (this iteration threshold can be set based on actual business needs), or it can refer to the loss value being less than the loss threshold. The convergence condition can be set accordingly based on specific business needs. This application does not limit it, and it is specifically explained here.
[0150] In an embodiment of the present application, a routing distribution mechanism can be set in the modal representation model to divide the routing layer into a single-modal routing layer and a cross-modal routing layer. Different routing layers can process data of different modal types, and during training, data of different modal types are trained and optimized separately. Each routing layer will only retain relevant information of the current modal type, so there is no need to discard data of any modal type, and representation vectors of cross-modal types and single-modal types can be output at the same time, which can greatly improve the accuracy of the representation results of data of various modal types.
[0151] Further, please refer to Figure 7, which is a schematic diagram of a process for optimizing a cross-modal routing layer provided by an embodiment of the present application. Among them, the process can correspond to the embodiment corresponding to Figure 2 above, for the predicted modal representation vector corresponding to the training task group whose modality type is a cross-modal type, the process for optimizing the cross-modal routing layer, the process is based on the training task group set, including at least two training task groups whose modality type is a cross-modal type, at least two training task groups whose modality type is a cross-modal type, including training task group S j (j is a positive integer), training task group S j The process is described by taking the third sample media data of the first modality type and the fourth sample media data of the second modality type as an example. At this time, the training task group S j The corresponding predicted modality representation vector may include the predicted representation features corresponding to the third sample media data and the predicted representation features corresponding to the fourth sample media data. As shown in FIG7 , the process may include at least the following steps S701 to S703:
[0152] Step S701 : Determine a second feature similarity between a predicted representation feature corresponding to the third sample media data and a predicted representation feature corresponding to the fourth sample media data.
[0153] Specifically, the task of cross-modal retrieval is that after the object inputs a certain information (such as text data), the platform can recall the image / video / audio content associated with it. Therefore, the core idea of cross-modal retrieval is to align and make it consistent between different modalities, that is, the matching pictures and text content or audio content, video content are closest and similar in the feature space, and the unmatched pictures and text content or audio content, video content are inconsistent in the feature space. Based on this, for the training task group containing different modality types, it is necessary to determine the second feature similarity between the predicted representation feature corresponding to the third sample media data and the predicted representation feature corresponding to the fourth sample media data. The second feature similarity should be as large as possible, so that the matching text, image, audio, and video can be consistent in the feature space.
[0154] Step S702: Each training task group whose modality type is a cross-modality type is determined as a cross-modality training task group, and the training task group S is determined. j The third feature similarity between the remaining cross-modal training task groups in at least two cross-modal training task groups; the remaining cross-modal training task groups refer to the at least two cross-modal training task groups, except the training task group S j Cross-modal training task group other than .
[0155] Specifically, based on the above, for sample media data with different media source channels, it is necessary to control their similarity as small as possible. Therefore, for the training task group S j Compared with other cross-modal training task groups, the similarity of the third feature generated by it needs to be gradually reduced.
[0156] Step S703: Optimize the cross-modal routing layer based on the second feature similarity and the third feature similarity.
[0157] Specifically, based on the above, it can be seen that for the second feature similarity, in this application, it is necessary to maximize it by continuously adjusting the parameters of the cross-modal routing layer. At the same time, for the third feature similarity, it is necessary to minimize it by continuously adjusting the parameters of the cross-modal routing layer until the second feature similarity is the maximum value and the third feature similarity is the minimum value. It is determined that the cross-modal routing layer meets the convergence condition, and the parameters of the cross-modal routing layer can no longer be adjusted, and it is determined that the cross-modal routing layer at this time is optimal.
[0158] In an embodiment of the present application, a routing distribution mechanism can be set in the modal representation model to divide the routing layer into a single-modal routing layer and a cross-modal routing layer. Different routing layers can process data of different modal types, and during training, data of different modal types are trained and optimized separately. Each routing layer will only retain relevant information of the current modal type, so there is no need to discard data of any modal type, and representation vectors of cross-modal types and single-modal types can be output at the same time, which can greatly improve the accuracy of the representation results of data of various modal types.
[0159] Further, please refer to Figure 8, which is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application. The data processing device can be a computer program (including program code) running on a computer device, for example, the data processing device is an application software; the data processing device can be used to execute the method shown in Figure 2. As shown in Figure 8, the data processing device 1 may include: a set acquisition module 11, an attention processing module 12, a routing layer determination module 13, a prediction module 14, a first optimization module 15, and a second optimization module 16.
[0160] The set acquisition module 11 is used to acquire a training task group set; the training task group set includes a training task group S i , i is a positive integer; training task group S i The modality type is single modality type or cross modality type; in the training task group S i When the modality type is single modality type, the training task group S i Contains a sample media data; in the training task group S i When the modality type is cross-modality, the training task group S i Contains sample media data of at least two different modal types;
[0161] Attention processing module 12 is used to process the training task group S in the modality representation model i Perform attention interaction processing to obtain the training task group S i The corresponding attention representation vector; attention interaction processing is used to group the training task S i The different elements in the game fully interact with each other;
[0162] The routing layer determination module 13 is used to determine and train the task group S in the single-modal routing layer and the cross-modal routing layer included in the modality representation model. i The target routing layer that matches the modality type;
[0163] Prediction module 14, for training task group S based on target routing layer i The corresponding attention representation vector is processed for feature prediction to obtain the training task group Si The corresponding predicted modal representation vector;
[0164] A first optimization module 15 is configured to optimize the unimodal routing layer based on the predicted modal representation vector corresponding to the training task group whose modality type is unimodal when the predicted modal representation vector corresponding to each training task group in the training task group set is determined;
[0165] The second optimization module 16 is used to optimize the cross-modal routing layer based on the predicted modal representation vector corresponding to the training task group whose modality type is a cross-modal type; the optimized single-modal routing layer is used to perform feature prediction processing on the task group whose modality type is a single-modal type, and the optimized cross-modal routing layer is used to perform feature prediction processing on the task group whose modality type is a cross-modal type.
[0166] Among them, the specific implementation methods of the set acquisition module 11, the attention processing module 12, the routing layer determination module 13, the prediction module 14, the first optimization module 15 and the second optimization module 16 can be found in the description of steps S101 to S104 in the embodiment corresponding to Figure 2 above, and will not be repeated here.
[0167] In one embodiment, the specific implementation of the set acquisition module 11 acquiring the training task group set includes:
[0168] Obtain N sample media data; N is a positive integer, and a modality type of one sample media data in the N media data is any one of a first modality type and a second modality type; the first modality type and the second modality type are different;
[0169] Based on the modality type to which each sample media data belongs in the N sample media data, a task group construction process is performed on the N sample media data to obtain a training task group set.
[0170] In one embodiment, the set acquisition module 11 performs task group construction processing on the N sample media data based on the modality type to which each sample media data belongs, and obtains a specific implementation method of the training task group set, including:
[0171] Combining first sample media data of the N sample media data with a first identifier representing a first modality type to obtain a first training task group whose modality type is the first modality type; the first sample media data refers to any sample media data of the N sample media data whose modality type is the first modality type;
[0172] Combining second sample media data from the N sample media data with a second identifier representing the second modality type to obtain a second training task group whose modality type is the second modality type; the second sample media data refers to any sample media data from the N sample media data whose modality type is the second modality type;
[0173] Based on the media source channels corresponding to the first sample media data and the second sample media data, respectively, cross-modal combination processing is performed on the first sample media data and the second sample media data to obtain a training task group whose modality type is a cross-modal type;
[0174] The first training task group and the second training task group are both determined as training task groups whose modality type is a single modality type, and the set consisting of the training task group whose modality type is the cross-modality type and the training task group whose modality type is the single modality type is determined as a training task group set.
[0175] In one embodiment, the number of the first sample media data and the second sample media data is one or more; the one or more first sample media data include the first sample media data M j , j is a positive integer; the modality type is a cross-modal training task group, including the first sample media data M j corresponding training task group;
[0176] The specific implementation method of the set acquisition module 11 performing cross-modal combination processing on the first sample media data and the second sample media data based on the media source channels corresponding to the first sample media data and the second sample media data, and obtaining a training task group whose modality type is a cross-modal type, includes:
[0177] The first sample media data M j The corresponding media source channel is determined as the target media source channel;
[0178] The second sample media data whose media source channel is the target media source channel in one or more second sample media data is determined as the first sample media data M j Corresponding associated sample media data;
[0179] The first sample media data, the associated sample media data, the first identifier and the second identifier are combined to obtain the first sample media data M j The corresponding training task group.
[0180] In one embodiment, the attention processing module 12 performs the training task group S in the modality representation model. i Perform attention interaction processing to obtain the training task group S iThe specific implementation of the corresponding attention representation vector includes:
[0181] In the modal representation model, the training task group S is trained through the feature extraction network layer. i The sample media data contained therein is processed for feature extraction to obtain the training task group S i Corresponding media characteristics;
[0182] Through the multi-head self-attention network layer in the modality representation model, the training task group S i The corresponding media features are processed by multi-head self-attention to obtain the training task group S i The corresponding attention representation vector.
[0183] In one embodiment, the multi-head self-attention network layer in the modality representation model includes Q self-attention sub-network layers; Q is a positive integer; the Q self-attention sub-network layers include the self-attention sub-network layer V k , k is a positive integer;
[0184] The attention processing module 12 uses the multi-head self-attention network layer in the modality representation model to train the task group S i The corresponding media features are processed by multi-head self-attention to obtain the training task group S i The specific implementation of the corresponding attention representation vector includes:
[0185] Get the self-attention sub-network layer V k The included attention parameter matrix will be the self-attention sub-network layer V k The included attention parameter matrix is related to the training task group S i The corresponding media features are processed to obtain the linear transformation matrix corresponding to the attention parameter matrix;
[0186] Through the self-attention sub-network layer V k The fully connected components in the linear transformation matrix perform feature integration processing to obtain the self-attention sub-network layer V k The corresponding attention representation sub-vector;
[0187] When the attention representation sub-vectors corresponding to the Q self-attention sub-network layers are determined, the Q attention representation sub-vectors are fused to obtain the training task group S i The corresponding attention representation vector.
[0188] In one embodiment, the single modality type includes a first modality type and a second modality type; the single modality routing layer includes a first modality sub-routing layer corresponding to the first modality type and a second modality sub-routing layer corresponding to the second modality type;
[0189] The routing layer determination module 13 determines and trains the task group S in the single-modal routing layer and the cross-modal routing layer included in the modality representation model. i The specific implementation of the target routing layer that matches the modality type includes:
[0190] If the training task group S i If the modality type is a cross-modal type, the cross-modal routing layer included in the modality representation model is determined to be the same as the training task group S i The target routing layer that matches the modality type;
[0191] If the training task group S i If the modality type is the first modality type, the first modality sub-routing layer corresponding to the first modality type is determined to be the same as the training task group S i The target routing layer that matches the modality type;
[0192] If the training task group S i If the modality type is the second modality type, the second modality sub-routing layer corresponding to the second modality type is determined to be the same as the training task group S i The target routing layer that matches the modality type.
[0193] In one embodiment, the single modality type includes a first modality type and a second modality type; the single modality routing layer includes a first modality sub-routing layer corresponding to the first modality type and a second modality sub-routing layer corresponding to the second modality type;
[0194] The specific implementation of the first optimization module 15 for optimizing the unimodal routing layer based on the predicted modal representation vector corresponding to the training task group of which the modality type is unimodal includes:
[0195] Determine the training task group whose modality type is the first modality type as the first modality training task group, and determine the training task group whose modality type is the second modality type as the second modality training task group;
[0196] Obtaining a true modal representation vector corresponding to the first modal training task group and a true modal representation vector corresponding to the second modal training task group;
[0197] Performing error calculation processing on the predicted modal representation vector corresponding to the first modal training task group and the true modal representation vector corresponding to the first modal training task group to obtain a first loss value corresponding to the first modal training task group;
[0198] Performing error calculation processing on the predicted modal representation vector corresponding to the second modal training task group and the true modal representation vector corresponding to the second modal training task group to obtain a second loss value corresponding to the second modal training task group;
[0199] The first modality sub-routing layer corresponding to the first modality type is optimized based on the first loss value, and the second modality sub-routing layer corresponding to the second modality type is optimized based on the second loss value.
[0200] In one embodiment, the first modality type is a text modality type, and the first modality training task group is a text word sequence; the predicted modality representation vector corresponding to the first modality training task group is composed of the predicted representation features corresponding to each text word in the text word sequence; the real modality representation vector corresponding to the first modality training task group is composed of the real representation features corresponding to each text word in the text word sequence;
[0201] The first optimization module 15 performs error calculation processing on the predicted modal representation vector corresponding to the first modal training task group and the true modal representation vector corresponding to the first modal training task group to obtain the first loss value corresponding to the first modal training task group. The specific implementation method includes:
[0202] Obtaining a masked text word from the text words included in the text word sequence; the masked text word refers to a text word that has been masked;
[0203] Obtaining the predicted representation features corresponding to the masked text words in the predicted modal representation vector corresponding to the first modal training task group, and obtaining the real representation features corresponding to the masked text words in the real modal representation vector corresponding to the first modal training task group;
[0204] Determining a first feature similarity between a predicted representation feature corresponding to the masked text word and a true representation feature corresponding to the masked text word;
[0205] The first feature similarity is determined as a first loss value corresponding to the first modality training task group.
[0206] In one embodiment, the training task group set includes at least two training task groups whose modality type is cross-modality type; at least two training task groups whose modality type is cross-modality type, including training task group S j , j is a positive integer; training task group S j The training task group S includes first sample media data of a first modality type and second sample media data of a second modality type. j The corresponding predicted modal representation vector includes the predicted representation features corresponding to the first sample media data and the predicted representation features corresponding to the second sample media data;
[0207] The specific implementation of the second optimization module 16 for optimizing the cross-modal routing layer based on the predicted modality representation vector corresponding to the training task group whose modality type is a cross-modal type includes:
[0208] Determining a second feature similarity between a predicted characterizing feature corresponding to the first sample media data and a predicted characterizing feature corresponding to the second sample media data;
[0209] Each training task group whose modality type is a cross-modal type is determined as a cross-modal training task group, and the training task group S is determined. j The third feature similarity between the remaining cross-modal training task groups in at least two cross-modal training task groups; the remaining cross-modal training task groups refer to the at least two cross-modal training task groups, except the training task group S j Cross-modal training task force other than
[0210] Based on the second feature similarity and the third feature similarity, the cross-modal routing layer is optimized.
[0211] In one embodiment, the data processing device 1 may further include: a task group acquisition module 17 , an interaction processing module 18 , a feature processing module 19 and a category identification module 20 .
[0212] The task group acquisition module 17 is used to acquire a target task group for describing the media data to be classified; the target task group includes at least two media data of different modal types;
[0213] An interaction processing module 18 is used to perform attention interaction processing on the target task group in the modality representation model to obtain an attention representation vector corresponding to the target task group;
[0214] A feature processing module 19 is used to perform feature prediction processing on the attention representation vector corresponding to the target task group in the optimized cross-modal routing layer to obtain a predicted modality representation vector corresponding to the target task group;
[0215] The category identification module 20 is used to perform category identification processing on the predicted modal representation vector corresponding to the target task group to obtain the media category to which the media data to be classified belongs; the media category to which the media data to be classified belongs is used to archive the media data to be classified.
[0216] Among them, the specific implementation methods of the task group acquisition module 17, the interaction processing module 18, the feature processing module 19 and the category identification module 20 can be found in the description of step S104 in the embodiment corresponding to Figure 2 above, and will not be repeated here.
[0217] In an embodiment of the present application, a routing distribution mechanism can be set in the modal representation model to divide the routing layer into a single-modal routing layer and a cross-modal routing layer. Different routing layers can process data of different modal types, and during training, data of different modal types are trained and optimized separately. Each routing layer will only retain relevant information of the current modal type, so there is no need to discard data of any modal type, and representation vectors of cross-modal types and single-modal types can be output at the same time, which can greatly improve the accuracy of the representation results of data of various modal types.
[0218] Further, please refer to Figure 9, which is a structural diagram of a computer device provided in an embodiment of the present application. As shown in Figure 9, the above-mentioned computer device 8000 may include: a processor 8001, a network interface 8004 and a memory 8005. In addition, the above-mentioned computer device 8000 also includes: a user interface 8003, and at least one communication bus 8002. Among them, the communication bus 8002 is used to realize the connection and communication between these components. Among them, the user interface 8003 may include a display screen (Display), a keyboard (Keyboard), and the optional user interface 8003 may also include a standard wired interface and a wireless interface. The network interface 8004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 8005 may be a high-speed RAM memory or a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 8005 may also be at least one storage device located away from the aforementioned processor 8001. As shown in Figure 9, the memory 8005 as a computer-readable storage medium may include an operating system, a network communication module, a user interface module and a device control application.
[0219] In the computer device 8000 shown in FIG9 , the network interface 8004 can provide network communication functions; the user interface 8003 is mainly used to provide an interface for user input; and the processor 8001 can be used to call the device control application stored in the memory 8005 to achieve:
[0220] Get the training task group set; the training task group set includes the training task group S i , i is a positive integer; training task group S i The modality type is single modality type or cross modality type; in the training task group S i When the modality type is single modality type, the training task group S i Contains a sample media data; in the training task group S i When the modality type is cross-modality, the training task group S iContains sample media data of at least two different modal types;
[0221] In the modal representation model, the training task group S i Perform attention interaction processing to obtain the training task group S i The corresponding attention representation vector; attention interaction processing is used to group the training task S i The different elements in the game fully interact with each other;
[0222] In the single-modal routing layer and cross-modal routing layer included in the modal representation model, the task group S is determined and trained. i The target routing layer that matches the modality type belongs to, and the training task group S is based on the target routing layer i The corresponding attention representation vector is processed for feature prediction to obtain the training task group S i The corresponding predicted modal representation vector;
[0223] When the predicted modal representation vector corresponding to each training task group in the training task group set is determined, the single-modal routing layer is optimized based on the predicted modal representation vector corresponding to the training task group whose modality type is a single-modal type, and the cross-modal routing layer is optimized based on the predicted modal representation vector corresponding to the training task group whose modality type is a cross-modal type; the optimized single-modal routing layer is used to perform feature prediction processing on the task group whose modality type is a single-modal type, and the optimized cross-modal routing layer is used to perform feature prediction processing on the task group whose modality type is a cross-modal type.
[0224] It should be understood that the computer device 8000 described in the embodiments of the present application can execute the description of the data processing method in the embodiment corresponding to FIG. 2 above, and can also execute the description of the data processing device 1 in the embodiment corresponding to FIG. 8 above, and the description thereof will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated here.
[0225] In addition, it should be pointed out here that: the embodiment of the present application also provides a computer-readable storage medium, and the above-mentioned computer-readable storage medium stores a computer program executed by the computer device 8000 for data processing mentioned above, and the above-mentioned computer program includes program instructions. When the above-mentioned processor executes the above-mentioned program instructions, it can execute the description of the above-mentioned data processing method in the embodiment corresponding to Figure 2 above, so it will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application.
[0226] The computer-readable storage medium may be the data processing device provided in any of the aforementioned embodiments or the internal storage unit of the computer device, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device. Furthermore, the computer-readable storage medium may also include both the internal storage unit of the computer device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.
[0227] In one aspect of the present application, a computer program product is provided, comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the method provided in one aspect of the embodiments of the present application.
[0228] The terms "first", "second", etc. in the description, claims, and drawings of the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "comprising" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, apparatus, product, or device comprising a series of steps or units is not limited to the listed steps or modules, but may optionally include steps or modules not listed, or may optionally include other step units inherent to these processes, methods, apparatuses, products, or devices.
[0229] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0230] The method and related apparatus provided in the embodiment of the present application are described with reference to the method flow chart and / or structural diagram provided in the embodiment of the present application, and specifically can be implemented by computer program instructions for each process and / or box of the method flow chart and / or structural diagram, and the combination of the process and / or box in the flow chart and / or block diagram. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the function specified in one process or multiple processes of the flow chart and / or one box or multiple boxes of the structural diagram. These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device, and the instruction device implements the function specified in one process or multiple processes of the flow chart and / or one box or multiple boxes of the structural diagram. These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the structural diagram.
[0231] The above disclosure is only a preferred embodiment of the present application, and certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.
Claims
1. A data processing method, It is characterized in that The method is performed by a computer device, and comprises: Get a training task group set; the training task group set includes a training task group S i , i is a positive integer; the training task group S i The modality type is a single modality type or a cross-modality type; in the training task group S i When the modality type is the single modality type, the training task group S i Contains a sample media data; in the training task group S i When the modality type is the cross-modality type, the training task group S i Contains sample media data of at least two different modality types; In the modal representation model, the training task group S i Perform attention interaction processing to obtain the training task group S i The attention interaction process is used to group the training task S i The different elements in the game interact fully with each other; In the single-modal routing layer and the cross-modal routing layer included in the modality representation model, determine the training task group S i The target routing layer that matches the modality type belongs to, and the training task group S based on the target routing layer i The corresponding attention representation vector is processed for feature prediction to obtain the training task group S i The corresponding predicted modal representation vector; When the predicted modal representation vector corresponding to each training task group in the training task group set is determined, the single modal routing layer is optimized based on the predicted modal representation vector corresponding to the training task group whose modality type is the single modal type, and the cross-modal routing layer is optimized based on the predicted modal representation vector corresponding to the training task group whose modality type is the cross-modal type; the optimized single modal routing layer is used to perform feature prediction processing on the task group whose modality type is the single modal type, and the optimized cross-modal routing layer is used to perform feature prediction processing on the task group whose modality type is the cross-modal type.
2. The method according to claim 1, It is characterized in that The obtaining of the training task group set includes: Acquire N sample media data; N is a positive integer, and a modality type to which one sample media data among the N media data belongs is any one of a first modality type and a second modality type; and the first modality type is different from the second modality type; Based on the modality type to which each sample media data belongs in the N sample media data, a task group construction process is performed on the N sample media data to obtain the training task group set.
3. The method according to any one of claims 1 to 2, It is characterized in that The step of performing task group construction processing on the N sample media data based on the modality type to which each sample media data belongs in the N sample media data to obtain the training task group set includes: combining a first sample media data of the N sample media data with a first identifier for representing the first modality type to obtain a first training task group whose modality type is the first modality type; the first sample media data refers to any sample media data of the N sample media data whose modality type is the first modality type; combining second sample media data among the N sample media data with a second identifier for characterizing the second modality type to obtain a second training task group whose modality type is the second modality type; the second sample media data refers to any sample media data among the N sample media data whose modality type is the second modality type; Based on the media source channels respectively corresponding to the first sample media data and the second sample media data, cross-modal combination processing is performed on the first sample media data and the second sample media data to obtain a training task group whose modality type is the cross-modal type; The first training task group and the second training task group are both determined as training task groups whose modality type is the single modality type, and the set consisting of the training task group whose modality type is the cross-modality type and the training task group whose modality type is the single modality type is determined as the training task group set.
4. The method according to claim 3, It is characterized in that The number of the first sample media data and the number of the second sample media data are both one or more; the one or more first sample media data include the first sample media data M j , j is a positive integer; the modality type is the training task group of the cross-modality type, including the first sample media data M j The corresponding training task group; The step of performing cross-modal combination processing on the first sample media data and the second sample media data based on the media source channels respectively corresponding to the first sample media data and the second sample media data to obtain a training task group whose modality type is the cross-modal type includes: The first sample media data M j The corresponding media source channel is determined as the target media source channel; The second sample media data whose media source channel is the target media source channel among the one or more second sample media data is determined as the first sample media data M j The corresponding associated sample media data; The first sample media data, the associated sample media data, the first identifier, and the second identifier are combined to obtain the first sample media data M j The corresponding training task group.
5. The method according to any one of claims 1 to 4, It is characterized in that The training task group S in the modality representation model i Perform attention interaction processing to obtain the training task group S i The corresponding attention representation vector includes: In the modality representation model, the training task group S is extracted through a feature extraction network layer. i The sample media data contained therein is processed for feature extraction to obtain the training task group S i Corresponding media features; The training task group S is trained by using the multi-head self-attention network layer in the modality representation model. i The corresponding media features are processed by multi-head self-attention to obtain the training task group S i The corresponding attention representation vector.
6. The method according to claim 5, It is characterized in that The multi-head self-attention network layer in the modality representation model includes Q self-attention sub-network layers; Q is a positive integer; the Q self-attention sub-network layers include a self-attention sub-network layer V k , k is a positive integer; The multi-head self-attention network layer in the modality representation model is used to train the training task group S i The corresponding media features are processed by multi-head self-attention to obtain the training task group S i The corresponding attention representation vector includes: Get the self-attention sub-network layer V k The included attention parameter matrix, the self-attention sub-network layer V k The included attention parameter matrix is related to the training task group S i The corresponding media features are processed to obtain a linear transformation matrix corresponding to the attention parameter matrix; Through the self-attention sub-network layer V k The fully connected components in the linear transformation matrix perform feature integration processing to obtain the self-attention sub-network layer V k The corresponding attention representation sub-vector; When the attention representation sub-vectors corresponding to the Q self-attention sub-network layers are determined, the Q attention representation sub-vectors are fused to obtain the training task group S i The corresponding attention representation vector.
7. The method according to any one of claims 1 to 6, It is characterized in that The single modality type includes a first modality type and a second modality type; the single modality routing layer includes a first modality sub-routing layer corresponding to the first modality type and a second modality sub-routing layer corresponding to the second modality type; In the single-modal routing layer and the cross-modal routing layer included in the modality representation model, determine the training task group S i The target routing layer that matches the modality type, including: If the training task group S i If the modality type is the cross-modality type, the cross-modality routing layer included in the modality representation model is determined to be consistent with the training task group S i The target routing layer that matches the modality type; If the training task group S i If the modality type is the first modality type, the first modality sub-routing layer corresponding to the first modality type is determined to be the first modality sub-routing layer corresponding to the training task group S i The target routing layer that matches the modality type; If the training task group S i If the modality type is the second modality type, the second modality sub-routing layer corresponding to the second modality type is determined to be the same as the training task group S i The target routing layer that matches the modality type.
8. The method according to any one of claims 1 to 7, It is characterized in that The single modality type includes a first modality type and a second modality type; the single modality routing layer includes a first modality sub-routing layer corresponding to the first modality type and a second modality sub-routing layer corresponding to the second modality type; The optimizing the single-modal routing layer based on the predicted modal representation vector corresponding to the training task group whose modality type is the single-modal type includes: Determine the training task group whose modality type is the first modality type as the first modality training task group, and determine the training task group whose modality type is the second modality type as the second modality training task group; Obtaining a true modal representation vector corresponding to the first modal training task group and a true modal representation vector corresponding to the second modal training task group; Performing error calculation processing on the predicted modal representation vector corresponding to the first modal training task group and the true modal representation vector corresponding to the first modal training task group to obtain a first loss value corresponding to the first modal training task group; Performing error calculation processing on the predicted modal representation vector corresponding to the second modal training task group and the true modal representation vector corresponding to the second modal training task group to obtain a second loss value corresponding to the second modal training task group; The first modality sub-routing layer corresponding to the first modality type is optimized based on the first loss value, and the second modality sub-routing layer corresponding to the second modality type is optimized based on the second loss value.
9. The method according to claim 8, It is characterized in that The first modality type is a text modality type, and the first modality training task group is a text word sequence; the predicted modality representation vector corresponding to the first modality training task group is composed of the predicted representation features corresponding to each text word in the text word sequence; the real modality representation vector corresponding to the first modality training task group is composed of the real representation features corresponding to each text word in the text word sequence; The performing error calculation processing on the predicted modal representation vector corresponding to the first modal training task group and the real modal representation vector corresponding to the first modal training task group to obtain a first loss value corresponding to the first modal training task group includes: Acquire a masked text word from the text words included in the text word sequence; the masked text word refers to a text word that has been masked; In the predicted modal representation vector corresponding to the first modal training task group, the predicted representation feature corresponding to the masked text word is obtained, and in the real modal representation vector corresponding to the first modal training task group, the real representation feature corresponding to the masked text word is obtained; Determine a first feature similarity between a predicted representation feature corresponding to the masked text word and a real representation feature corresponding to the masked text word; The first feature similarity is determined as a first loss value corresponding to the first modality training task group.
10. The method according to any one of claims 1 to 9, It is characterized in that The training task group set includes at least two training task groups whose modality types are the cross-modality type; at least two training task groups whose modality types are the cross-modality type include training task group S j , j is a positive integer; the training task group S j The training task group S comprises third sample media data whose modality type is the first modality type and fourth sample media data whose modality type is the second modality type. j The corresponding predicted modal representation vector includes the predicted representation feature corresponding to the third sample media data and the predicted representation feature corresponding to the fourth sample media data; The optimizing the cross-modal routing layer based on the predicted modality representation vector corresponding to the training task group whose modality type is the cross-modal type includes: Determine a second feature similarity between the predicted characterization feature corresponding to the third sample media data and the predicted characterization feature corresponding to the fourth sample media data; Each training task group whose modality type is the cross-modality type is determined as a cross-modality training task group, and the training task group S is determined. j and the third feature similarity between the remaining cross-modal training task groups in the at least two cross-modal training task groups; the remaining cross-modal training task groups refer to the at least two cross-modal training task groups, except the training task group S j Cross-modal training task group other than The cross-modal routing layer is optimized based on the second feature similarity and the third feature similarity.
11. The method according to any one of claims 1 to 10, It is characterized in that The method further comprises: Acquire a target task group for describing the media data to be classified; the target task group includes at least two media data of different modal types; Performing attention interaction processing on the target task group in the modality representation model to obtain an attention representation vector corresponding to the target task group; In the optimized cross-modal routing layer, feature prediction processing is performed on the attention representation vector corresponding to the target task group to obtain a predicted modality representation vector corresponding to the target task group; The predicted modal representation vector corresponding to the target task group is subjected to category identification processing to obtain the media category to which the media data to be classified belongs; the media category to which the media data to be classified belongs is used to archive the media data to be classified.
12. A data processing device, It is characterized in that The device is applied to a computer device, and comprises: A set acquisition module is used to acquire a training task group set; the training task group set includes a training task group S i , i is a positive integer; the training task group S i The modality type is a single modality type or a cross-modality type; in the training task group S i When the modality type is the single modality type, the training task group S i Contains a sample media data; in the training task group S i When the modality type is the cross-modality type, the training task group S i Contains sample media data of at least two different modality types; An attention processing module is used to process the training task group S in the modality representation model i Perform attention interaction processing to obtain the training task group S i The attention interaction process is used to group the training task S i The different elements in the game interact fully with each other; A routing layer determination module is used to determine the single-modal routing layer and the cross-modal routing layer included in the modality representation model, and the training task group S i The target routing layer that matches the modality type; A prediction module is used to predict the training task group S based on the target routing layer. i The corresponding attention representation vector is processed for feature prediction to obtain the training task group S i The corresponding predicted modal representation vector; A first optimization module is used to optimize the single-modal routing layer based on the predicted modal representation vector corresponding to the training task group whose modality type is the single-modal type when determining the predicted modal representation vector corresponding to each training task group in the training task group set; The second optimization module is used to optimize the cross-modal routing layer based on the predicted modal representation vector corresponding to the training task group whose modality type is the cross-modal type; the optimized single-modal routing layer is used to perform feature prediction processing on the task group whose modality type is the single-modal type, and the optimized cross-modal routing layer is used to perform feature prediction processing on the task group whose modality type is the cross-modal type.
13. A computer device, It is characterized in that include: Processor, memory, and network interface; The processor is connected to the memory and the network interface, wherein the network interface is used to provide a network communication function, the memory is used to store a computer program, and the processor is used to call the computer program so that the computer device executes the method described in any one of claims 1-11.
14. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the method according to any one of claims 1 to 11.
15. A computer program product, It is characterized in that The computer program product comprises a computer program, which is stored in a computer-readable storage medium. The computer program is suitable for being read and executed by a processor, so that a computer device having the processor executes the method according to any one of claims 1 to 11.
Citation Information
Patent Citations
Pre-training model obtaining method and device
CN112668671A
Multi-modal data joint learning model training method and device
CN113240056A
Video classification method, device and equipment based on multi-modal representation, and storage medium
CN113762322A
Data processing method and device, computer equipment and storage medium
CN114419514A
Data processing method, device and equipment and readable storage medium
CN116975016A
