A model determination method and related apparatus

Abstract

The application discloses a model determination method and related device, obtaining N initial content vectors corresponding to N modalities one by one, and obtaining intra-modality restoration loss function and cross-modality restoration loss function through an initial merging model and a single-modality prediction model according to the N initial content vectors. The vector merging model trained according to the two restoration loss functions determines content vectors carrying more abundant information for each modality, and the correlation between content vectors of different modalities is better because the content vector of a modality is determined by considering relevant information of other modalities. The merged content vector obtained by merging the N content vectors is not only rich in information, but also has relevance from information of different modalities, so that the vector merging model can be directly used to realize content vector extraction when processing a classification task, and the training efficiency of the classification task is greatly improved.

Description

Technical Field

[0001] This application relates to the field of data processing, and in particular to a model determination method and related apparatus. Background Technology

[0002] With the application of artificial intelligence in the content field, various classification tasks for content can be achieved through network models, such as content popularity prediction, content tagging, and content type classification.

[0003] When processing classification tasks, it is necessary to extract the content vector as the basis for understanding subsequent tasks. Since the network model may emphasize different information carried in the content vector for different classification tasks, in related technologies, even for different classification tasks within the same content range, it is necessary to retrain the model for content vector extraction.

[0004] Training this type of model is time-consuming, and repeated training can significantly increase training costs when multiple classification tasks need to be handled, resulting in low training efficiency for classification tasks and difficulty in adapting to rapidly changing content classification needs. Summary of the Invention

[0005] To address the aforementioned technical issues, this application provides a model determination method that enables the direct use of a vector merging model to extract content vectors when processing classification tasks. This eliminates the need for repeated model training for content vector extraction, significantly improving the training efficiency of classification tasks and effectively meeting the rapidly evolving content classification needs.

[0006] The embodiments of this application disclose the following technical solutions:

[0007] On the one hand, embodiments of this application provide a model determination method, the method comprising:

[0008] Obtain N initial content vectors that correspond one-to-one with the N modalities of the content training samples, where N>1;

[0009] Based on the N initial content vectors, intra-modal reconstruction prediction is performed using an initial merging model and a single-modal prediction model to obtain an intra-modal reconstruction loss function. For the target modality among the N modalities, the corresponding intra-modal reconstruction loss function is determined based on the difference between the initial content vector and the intra-modal reconstruction vector of the target modality. The intra-modal reconstruction vector is obtained by reconstructing the initial content vector of the target modality with missing features using the initial merging model and the single-modal prediction model.

[0010] Based on the N initial content vectors, cross-modal reconstruction prediction is performed using the initial merging model and the single-modal prediction model to obtain a cross-modal reconstruction loss function. For the target modality, the corresponding cross-modal reconstruction loss function is determined based on the difference between the initial content vector and the cross-modal reconstruction vector of the target modality. The cross-modal reconstruction vector is obtained from the initial content vector of the modality to be synthesized from the N modalities using the initial merging model and the single-modal prediction model.

[0011] The initial merging model is trained using the intra-modal restoration loss function and the cross-modal restoration loss function to obtain a vector merging model. The vector merging model is used to obtain a merged content vector based on the content to be processed and the N content vectors corresponding one-to-one with the N modalities.

[0012] On the other hand, embodiments of this application provide a model determination apparatus, the apparatus comprising an acquisition unit, a prediction unit, and a training unit:

[0013] The acquisition unit is used to acquire N initial content vectors that correspond one-to-one with N modalities of the content training samples, where N>1;

[0014] The prediction unit is used to perform intra-modal reconstruction prediction based on the N initial content vectors using an initial merging model and a single-modal prediction model, to obtain an intra-modal reconstruction loss function. For the target modality among the N modalities, the corresponding intra-modal reconstruction loss function is determined based on the difference between the initial content vector and the intra-modal reconstruction vector of the target modality. The intra-modal reconstruction vector is obtained by reconstructing the initial content vector of the target modality with missing features using the initial merging model and the single-modal prediction model.

[0015] The prediction unit is further configured to perform cross-modal reconstruction prediction based on the N initial content vectors, using the initial merging model and the single-modal prediction model, to obtain a cross-modal reconstruction loss function; for the target modality, the corresponding cross-modal reconstruction loss function is determined based on the difference between the initial content vector of the target modality and the cross-modal reconstruction vector, wherein the cross-modal reconstruction vector is obtained from the initial content vector of the modality to be synthesized from the N modalities through the initial merging model and the single-modal prediction model;

[0016] The training unit is used to train the initial merging model using the intra-modal restoration loss function and the cross-modal restoration loss function to obtain a vector merging model. The vector merging model is used to obtain a merged content vector based on the content to be processed and the N content vectors corresponding one-to-one with the N modalities.

[0017] On the other hand, embodiments of this application provide a computer device, the computer device including a processor and a memory:

[0018] The memory is used to store computer programs and to transfer the computer programs to the processor;

[0019] The processor is configured to execute the methods described above according to the computer program.

[0020] On the other hand, embodiments of this application provide a computer-readable storage medium for storing a computer program for performing the methods described above.

[0021] On the other hand, embodiments of this application provide a computer program product including a computer program, which, when run on a computer device, causes the computer device to perform the methods described above.

[0022] As can be seen from the above technical solution, N initial content vectors are obtained, each corresponding to one of the N modalities, from the content training samples. Based on these N initial content vectors, the missing features are reconstructed and predicted using an initial merging model and a single-modal prediction model, resulting in an intra-modal reconstruction loss function. Furthermore, the initial content vector of the target modality is reconstructed and predicted based on the initial content vectors of other modalities, resulting in a cross-modal reconstruction loss function. This intra-modal reconstruction loss function guides the initial merging model to learn how to extract richer information within a modality. The cross-modal reconstruction loss function guides the initial merging model to learn how to reference relevant information from other modalities when determining the content vector corresponding to a modality, thus avoiding overfitting during training of the initial merging model for determining the content vector of a single modality. The vector merging model trained based on these two reduction loss functions produces content vectors with richer information for each modality. Furthermore, since determining the content vector for one modality considers relevant information from other modalities, the correlation between content vectors from different modalities is better. The merged content vector obtained by merging N content vectors is not only rich in information but also exhibits correlation between information from different modalities. High-quality merged content vectors are applicable to various classification tasks, allowing the vector merging model to be used directly for content vector extraction without repeated model training for content vector extraction. This significantly improves the training efficiency of classification tasks and effectively meets the rapidly evolving content classification needs. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A schematic diagram illustrating a model-determined scenario provided in an embodiment of this application;

[0025] Figure 2 A flowchart illustrating a model determination method provided in this application embodiment;

[0026] Figure 3 A schematic diagram illustrating an intramodal reconstruction prediction provided in an embodiment of this application;

[0027] Figure 4 A schematic diagram illustrating a cross-modal reconstruction prediction provided in an embodiment of this application;

[0028] Figure 5 This application provides a training schematic diagram illustrating a content merging model determined through self-supervised training and supervised training, as part of an embodiment of the present application.

[0029] Figure 6 A schematic diagram illustrating the training of a new classification task assisted by a content merging model, provided as an embodiment of this application;

[0030] Figure 7 This application provides a schematic diagram of content vector extraction for different modalities in an embodiment of the present application.

[0031] Figure 8 A structural diagram of a model determination device provided in an embodiment of this application;

[0032] Figure 9 A structural diagram of a terminal device provided in an embodiment of this application;

[0033] Figure 10 This is a structural diagram of a server provided in an embodiment of this application. Detailed Implementation

[0034] The embodiments of this application will now be described with reference to the accompanying drawings.

[0035] With the application of artificial intelligence in the content field, various classification tasks for content can be achieved through network models, such as content popularity prediction, content tagging, and content type classification.

[0036] For example, when the content is video, video deduplication applications often require multiple video understanding-based classification tasks to be processed together due to the diverse sources of business. These tasks include video channel classification (such as sports, news, film and television, variety shows, lifestyle, and arts) to determine which channel's predefined parameters to use for deduplication tasks; video plot recognition to determine the main content of the video to support customized plot services (such as specifying certain plots to be returned to the business party for further judgment on whether it constitutes infringement); predicting the popularity of newly uploaded videos based on their content to support the popularity score evaluation when recommending the video; and adding multiple tags to video elements (such as football field, stands, ancient costumes, etc.) so that specific identification can be performed based on different tag elements (such as similarity recognition).

[0037] In processing classification tasks, content vectors need to be extracted as the basis for understanding subsequent tasks. Since the emphasis of network models on the information carried in the content vectors may differ for different classification tasks, related technologies require retraining of the content vector extraction model even for different classification tasks within the same content range. This model training is time-consuming, and repeated training significantly increases training costs when handling multiple classification tasks, resulting in low model training efficiency for classification tasks and difficulty in adapting to rapidly changing content classification needs.

[0038] For example, in the aforementioned video deduplication application, multiple classification tasks need to be handled, such as video channel classification, video plot recognition, video popularity prediction, and video multi-labeling. These classification tasks all require extracting feature vectors from the video for subsequent task understanding. If relevant technologies are used, each classification task needs to train a feature vector extraction model suitable for its own classification needs. That is, feature vector extraction needs to be repeatedly trained multiple times before the functions required by the video deduplication application can be realized. This not only requires a lot of manpower and time to build video training samples for different classification tasks, but also, due to the overlap of various classification tasks, there will be a large proportion of repeated training of video training samples, which will also seriously reduce the overall training efficiency and delay the launch time of the video deduplication application.

[0039] Therefore, this application provides a model determination method that enables the direct use of a vector merging model to extract content vectors when processing classification tasks, eliminating the need for repeated model training for content vector extraction. This greatly improves the training efficiency of classification tasks and effectively meets the rapidly evolving content classification needs.

[0040] The model determination method provided in this application can be implemented using computer equipment, which can be a terminal device or a server. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. Terminal devices include, but are not limited to, mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, and aircraft. Terminal devices and servers can be directly or indirectly connected via wired or wireless communication, and this application does not impose any limitations on this connection.

[0041] The embodiments of this application can be applied to the field of artificial intelligence (AI). AI is the theory, methods, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0042] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, as well as machine learning / deep learning, autonomous driving, and intelligent transportation.

[0043] In this embodiment of the application, when the content to be processed is video, and the N modalities include video frames, audio frames and text, it will involve obtaining and determining the content vectors corresponding to these modalities through computer vision technology, speech processing technology and natural language processing technology.

[0044] This application also involves machine learning. Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, and many other disciplines. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0045] In this embodiment of the application, the initial merging model, classification model, target classification model, etc. are trained based on machine learning to obtain the model required to implement the embodiment of the application, such as the vector merging model.

[0046] Figure 1 This is a schematic diagram of a model-determined scenario provided in an embodiment of this application. In this embodiment, server 100 is used as the aforementioned computer device, video training samples are used as content training samples, and N=3 is used as an example for illustration.

[0047] Server 100 obtains three initial content vectors corresponding to the three modalities in the video training samples: video frame vector, audio frame vector, and text vector. Based on these three initial content vectors, it performs reconstruction prediction on the missing features using an initial merging model and a single-modal prediction model, thus obtaining the intra-modal reconstruction loss function.

[0048] The initial content vector of the target modality is then reconstructed and predicted based on the initial content vectors of other modalities, resulting in a cross-modal reconstruction loss function.

[0049] Among them, three single-mode prediction models are set for the three modes, namely single-mode prediction models 1-3, which correspond one-to-one with these three modes.

[0050] The intramodal restoration loss function can guide the initial merging model to learn to extract richer information within a modality. The cross-modal restoration loss function can guide the initial merging model to learn to refer to relevant information from other modalities when determining the content vector corresponding to a modality, and avoid overfitting when the initial merging model is trained to determine the content vector of a single modality.

[0051] The vector merging model trained by server 100 based on these two reduction loss functions has richer information in the content vector determined by the vector merging model for each modality. Moreover, since the relevant information of other modalities is taken into account when determining the content vector of a modality, the correlation between the content vectors of different modalities is better. The merged content vector obtained by merging N content vectors is not only rich in information, but also has correlation between information from different modalities.

[0052] High-quality merged content vectors are applicable to various classification tasks, enabling the direct use of vector merging models to extract content vectors when processing classification tasks. This eliminates the need for repeated model training for content vector extraction, significantly improving the training efficiency of classification tasks and effectively meeting the rapidly evolving content classification needs.

[0053] Figure 2This application provides a flowchart of a model determination method, which can be executed by a computer device. In this embodiment, the computer device is a server as an example. The method includes:

[0054] S201: Obtain the N initial content vectors that correspond one-to-one with the N modalities of the content training samples.

[0055] Where N>1, meaning that in this embodiment, for a single content training sample, it is necessary to obtain the initial content vectors of that content training sample across multiple modalities. This is because the vector merging model in this application is used to provide merged vector features of the content to be processed for subsequent different classification tasks. To ensure that these merged vector features are applicable to different classification tasks, they need to carry sufficient and rich information. The information carried by content vectors in different modalities will have different emphases. By integrating content vectors from N modalities, the information richness of the merged vector features provided by the vector merging model can be significantly improved.

[0056] Modality can be understood as a data dimension. Different modalities identify different data dimensions, which means that for the same content, the information carried by content vectors from different modalities will have different emphases, the information itself will be different, and there may even be very little overlap.

[0057] This application does not limit the type of content; for example, it can be video, comics, text, audio, etc. Taking video as an example, the N modalities of the video can include at least three modalities: video frames, audio frames (if audio is present), and text (if related text is present). The initial content vector extracted from the video frames is mainly based on the information carried by the video content; the initial content vector extracted from the audio frames is mainly based on the information carried by the audio content; and the initial content vector extracted from the video-related text is mainly based on the information carried by the text content.

[0058] In this application, the specific value of N is not limited, as long as it is greater than 1. The specific value of N can be determined based on the different types of content and the number of modalities that the content can provide.

[0059] S202: Based on the N initial content vectors, intra-modal reconstruction prediction is performed through the initial merging model and the single-modal prediction model to obtain the intra-modal reconstruction loss function.

[0060] Intramodal reconstruction prediction refers to the process of reconstructing and predicting the initial content vector of a modality by using an initial merging model and a single-modal prediction model to recover the missing features of that modality.

[0061] Taking any one of the N modalities as the target modality as an example, the corresponding intramodal reconstruction loss function is determined based on the difference between the initial content vector of the target modality and the intramodal reconstruction vector. The intramodal reconstruction vector is obtained by reconstructing the initial content vector of the target modality with missing features through the initial merging model and the single-modal prediction model.

[0062] In other words, the server discards features from the initial content vector of the target modality to obtain an initial missing vector. Then, using an initial merging model and a single-modal prediction model corresponding to the target modality, it predicts the initial content vector of the target modality based on the initial missing vector, essentially restoring the missing features. The initial content vector itself is used to measure the accuracy of the restored prediction. That is, the server can determine the intra-modal restoration loss function by the difference between the initial content vector and the vector obtained from the intra-modal restoration prediction. Therefore, intra-modal restoration prediction is a self-supervised model training method that does not require additional labeling and does not increase the cost of model training.

[0063] For example Figure 3 In the scenario shown, the initial content vector corresponding to the target modality is initial content vector 1. The server performs feature discarding processing based on this initial content vector 1 to obtain an initial missing vector, where the white blocks represent the discarded feature dimensions in initial content vector 1. Through the initial merging model and the single-modal prediction model corresponding to the target modality, the intra-modal reconstruction vector can be determined. Then, based on initial content vector 1 as supervision information, the server determines the intra-modal reconstruction loss function by comparing the difference between the intra-modal reconstruction vector and this supervision information.

[0064] When training the initial merging model, this intramodal reconstruction loss function guides the model to learn how to extract richer intramodal information. This is because the less effective information in the initial missing vector, the less information the model can rely on to reconstruct the initial content vector, resulting in a greater difference between the reconstructed initial content vector and the original initial content vector. Conversely, the more effective information in the initial missing vector, the more information the model can rely on to reconstruct the initial content vector, resulting in a smaller difference between the reconstructed initial content vector and the original initial content vector.

[0065] For each of the N modalities, the server can implement the S202 scheme to determine the intramodal reduction loss function for different modalities. In this way, during subsequent training, the initial merged model can learn knowledge of content vectors that can be determined for different modalities with richer information.

[0066] It should be noted that there can be N single-mode prediction models, each corresponding one-to-one with one of the N modes. That is, different modes have different single-mode prediction models, which can effectively improve the accuracy of intra-mode and cross-mode reconstruction prediction.

[0067] S203: Based on the N initial content vectors, cross-modal reconstruction prediction is performed using the initial merging model and the single-modal prediction model to obtain the cross-modal reconstruction loss function.

[0068] Cross-modal reconstruction prediction refers to the process of reconstructing and predicting the initial content vector of the target modality based on the initial content vector of other modalities that are not the target modality among N modalities, using an initial merging model and a single-modal prediction model.

[0069] Taking any one of the N modalities as the target modality as an example, the corresponding cross-modal reconstruction loss function is determined based on the difference between the initial content vector of the target modality and the cross-modal reconstruction vector. The cross-modal reconstruction vector is obtained from the initial content vector of the modality to be synthesized from the N modalities through the initial merging model and the single-modal prediction model.

[0070] In other words, the server predicts the initial content vector of the target modality based on the initial content vectors of other modalities using an initial merging model and a single-modal prediction model corresponding to the target modality. This is equivalent to performing cross-modal feature restoration. The initial content vector of the target modality itself is used to measure the accuracy of the restoration prediction. That is, the server can determine the cross-modal restoration loss function by the difference between the initial content vector and the vector obtained from the cross-modal restoration prediction. Therefore, cross-modal restoration prediction is a self-supervised model training method that does not require additional labeling and does not increase the cost of model training.

[0071] For example Figure 4 In the scenario shown, the content is exemplified as video, and the N modalities include video frames, audio frames, and text. The initial content vector 1 is the content vector of the video frame, the initial content vector 2 is the content vector of the audio frame, and the initial content vector 3 is the content vector of the text.

[0072] The server performs a reconstruction prediction based on initial content vector 1 using initial content vector 2 and initial content vector 3. Through the initial merging model and the single-modal prediction model corresponding to the target modality, the cross-modal reconstruction vector can be determined. Then, based on initial content vector 1 as supervision information, the server determines the cross-modal reconstruction loss function by comparing the cross-modal reconstruction vector with this supervision information.

[0073] When training the initial merging model, this cross-modal reduction loss function can guide the initial merging model to learn that when determining the content vector corresponding to a modality, it can refer to the relevant information of other modalities, and avoid the overfitting problem that occurs when the initial merging model is trained to determine the content vector of a single modality.

[0074] The reason is that the lower the correlation between the initial content vectors of other modalities and the initial content vector of the target modality, the less information can be relied upon to reconstruct the initial content vector of the target modality using the above model, and the greater the difference between the reconstructed initial content vector and the original initial content vector of the target modality. Conversely, the higher the correlation between the initial content vectors of other modalities and the initial content vector of the target modality, the more information can be relied upon to reconstruct the initial content vector of the target modality using the above model, and the smaller the difference between the reconstructed initial content vector and the original initial content vector of the target modality.

[0075] The benefit of enhancing the correlation between initial content vectors of different modalities is that when the N content vectors corresponding to N modalities are merged one-to-one through the vector merging model to obtain the merged content vector, the correlation between the content vectors of different modalities is better. That is, the isolation of these N content vectors in the merged content vector is lower. Through these correlations, more in-depth intermodal effective information can be provided for subsequent classification tasks. High-quality merged content vectors can also improve the applicability to different classification tasks.

[0076] For each of the N modalities, the server can implement the S203 scheme to determine the cross-modal restoration loss function for different modalities. In this way, during subsequent training, the initial merging model can learn the knowledge to enhance the correlation with other modalities when determining the content vector for different modalities.

[0077] S204: The initial merging model is trained using the intra-modal restoration loss function and the cross-modal restoration loss function to obtain the vector merging model.

[0078] As mentioned earlier, the intramodal restoration loss function and the cross-modal restoration loss function can respectively guide the initial merging model to learn the knowledge of extracting richer information within a modality, and guide the initial merging model to learn the knowledge of referencing relevant information from other modalities when determining the content vector corresponding to a modality, and avoid the problem of overfitting when the initial merging model is trained for determining the content vector of a single modality.

[0079] Therefore, a high-quality vector merging model can be trained using the intra-modal restoration loss function and the cross-modal restoration loss function. This vector merging model is used to obtain a merged content vector based on the content to be processed and the N content vectors that correspond one-to-one with the N modalities. This allows the vector merging model to integrate the information of the N content vectors into the merged content vector based on the knowledge learned by the model training in this application.

[0080] There are multiple ways to obtain a merged content vector based on N content vectors, and the specific method is related to the model structure of the vector merging model. For example, this vector merging model is used to merge the content to be processed with the N content vectors that correspond one-to-one with the N modalities to obtain a merged content vector.

[0081] Through the above training, the merged content vectors carry richer information about each modality, and the correlation between the modalities is better. It can adapt to the feature requirements and emphases of different classification tasks. It is equivalent to providing a unified vector extraction architecture for multiple classification tasks through the vector merging model. When a new classification task needs to extract content vectors, the vector merging model can be used directly without retraining.

[0082] Therefore, it is evident that the vector merging model trained based on the intra-modal and cross-modal restoration loss functions provides richer information in the content vectors determined for each modality. Furthermore, since the determination of a modality's content vector takes into account relevant information from other modalities, the correlation between content vectors from different modalities is better. The merged content vector obtained by merging N content vectors is not only rich in information but also exhibits correlation between information from different modalities. High-quality merged content vectors can be applied to various classification tasks, enabling the direct use of the vector merging model to extract content vectors when processing classification tasks, eliminating the need for repeated model training for content vector extraction. This significantly improves the training efficiency of classification tasks and effectively meets the rapidly evolving content classification needs.

[0083] The model structure of the initial merging model or the vector merging model will be illustrated in Table 1 below.

[0084]

[0085] Table 1

[0086] Taking the vector merging model as an example, Table 1 illustrates the various layers in the model. The first column in Table 1 is the layer name, the second column is the output size of the layer, and the third column is the type of the layer (Module).

[0087] Among them, Map1-Map3, as well as Concat (concatenation) and FC1 (fully connected) are fully connected layers, which are feature dimension transformation layers. They can output a vector with a uniform feature dimension regardless of the feature dimension of the input vector, such as the 1x768 dimension output vector in Table 1.

[0088] Between Concat and FC1, there are two more attention layers, which are self-attention layers. It should be noted that the two attention layers are just an example, and other numbers of attention layers can also be set, which is not limited in this application.

[0089] Table 1 shows the model structure for N=3, where the input vectors of Map1-Map3 come from different modalities. For example, when the content is video, the input vector of Map1 comes from video frames, the input vector of Map2 comes from audio frames, and the input vector of Map3 comes from text. In other words, Map1-Map3 are feature dimension transformation layers corresponding one-to-one with the three modalities.

[0090] The output vectors of Map1-Map3 (each with a feature dimension of 1x768) are used in parallel as the input vectors of Concat for concatenation. The output vector of Concat passes through two attention layers and is then input into FC1 for the final output, resulting in a merged content vector with a feature dimension of 256.

[0091] As can be seen in the examples in Table 1, the feature dimension of the input vector of the vector merging model is smaller than the feature dimension of the output vector (i.e., the merged content vector). Compressing high-dimensional data into a low-dimensional space using the vector merging model can effectively reduce the interference of redundant information and help to explore the correlation between information from different modalities.

[0092] Therefore, in one possible implementation, the initial merging model includes an attention layer, which is used to map the input feature vector into an attention space, the dimension of which is lower than the dimension of the feature space of the input feature vector.

[0093] The input vector is encoded using the attention layer of the vector merging model (query vector, key vector, and value vector) by qkv encoding, which maps the high-dimensional input vector to the low-dimensional qkv space (i.e., attention space). Then, feature weights are obtained based on the attention relationship between q and k. These feature weights are added to v to obtain a weighted value, which produces the output vector.

[0094] For example, the attention layer can obtain three vectors q, k, and v based on the input vector; calculate a score for each vector: score = q ⋅ k; normalize the score to stabilize the gradient, i.e., divide it by dk; apply the softmax activation function to the score; multiply the softmax dot product by the value v to obtain the weighted score v for each input vector.

[0095] Each attention layer needs to learn three matrices that map the input vector to q, k, and v. Using a self-attention network allows for the extraction of correlations within the vectors within the attention space defined by qkv. Due to the highly aggregated nature of the qkv space (its dimension is generally smaller than the original features), redundant information in the vectors can be compressed. Furthermore, feature mapping is performed in this highly aggregated, low-dimensional space, thus enabling better extraction of correlations within the vectors.

[0096] Next, taking the target mode among N modes as an example, we will illustrate a possible intra-modal reconstruction prediction method for S202.

[0097] In one possible implementation, S202: Based on the N initial content vectors, intra-modal reconstruction prediction is performed using an initial merging model and a single-modal prediction model to obtain an intra-modal reconstruction loss function, including:

[0098] S2021: Based on the initial content vector of the target modality, some features are discarded to obtain the initial missing vector;

[0099] S2022: Using the initial merging model, determine the merged sample missing vector corresponding to the initial missing vector;

[0100] S2023: Based on the merged sample missing vector, the intra-modal reconstruction vector of the target modality is predicted using the single-modal prediction model corresponding to the target modality;

[0101] S2024: Determine the intramodal restoration loss function based on the difference between the initial content vector and the intramodal restoration vector.

[0102] This application does not limit how to partially discard features from the initial content vector, i.e., it does not limit which dimensions of features to discard, or how many dimensions of features to discard, as long as the initial missing vector still contains some features of the original initial content vector. For example, it can be like this: Figure 3 As shown.

[0103] For example, the content is video, and the N modalities include video frames, audio frames and text. Under the model structure shown in Table 1, the content vector obtained from the video frame is used to obtain the initial content vector with 768-dimensional features through Map1. This initial content vector is randomly discarded (dropout=0.2, discarding 20% ​​of the feature bits, for example, setting the feature bits to 0). There are no input vectors for Map2 and Map3.

[0104] After the other structures in Table 1, a 1x256-dimensional content merging vector is generated based on FC1. Then, a 768-dimensional feature prediction is performed on the content merging vector using a single-modal prediction model corresponding to this video frame modality. The model structure of this single-modal prediction model is shown in Table 2.

[0105]

[0106] Table 2

[0107] The single-modal prediction model includes a feature prediction layer (Feat_predict), whose output vector has the same dimension as the initial content vector, in order to facilitate the determination of differences.

[0108] This application does not limit the method by which the server calculates the intramodal restoration loss function. It can be the mean square error (MSE) loss (also known as L2 loss) or other regression losses, such as the mean absolute error (MAE) loss (also known as L1 loss).

[0109] It should be noted that although Table 2 shows the model structure of the single-modal prediction model for the video frame modality, the model structure of the single-modal prediction model for other modalities can also be shown in Table 2, so they will not be listed one by one.

[0110] By using the intramodal reduction loss function as a guide during the training process, the overall feature information of each modality can be well preserved in the content vector of each modality before merging to obtain the merged content vector, which helps to avoid overfitting of the model in classification tasks.

[0111] When performing intramodal reconstruction prediction, the initial content vector is used as the basis. That is, the initial missing vector is obtained based on the initial content vector, and the intramodal reconstruction vector needs to be determined by the intramodal reconstruction loss function through the initial content vector.

[0112] In this application, the initial content vector of the target modality is the content vector obtained from the target modality based on the content training samples. It can be the vector before Map1-Map3 in the input table 1, or it can be the vector output after feature dimension transformation through Map1-Map3.

[0113] In one possible implementation, the N initial content vectors are feature vectors extracted from the corresponding modal of the content training samples, or the N initial content vectors are feature vectors obtained by transforming the feature dimensions of the same modality in the initial merging model based on the feature vectors.

[0114] The server can determine which of the aforementioned feature vectors to use as the initial content vector based on specific application scenarios or computational requirements. By providing multiple possible ways to select the initial content vector, the applicability of the embodiments of this application can be improved, and the execution flexibility can be increased.

[0115] Next, taking the target mode among N modes as an example, we will illustrate a possible cross-modal reconstruction prediction method for S203.

[0116] In one possible implementation, S203: Based on the N initial content vectors, cross-modal reconstruction prediction is performed using the initial merging model and the single-modal prediction model to obtain a cross-modal reconstruction loss function, including:

[0117] S2031: Determine the mode to be synthesized from the N modes;

[0118] The mode to be synthesized mentioned in this step can be any other mode described in the foregoing embodiments.

[0119] S2032: Determine the merged sample content vector corresponding to the initial content vector of the modality to be synthesized through the initial merging model;

[0120] S2033: Based on the merged sample content vector, the cross-modal reconstruction vector of the target modality is predicted using the single-modal prediction model corresponding to the target modality;

[0121] S2034: Determine the cross-modal reconstruction loss function based on the difference between the initial content vector of the target modality and the cross-modal reconstruction vector.

[0122] This application does not limit the number of modes to be synthesized; the number can be any integer value greater than or equal to 1 and less than or equal to N-1.

[0123] For example, the content is a video, and the N modalities include video frames, audio frames, and text, wherein the target modality is text, and the modalities to be synthesized are video frames and audio frames.

[0124] Under the model structure shown in Table 1, the content vectors of video frames are input into the initial merging model through Map1, and the content vectors of audio frames are input into the initial merging model through Map2. Map3 has no input vector. The initial merging model obtains the corresponding merged content vector based on these two input vectors. Then, the cross-modal reconstruction vector corresponding to this merged content vector is obtained through the single-modal prediction model corresponding to the text modality.

[0125] Input the text content vector into Map3 to obtain the initial content vector corresponding to the text. Use this initial content vector as supervision information to determine the difference between the cross-modal reconstruction vector and this supervision information, and obtain the cross-modal reconstruction loss function.

[0126] Cross-modal reconstruction prediction can guide the model to learn that when determining the content vector of one modality, it will consider the relevant information of other modalities. Therefore, before merging to obtain the merged content vector, the correlation between the content vectors of each modality is better. This correlation can effectively improve the quality of the merged content vector and increase the depth of information, thereby expanding the applicability of the merged content vector.

[0127] This application does not limit the method by which the server calculates the cross-modal recovery loss function; it can be MSE loss or other regression losses, such as L1 loss.

[0128] For the intra-modal and cross-modal reconstruction loss functions that need to be determined in self-supervised training, if the MSE loss is used for determination, the root mean square error (MSE) can be specifically expressed as follows:

[0129]

[0130] Where yi is the output vector of the initial merging model under self-supervised training, and yi with p is the initial content vector used as supervision information. The loss function obtained from this self-supervised training is used to guide the aforementioned output vector to be as close as possible to the supervision information.

[0131] In cross-modal reconstruction prediction, this application does not limit the relationship between the target modality and the modality to be synthesized. However, in some scenarios, the initial content vector with finer feature granularity can carry more information. Using the initial feature vector with fine feature granularity to predict the initial feature vector with coarse feature granularity across modalities can achieve better prediction results.

[0132] Therefore, in one possible implementation, S2031: determining the mode to be synthesized from the N modes includes:

[0133] S20311: Determine the feature granularity of the N initial content vectors;

[0134] S20312: Determine the modality to be synthesized from modalities with a feature granularity finer than that of the initial content vector of the target modality.

[0135] The feature granularity of an initial content vector is used to characterize the degree of refinement or richness of the information carried by the initial content vector. For example, the higher the degree of refinement of the information in an initial content vector, the finer its corresponding feature granularity.

[0136] By comparing the feature granularity of N initial content vectors, the server can find modalities with finer feature granularity than the initial content vector of the target modality from among the N modalities. The server then identifies all or part of these modalities as the modal to be synthesized.

[0137] Since the initial content vector of the modality to be synthesized carries rich information, predicting the initial content vector of the coarse feature granularity using the initial content vector of the fine feature granularity allows the initial merging model to learn the knowledge that the initial content vector of the fine feature granularity retains information related to the initial content vector of the coarse feature granularity. This improves the information richness and coverage of the initial content vector of the fine feature granularity and effectively avoids the overfitting problem of the modality corresponding to the initial content vector of the fine feature granularity during model training.

[0138] Regarding the aforementioned intra-modal reconstruction prediction and cross-modal reconstruction prediction, these can be performed in parallel or sequentially in this application, without limitation. The obtained intra-modal reconstruction loss function and cross-modal reconstruction loss function can be combined to construct a unified prediction loss function for model training. However, in some scenarios, to improve training accuracy, whether to use the cross-modal reconstruction loss function in model training needs to consider the situation with the intra-modal reconstruction loss function.

[0139] In one possible implementation, S204: Training the initial merging model using the intra-modal restoration loss function and the cross-modal restoration loss function includes:

[0140] S2041: Determine the degree of difference identified by the intramodal reduction loss function;

[0141] S2042: In response to the difference level being lower than the difference condition, the initial merging model is trained according to the intra-modal restoration loss function and the cross-modal restoration loss function;

[0142] S2043: In response to the difference level being higher than or equal to the difference condition, discard the cross-modal restoration loss function and train the initial merged model based on the intra-modal restoration loss function.

[0143] In other words, before training the initial merged model using the intra-modal and cross-modal reconstruction loss functions, the decision to use the cross-modal reconstruction loss function depends on the degree of difference reflected by the intra-modal reconstruction loss function. This degree of difference reflects the accuracy of the initial merged model in determining the intra-modal content vectors. When the accuracy of the intra-modal content vectors is insufficient, using poor-quality content vectors for cross-modal reconstruction prediction will inevitably fail to achieve the desired prediction results. Consequently, the resulting cross-modal reconstruction loss function may negatively influence model training or even reduce model accuracy.

[0144] For example, when training reaches the tenth batch, the degree of difference reflected by the intramodal reconstruction loss function can be recorded as l0. In subsequent batch training, if the degree of difference reflected by the intramodal reconstruction loss function is lower than l0*0.4, the cross-modal reconstruction loss function of the same batch can be introduced into the model training. If the loss of a certain batch is higher than l0*0.4, the cross-modal reconstruction loss function of that batch is stopped from being introduced into the model training, and only the intramodal reconstruction loss function is used to train the initial merged model.

[0145] Therefore, by measuring the degree of difference indicated by the intramodal reconstruction loss function, it is possible to determine whether to introduce a cross-modal reconstruction loss function for model training. This can effectively ensure that the prediction loss function accurately guides model training and improves the model training accuracy.

[0146] In this embodiment of the application, the trained vector merging model mainly provides a unified vector extraction framework for different classification tasks, so as to reduce the difficulty of model training for subsequent classification tasks and improve the online speed of classification tasks.

[0147] Since classification tasks differ in terms of data and labeling, in order to better establish a connection between the vector merging model and the classification task, in this embodiment of the application, a downstream classification task can be introduced during the training of the vector merging model, so that the model can obtain sufficient mining and learning, thereby improving the adaptability of the vector merging model to the classification task.

[0148] Based on the foregoing embodiments, in one possible implementation, the method further includes:

[0149] S11: Based on the N initial content vectors, obtain the merged sample vector of the content training samples through the initial merging model;

[0150] S12: Determine the classification result corresponding to the merged sample vector using the classification model corresponding to the classification task;

[0151] S13: Determine the classification loss function based on the difference between the classification results and the classification labels of the content training samples;

[0152] S204: The initial merging model is trained using the intra-modal restoration loss function and the cross-modal restoration loss function to obtain a vector merging model, including:

[0153] The initial merging model is trained using the classification loss function, the intra-modal restoration loss function, and the cross-modal restoration loss function to obtain the vector merging model.

[0154] The classification model corresponds to the classification task. For example, when the classification task is a multi-label labeling task, the classification model can determine the corresponding label based on the input merged sample vector. This classification model can be a pre-trained model or a model that has not yet been fully trained, such as a model that needs to be trained together with the initial merged model. This application does not impose any restrictions on this.

[0155] After introducing the classification model, the classification loss function needs to be obtained through supervised training. Therefore, when performing S11-S13, the content training samples need to be labeled with classification labels related to the classification task.

[0156] In this supervised training, the merged sample vector output by the initial merging model is obtained based on all N initial content vectors, ensuring that the merged sample vector carries relevant information from N modalities. This information is then used as input to the classification model to obtain the classification result. The server determines the classification loss function based on the difference between the classification label and the classification result.

[0157] Depending on the classification task, the model structure of the corresponding classification model may differ. For example, for the channel classification task, the model structure of the classification model can be shown in Table 3:

[0158]

[0159] Table 3

[0160] The input to this classification model is the output of Table 1. Fc_class is the channel classification module. For example, channels include: movies and TV series, lifestyle, news, talent shows, variety shows, sports, etc.

[0161] For example, for the classification task of multi-label classification, the model structure of its classification model can be shown in Table 4:

[0162]

[0163] Table 4

[0164] The input to this classification model is the output of Table 1. For example, it can make predictions for more than 20,000 tags for content, such as multiple tags including: face, adult, embroidered shoes, dress, wearing a hat, wearing sunglasses, inside a car, funny, etc.

[0165] This large-scale label classification is important in training the unified framework of the vector merging model. It serves as supervisory information to ensure the comprehensiveness of the unified feature vector, that is, to ensure that the unified merged content vector can support large-scale label recognition downstream, so that the merged content vector can carry rich information to adapt to other classification tasks.

[0166] For example, for the classification task of predicting video popularity, the model structure of the classification model can be shown in Table 5:

[0167]

[0168] Table 5

[0169] The input to this classification model is the output of Table 1, and it can predict the popularity score (0 to 1) for each content.

[0170] By introducing a classification loss function into the training of the initial merging model, the initial merging model can be sufficiently mined and learned under the guidance of the classification loss function. This allows the initial merging model to learn the correlation between the merged sample vector and the downstream classification task, so that the determined merged sample vector can be applied to the classification task and carry richer information that the classification task emphasizes.

[0171] When training the initial merging model using classification tasks, more than one classification task can be introduced to assist in the training.

[0172] In one possible implementation, the classification task includes M types, where M>1.

[0173] S12: Determine the classification result corresponding to the merged sample vector using the classification model corresponding to the classification task, including:

[0174] By using the classification models corresponding to the M classification tasks respectively, the classification results of the merged sample vector corresponding to the M classification tasks are determined.

[0175] S13: Determine the classification loss function based on the difference between the classification results and the classification labels of the content training samples, including:

[0176] S131: For each classification task, based on the classification result of the classification task and the difference between the sub-classification labels of the content training samples corresponding to the classification task, determine the sub-classification loss function corresponding to the classification task;

[0177] S132: Determine the classification loss function based on the sub-classification loss function for each classification task.

[0178] For each of the M classification tasks, the method for determining the corresponding loss function is the same as in the previous embodiment, and will not be repeated here.

[0179] Since M classification tasks are introduced, the content training samples need to be configured for different classification tasks. The same content training sample can be used for prediction of different classification tasks, or it can be used for prediction of only one classification task. The specific number of classification tasks it is used for can be determined based on the classification labels of the content training sample, or it can be determined based on the training requirements.

[0180] For M classification tasks, we can identify M corresponding training sample sets. Based on the training possibilities mentioned above, a content training sample can belong to one training sample set or multiple training sample sets. For a training sample set corresponding to a classification task, all content training samples in that set are labeled with classification labels related to that classification task.

[0181] By introducing M classification tasks and determining the classification loss function based on the M sub-classification loss functions, the initial merging model can learn the correlation between the determined merged sample vector and the downstream M classification tasks, so that the determined merged sample vector can be applied to the M classification tasks and carry richer information that the M classification tasks emphasize.

[0182] Since the sub-classification loss functions for the M classification tasks may not be in the same scale space, this application provides a possible implementation method to better determine the classification loss function based on the M sub-classification loss functions:

[0183] S132: Determine the classification loss function based on the sub-classification loss function for each classification task, including:

[0184] Based on the sub-classification loss function of each classification task, determine the sub-loss weights corresponding to the sub-classification loss function respectively. The sub-loss weights are used to normalize the sub-classification loss function of each classification task to the same scale space.

[0185] The classification loss function is determined based on the sub-classification loss function for each classification task and the corresponding sub-loss weights.

[0186] By constructing sub-loss weights to normalize each sub-loss weight to the same scale space, the influence of each sub-classification loss function is relatively similar when determining the classification loss function with the normalized sub-classification loss function. This prevents any one sub-classification loss function from having a decisive impact on the classification loss function, thus effectively balancing the roles of the M classification tasks in the classification loss function and avoiding overfitting to a particular classification task during training.

[0187] This application does not limit the method for determining the sub-loss weights; for example, they can be determined using the following formula:

[0188] Wei3 = 1-loss3 / (loss4+loss5)

[0189] Wei4 = 1-loss4 / (loss3+loss5)

[0190] Wei5 = 1-loss5 / (loss4+loss3)

[0191] Loss2 = wei3*loss3+wei4*loss4+wei5*loss5

[0192] The above formula is an example using M=3 classification tasks. loss3 is the sub-classification loss function for the first classification task, loss4 is the sub-classification loss function for the second classification task, and loss5 is the sub-classification loss function for the third classification task. Wei3 is the sub-loss weight for the first classification task, Wei4 is the sub-loss weight for the second classification task, and Wei5 is the sub-loss weight for the third classification task.

[0193] It should be noted that the denominator in the above formula can also be the sum of the three sub-classification loss functions, which is not limited here.

[0194] It should be noted that in the scenario described above where M training sample sets are set up one-to-one for each of the M classification tasks, the number of training samples included in these training sample sets may be inconsistent or even significantly different due to the number of classification labels.

[0195] Therefore, in one possible implementation, the number of training samples called in a training batch is the same for each classification task.

[0196] In other words, even if the number of training samples in the training sample set is inconsistent, the number of training samples extracted in each batch is consistent. This can achieve the effect of sample balancing and improve the learning balance of different classification tasks during model training.

[0197] To make it easier to explain, we will first briefly describe the basic situation of model training, taking video training samples as content training samples as an example.

[0198] For a training sample set, which includes N video training samples from the full dataset, the model is trained through multiple epochs.

[0199] An iteration is defined as feeding all N video training samples into the model for training at once. In each iteration, due to limited video memory resources on the computer used for training, it's impossible to feed all N video training samples into the model at once. Therefore, all N video training samples are trained in batches, with each batch consisting of 'bs' samples. These batches are then fed into the model for forward computation, backward computation, and model parameter updates. An iteration is considered complete when all N / bs batches have completed their model parameter updates.

[0200] In each batch, bs video training samples are randomly selected. Each video training sample is input into the model in Tables 1-5 to obtain the merged content vector and the predictions for the three downstream classification tasks. The loss loss1 (including intra-modal and cross-modal restoration loss functions) is calculated based on self-supervised learning (intra-modal and cross-modal). The loss2 (i.e., the classification loss function) is calculated based on the classification task. The gradients of the network parameters are calculated based on loss1 and loss2 and fed back into the network to update the network parameters, thus completing the model training for one batch.

[0201] Training stops when the loss stops decreasing or after 100 rounds of training on the full dataset.

[0202] Having introduced the basics of model training, let's return to the point mentioned earlier that for each classification task, the number of training samples used in a training batch is the same. For example, if the number of training sample sets are N1, N2, and N3 respectively, then for the corresponding classification task, the number of training samples extracted from the training sample sets in each batch is bs.

[0203] To enhance the role of the vector merging model as a unified vector extraction framework, different loss functions need to be controlled in a targeted manner during model training.

[0204] In one possible implementation, one embodiment of the aforementioned S204 involves training the initial merging model using the classification loss function, the intra-modal reconstruction loss function, and the cross-modal reconstruction loss function to obtain a vector merging model, which may include:

[0205] S21: Configure a first loss weight for the intra-modal restoration loss function and the cross-modal restoration loss function, and configure a second loss weight for the classification loss function, wherein the first loss weight is greater than the second loss weight;

[0206] S22: Determine the total loss function based on the intra-modal restoration loss function, the cross-modal restoration loss function, the first loss weight, the classification loss function, and the second loss weight;

[0207] S23: Train the initial merging model according to the total loss function to obtain the vector merging model.

[0208] In other words, the server sets different loss weights to differentiate the influence of the intra-modal and cross-modal reconstruction loss functions obtained based on self-supervised training, as well as the classification loss function obtained based on supervised training. This is to strengthen the influence of self-supervised training on the initial merged model and weaken the influence of supervised training on the initial merged model.

[0209] The reason is that strengthening the intra-modal and cross-modal reconstruction loss functions obtained from self-supervised training can improve the knowledge that the initial merging model learns about how to provide a unified merged content vector for different classification tasks. Weakening the classification loss function obtained from supervised training can prevent the initial merging model from learning too much preference for the information carried by the content vector for a specific classification task, thereby ensuring the applicability of the merged content vector.

[0210] For example, in the aforementioned example, loss weights of 0.9 and 0.1 can be set for loss1 and loss2 respectively to reduce the impact of the classification task involved in training the initial merging model and ensure the applicability of the merged content vector determined by the vector merging model.

[0211] Next, we will explain the types of classification loss functions that can be used for different classification tasks.

[0212] When the classification task is video classification, cross entropy loss can be used to calculate the classification loss function:

[0213]

[0214] Where pic represents the predicted probability that video training sample i belongs to category c, and yic represents whether the label of video training sample i is c; if it is c, yic = 1, otherwise it is 0. N is the number of video training samples in the current batch. The classification loss function for each batch is: calculate the corresponding classification loss function for each video training sample, and then obtain the final classification loss function by averaging.

[0215] When the classification task is a multi-label classification task, its classification loss function (bce loss) can be expressed by the following formula:

[0216]

[0217] For a training sample i, its true label vector t[i] is a 0 and 1 vector of 1*Nclass, and its predicted value o[i] is the predicted probability of the training sample i for each of the N labels.

[0218] When a certain label has a true value of 1, the left side of the plus sign in the above expression takes effect; when the true value is 0, the right side of the plus sign takes effect.

[0219] When the classification task is popularity prediction, the regression loss of the root mean square error (MSE) can be used to determine the classification loss function: This formula has already been described in the foregoing embodiments and will not be repeated here.

[0220] After obtaining the vector merging model through self-supervised and supervised training, this model can serve as a unified vector extraction framework, providing vector extraction services for different classification tasks. This application provides a corresponding solution when a new classification task needs to utilize this vector merging model before deployment.

[0221] In one possible implementation, based on the vector merging model determined in the foregoing embodiments, the method further includes:

[0222] S31: Obtain the initial classification model corresponding to the target classification task;

[0223] S32: Based on the target content training samples of the target classification task, determine the corresponding target classification result through the vector merging model and the initial classification model;

[0224] S33: Determine the target classification loss function based on the difference between the target classification result and the target classification label of the target content training sample;

[0225] S34: Train the initial classification model using the target classification loss function to obtain the target classification model corresponding to the target classification task.

[0226] The initial classification model is the classification model that needs to be trained before the target classification task goes online. Based on the target content training samples of the target classification task, the corresponding merged sample vector can be obtained directly through the already trained vector merging model. The merged sample vector is used as the input vector of the initial classification model, and the target classification result corresponding to the target content training samples is obtained according to the initial classification model.

[0227] Similar to the supervised training described above, the target classification loss function is determined by using the target classification results and the target content training samples' target classification labels. The difference is that the model training for the target classification task only involves the initial classification model and does not involve a high-quality vector merging model. That is, during the model training process using the target classification loss function, the model parameters of the vector merging model are fixed, and only the model parameters of the initial classification model are adjusted so that the trained target classification model can provide accurate classification services for the target classification task.

[0228] Since the model parameters of the vector merging model are not adjusted throughout the process, the convergence speed of model training can be greatly accelerated, improving the deployment speed and ensuring the deployment quality of target classification tasks.

[0229] The following section describes the training method of the vector merging model in this application embodiment from an overall perspective, as well as the training method when a target classification task is added.

[0230] like Figure 5 The diagram illustrates a scenario where the initial merging model is trained using video training samples. In this scenario, there are three modalities: video frames, audio frames, and text. Through vector extraction, video frame vectors, audio frame vectors, and text vectors are extracted for each modality. These three vectors are then input into the initial merging model. Based on the aforementioned self-supervised training, including intra-modal reconstruction prediction and cross-modal reconstruction prediction, Loss1, consisting of intra-modal reconstruction loss functions and cross-modal reconstruction loss functions, is obtained. The merged sample vectors determined by the initial merging model are then input into single-modal prediction models corresponding to the three classification tasks (shown as single-modal prediction models 1-3 in the diagram). Supervised training using the classification labels of the video training samples yields the classification loss function lLoss2.

[0231] The server trains the initial merging model using Loss1 and Loss2 to obtain the vector merging model.

[0232] Once the vector merging model is trained, if a new classification task requires calling the vector merging model, it can be done as follows: Figure 6 As shown, the single-modal prediction model 4 corresponds to the classification task, namely the aforementioned initial classification model.

[0233] In this scenario, based on the video training samples related to the classification task, the initial content vectors of the three modalities of the video training samples are merged into a content sample vector through the vector merging model. The corresponding target classification result is determined by the single-modal prediction model 4, and the target classification loss function as Loss3 is obtained based on supervised training.

[0234] Guided by loss3, a single-modal prediction model 4 is trained to obtain the target classification model.

[0235] The following example uses video to illustrate how to extract vectors from the three modalities of video frames, audio frames, and text.

[0236] For the video frame modality, images are obtained by extracting frames from the video, and image embedding models from related technologies are used to extract embedding features for each image.

[0237] This model is used to extract embedding features from video frames: it employs the feature extraction module of ResNet101 and pooling layers, followed by an embedding layer to output image embedding features. The network structure is shown in Table 6. The model is trained on business images using metric learning and triplet loss. This solution does not require updating the model.

[0238] For each video frame extraction, one frame is extracted per second for feature extraction. For videos with more than 64 frames, a uniform frame extraction method is used to extract 64 frames of image features evenly from the video training samples as subsequent unified feature training. For videos with less than 64 seconds, a copying method is used to reach 64 seconds. Thus, each video has 64 video feature vectors (each feature is 1x128 dimensional).

[0239]

[0240] Table 6

[0241] For vector extraction of video frames, please refer to Figure 7 Part a shows a scenario where the extracted vector is used as the initial content vector. In other scenarios, the content vector obtained by vector extraction is input into the initial merging model, and the output of the map layer in Table 1 is the initial content vector.

[0242] For the audio frame modality, one audio feature is extracted from each second (0.96s) of the audio frames in the video training samples. These audio features can be extracted using the open-source vggish tool. For audio longer than 64 seconds, a uniform sampling method is used to extract 64 frames of features evenly from the entire audio stream for subsequent unified feature training. For audio shorter than 64 seconds, a copying method is used to reach 64 seconds. Thus, each video has 64 audio features (each feature is 1x128-dimensional).

[0243] Preparation for audio frames: First, extract the audio WAV files from the video training samples. The feature extraction process for the audio files is as follows:

[0244] 1. Resample the audio to 16kHz mono.

[0245] 2. Using a 25 ms Hann time window and a 10 ms frame shift, a short-time Fourier transform is performed on the audio to obtain the spectrogram;

[0246] 3. Calculate the mel spectrum by mapping the spectrogram to a 64th-order Mel cepstral (mel) filter bank;

[0247] 4. Calculate log(mel-spectrum + 0.01) to obtain a stable mel spectrum. The added bias of 0.01 is to avoid taking the logarithm of 0.

[0248] 5. These features are then framed with a duration of 0.96s, with no frame overlap. Each frame contains 64 mel frequency bands and a duration of 10ms (i.e., a total of 96 frames), resulting in the embedding result: The VGGish model output data format is [nums_frames, 128], where nums_frames is the frame length, and nums_frames = audio duration / 0.96.

[0249] For vector extraction of audio frames, please refer to Figure 7 Part b shows a scenario where the extracted vector is used as the initial content vector. In other scenarios, the content vector obtained by vector extraction is input into the initial merging model, and the output of the map layer in Table 1 is the initial content vector.

[0250] For the text modality, the open-source BERT text extraction embedding model based on the transformer network structure can be used to extract the video title embedding.

[0251] For vector extraction of text, please refer to Figure 7 Part c shows a scenario where the extracted vector is used as the initial content vector. In other scenarios, the content vector obtained by vector extraction is input into the initial merging model, and the output of the map layer in Table 1 is the initial content vector.

[0252] In summary, the model determination method provided in the embodiments of this application can at least achieve the following:

[0253] To address the issue of excessive training resource consumption, this solution performs classification learning based on a unified initial content vector, rather than re-extracting initial content vectors from the content training samples. By effectively designing feature transformation enhancement methods, it achieves rapid training on massive datasets.

[0254] We abstract a unified vector extraction framework and use intra-modal and cross-modal self-supervised training to assist learning, thereby achieving a unified representation effect for content vectors.

[0255] For different classification tasks, a weighted auxiliary learning method with balanced sampling and loss adjustment is designed in the training process of classification tasks to improve the representation effect of the unified content vector while learning multiple downstream tasks.

[0256] In the foregoing Figure 1-7 Based on the corresponding embodiments, Figure 8 This is a structural diagram of a model determination device provided in an embodiment of this application. The model determination device 800 includes an acquisition unit 801, a prediction unit 802, and a training unit 803.

[0257] The acquisition unit 801 is used to acquire N initial content vectors that correspond one-to-one with N modalities for the content training samples, where N>1;

[0258] The prediction unit 802 is used to perform intra-modal reconstruction prediction based on the N initial content vectors through an initial merging model and a single-modal prediction model to obtain an intra-modal reconstruction loss function. For the target modality among the N modalities, the corresponding intra-modal reconstruction loss function is determined based on the difference between the initial content vector and the intra-modal reconstruction vector of the target modality. The intra-modal reconstruction vector is obtained by reconstructing the initial content vector of the target modality with missing features through the initial merging model and the single-modal prediction model.

[0259] The prediction unit 802 is further configured to perform cross-modal reconstruction prediction based on the N initial content vectors, using the initial merging model and the single-modal prediction model, to obtain a cross-modal reconstruction loss function; for the target modality, the corresponding cross-modal reconstruction loss function is determined based on the difference between the initial content vector of the target modality and the cross-modal reconstruction vector, wherein the cross-modal reconstruction vector is obtained from the initial content vector of the modality to be synthesized from the N modalities through the initial merging model and the single-modal prediction model;

[0260] The training unit 803 is used to train the initial merging model using the intra-modal restoration loss function and the cross-modal restoration loss function to obtain a vector merging model. The vector merging model is used to obtain a merged content vector based on the content to be processed and the N content vectors corresponding one-to-one with the N modalities.

[0261] In one possible implementation, the prediction unit is further configured to:

[0262] Based on the initial content vector of the target modality, some features are discarded to obtain the initial missing vector;

[0263] The initial merging model is used to determine the merged sample missing vector corresponding to the initial missing vector;

[0264] Based on the merged sample missing vector, the intra-modal reconstruction vector of the target modality is predicted using the single-modal prediction model corresponding to the target modality;

[0265] The intramodal restoration loss function is determined based on the difference between the initial content vector and the intramodal restoration vector.

[0266] In one possible implementation, the N initial content vectors are feature vectors extracted from the corresponding modality of the content training samples, or the N initial content vectors are feature vectors obtained by passing through the feature dimension transformation layer of the corresponding modality in the initial merging model based on the feature vectors extracted from the corresponding modality of the content training samples.

[0267] In one possible implementation, the prediction unit is further configured to:

[0268] From the N modes, determine the mode to be synthesized for synthesis;

[0269] The initial merging model is used to determine the merged sample content vector corresponding to the initial content vector of the modality to be synthesized.

[0270] Based on the merged sample content vector, the cross-modal reconstruction vector of the target modality is predicted using the single-modal prediction model corresponding to the target modality;

[0271] The cross-modal reconstruction loss function is determined based on the difference between the initial content vector of the target modality and the cross-modal reconstruction vector.

[0272] In one possible implementation, the prediction unit is further configured to:

[0273] Determine the feature granularity of the N initial content vectors;

[0274] The modality to be synthesized is determined from modalities with a finer feature granularity than the initial content vector of the target modality.

[0275] In one possible implementation, the training unit is further used for:

[0276] Determine the degree of difference identified by the intramodal reduction loss function;

[0277] In response to the difference level being lower than the difference condition, the initial merging model is trained according to the intra-modal restoration loss function and the cross-modal restoration loss function;

[0278] In response to the difference level being greater than or equal to the difference condition, the cross-modal restoration loss function is discarded, and the initial merged model is trained based on the intra-modal restoration loss function.

[0279] In one possible implementation, the initial merging model includes an attention layer that maps the input feature vectors into an attention space, the dimension of which is lower than the dimension of the feature space of the input feature vectors.

[0280] In one possible implementation, the prediction unit is further configured to:

[0281] Based on the N initial content vectors, the merged sample vector of the content training samples is obtained through the initial merging model;

[0282] The classification result corresponding to the merged sample vector is determined by the classification model corresponding to the classification task;

[0283] Based on the difference between the classification results and the classification labels of the content training samples, determine the classification loss function;

[0284] The training unit is also used to train the initial merging model using the classification loss function, the intra-modal restoration loss function, and the cross-modal restoration loss function to obtain the vector merging model.

[0285] In one possible implementation, the prediction unit is further configured such that when the classification task includes M types, M>1:

[0286] By using the classification models corresponding to the M classification tasks respectively, the classification results of the merged sample vector corresponding to the M classification tasks are determined.

[0287] The step of determining the classification loss function based on the difference between the classification result and the classification labels of the content training samples includes:

[0288] For each classification task, a subclassification loss function corresponding to the classification task is determined based on the classification result of the classification task and the difference between the subclassification labels of the content training samples corresponding to the classification task.

[0289] The classification loss function is determined based on the sub-classification loss function for each classification task.

[0290] In one possible implementation, the prediction unit is further configured to:

[0291] Based on the sub-classification loss function of each classification task, determine the sub-loss weights corresponding to the sub-classification loss function respectively. The sub-loss weights are used to normalize the sub-classification loss function of each classification task to the same scale space.

[0292] The classification loss function is determined based on the sub-classification loss function for each classification task and the corresponding sub-loss weights.

[0293] In one possible implementation, for each classification task, the number of training samples called in a training batch is the same.

[0294] In one possible implementation, the training unit is further used for:

[0295] Configure a first loss weight for the intra-modal reconstruction loss function and the cross-modal reconstruction loss function, and configure a second loss weight for the classification loss function, wherein the first loss weight is greater than the second loss weight;

[0296] The total loss function is determined based on the intra-modal restoration loss function, the cross-modal restoration loss function and the first loss weight, as well as the classification loss function and the second loss weight;

[0297] The initial merging model is trained based on the total loss function to obtain the vector merging model.

[0298] In one possible implementation, the training unit is also used after obtaining the vector merging model:

[0299] Obtain the initial classification model corresponding to the target classification task;

[0300] Training samples are used based on the target content of the target classification task, and the corresponding target classification result is determined by the vector merging model and the initial classification model.

[0301] Based on the difference between the target classification result and the target classification label of the target content training sample, the target classification loss function is determined;

[0302] The initial classification model is trained using the target classification loss function to obtain the target classification model corresponding to the target classification task.

[0303] As can be seen, N initial content vectors are obtained, each corresponding one-to-one with one of the N modalities, and based on these N initial content vectors, the initial merging model and the single-modal prediction model are used to reconstruct and predict the missing features, resulting in an intra-modal reconstruction loss function. Furthermore, based on the initial content vectors of other modalities, the initial content vector of the target modality is reconstructed and predicted, resulting in a cross-modal reconstruction loss function. This intra-modal reconstruction loss function guides the initial merging model to learn how to extract richer information within a modality, while the cross-modal reconstruction loss function guides the initial merging model to learn how to reference relevant information from other modalities when determining the content vector corresponding to a modality, thus avoiding overfitting during training of the initial merging model for determining the content vector of a single modality. The vector merging model trained based on these two reduction loss functions produces content vectors with richer information for each modality. Furthermore, since determining the content vector for one modality considers relevant information from other modalities, the correlation between content vectors from different modalities is better. The merged content vector obtained by merging N content vectors is not only rich in information but also exhibits correlation between information from different modalities. High-quality merged content vectors are applicable to various classification tasks, allowing the vector merging model to be used directly for content vector extraction without repeated model training for content vector extraction. This significantly improves the training efficiency of classification tasks and effectively meets the rapidly evolving content classification needs.

[0304] This application also provides a computer device, which is the computer device described above, and may include a terminal device or a server. The aforementioned model determining device may be configured in this computer device. The computer device will now be described in conjunction with the accompanying drawings.

[0305] If the computer device is a terminal device, please refer to Figure 9 As shown, this application provides a terminal device, taking a mobile phone as an example:

[0306] Figure 9 This diagram illustrates a partial structural representation of a mobile phone related to the terminal device provided in this embodiment. (Reference) Figure 9 The mobile phone includes components such as a radio frequency (RF) circuit 1410, a memory 1420, an input unit 1430, a display unit 1440, a sensor 1450, an audio circuit 1460, a Wi-Fi module 1470, a processor 1480, and a power supply 1490. Those skilled in the art will understand that... Figure 9The mobile phone structure shown does not constitute a limitation on the mobile phone and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0307] The following is combined with Figure 9 A detailed introduction to each component of a mobile phone:

[0308] The RF circuit 1410 can be used to receive and transmit signals during information transmission or calls. In particular, it receives downlink information from the base station and processes it with the processor 1480; in addition, it transmits uplink data to the base station.

[0309] The memory 1420 can be used to store software programs and modules. The processor 1480 executes various mobile phone functions and data processing by running the software programs and modules stored in the memory 1420. The memory 1420 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 1420 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0310] The input unit 1430 can be used to receive input numeric or character information, and to generate key signal inputs related to user settings and function control of the mobile phone. Specifically, the input unit 1430 may include a touch panel 1431 and other input devices 1432.

[0311] The display unit 1440 can be used to display information input by the user or information provided to the user, as well as various menus of the mobile phone. The display unit 1440 may include a display panel 1441.

[0312] The mobile phone may also include at least one sensor 1450, such as a light sensor, a motion sensor, and other sensors.

[0313] Audio circuitry 1460, speaker 1461, and microphone 1462 provide an audio interface between the user and the mobile phone.

[0314] WiFi is a short-range wireless transmission technology. Through the WiFi module 1470, mobile phones can help users send and receive emails, browse web pages, and access streaming media, providing users with wireless broadband internet access.

[0315] The processor 1480 is the control center of the mobile phone. It connects to various parts of the mobile phone through various interfaces and lines. It performs various functions of the mobile phone and processes data by running or executing software programs and / or modules stored in the memory 1420 and calling data stored in the memory 1420.

[0316] The phone also includes a power supply 1490 (such as a battery) that powers the various components.

[0317] In this embodiment, the processor 1480 included in the terminal device also has the following functions:

[0318] Obtain N initial content vectors that correspond one-to-one with the N modalities of the content training samples, where N>1;

[0319] Based on the N initial content vectors, intra-modal reconstruction prediction is performed using an initial merging model and a single-modal prediction model to obtain an intra-modal reconstruction loss function. For the target modality among the N modalities, the corresponding intra-modal reconstruction loss function is determined based on the difference between the initial content vector and the intra-modal reconstruction vector of the target modality. The intra-modal reconstruction vector is obtained by reconstructing the initial content vector of the target modality with missing features using the initial merging model and the single-modal prediction model.

[0320] Based on the N initial content vectors, cross-modal reconstruction prediction is performed using the initial merging model and the single-modal prediction model to obtain a cross-modal reconstruction loss function. For the target modality, the corresponding cross-modal reconstruction loss function is determined based on the difference between the initial content vector and the cross-modal reconstruction vector of the target modality. The cross-modal reconstruction vector is obtained from the initial content vector of the modality to be synthesized from the N modalities using the initial merging model and the single-modal prediction model.

[0321] The initial merging model is trained using the intra-modal restoration loss function and the cross-modal restoration loss function to obtain a vector merging model. The vector merging model is used to obtain a merged content vector based on the content to be processed and the N content vectors corresponding one-to-one with the N modalities.

[0322] If the computer device is a server, this application embodiment also provides a server; please refer to [link to relevant documentation]. Figure 10 As shown, Figure 10This is a structural diagram of a server 1500 provided in an embodiment of this application. The server 1500 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 1522 (e.g., one or more processors) and memory 1532, and one or more storage media 1530 (e.g., one or more mass storage devices) for storing application programs 1542 or data 1544. The memory 1532 and storage media 1530 can be temporary or persistent storage. The program stored in the storage media 1530 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the server. Furthermore, the CPU 1522 may be configured to communicate with the storage media 1530 and execute the series of instruction operations in the storage media 1530 on the server 1500.

[0323] Server 1500 may also include one or more power supplies 1526, one or more wired or wireless network interfaces 1550, one or more input / output interfaces 1558, and / or one or more operating systems 1541, such as Windows Server. TM Mac OS X TM Unix TM Linux TM FreeBSD TM etc.

[0324] The steps performed by the server in the above embodiments can be based on Figure 10 The server structure shown.

[0325] In addition, this application embodiment also provides a storage medium for storing a computer program for executing the method provided in the above embodiment.

[0326] This application also provides a computer program product including a computer program, which, when run on a computer device, causes the computer device to perform the method provided in the above embodiments.

[0327] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium can be at least one of the following media: read-only memory (ROM), RAM, magnetic disk or optical disk, and other media that can store computer programs.

[0328] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the device and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments. The device and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the solution in this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0329] The above description is merely one specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Moreover, based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A model determination method, characterized in that, The method includes: Obtain N initial content vectors that correspond one-to-one with N modalities for the content training samples, where N>1, and the content training samples are video training samples; Based on the N initial content vectors, intra-modal reconstruction prediction is performed using an initial merging model and a single-modal prediction model to obtain an intra-modal reconstruction loss function. For the target modality among the N modalities, the corresponding intra-modal reconstruction loss function is determined based on the difference between the initial content vector and the intra-modal reconstruction vector of the target modality. The intra-modal reconstruction vector is obtained by partially discarding features from the initial content vector of the target modality and then reconstructing it from the initial content vector of the target modality with missing features using the initial merging model and the single-modal prediction model. The initial merging model is used to merge the input vectors and output a merged vector, and the single-modal prediction model is used to restore the merged vector to the complete content vector of the corresponding modality. Based on the N initial content vectors, cross-modal reconstruction prediction is performed using the initial merging model and the single-modal prediction model to obtain a cross-modal reconstruction loss function. For the target modality, the corresponding cross-modal reconstruction loss function is determined based on the difference between the initial content vector of the target modality and the cross-modal reconstruction vector. The cross-modal reconstruction vector is obtained from the initial content vectors of the modal to be synthesized (excluding the target modality) among the N modalities using the initial merging model and the single-modal prediction model. The initial merging model is trained using the intra-modal restoration loss function and the cross-modal restoration loss function to obtain a vector merging model. The vector merging model is used to obtain a merged content vector based on the content to be processed and the N content vectors corresponding one-to-one with the N modalities.

2. The method according to claim 1, characterized in that, The step of performing intra-modal reconstruction prediction based on the N initial content vectors through an initial merging model and a single-modal prediction model to obtain the intra-modal reconstruction loss function includes: Based on the initial content vector of the target modality, some features are discarded to obtain the initial missing vector; The initial merging model is used to determine the merged sample missing vector corresponding to the initial missing vector; Based on the merged sample missing vector, the intra-modal reconstruction vector of the target modality is predicted using the single-modal prediction model corresponding to the target modality; The intramodal restoration loss function is determined based on the difference between the initial content vector and the intramodal restoration vector.

3. The method according to claim 2, characterized in that, The N initial content vectors are feature vectors extracted from the corresponding modal of the content training samples, or the N initial content vectors are feature vectors obtained by passing through the feature dimension transformation layer of the corresponding modality in the initial merging model based on the feature vectors extracted from the corresponding modal of the content training samples.

4. The method according to claim 1, characterized in that, The step of performing cross-modal reconstruction prediction based on the N initial content vectors, using the initial merging model and the single-modal prediction model, to obtain the cross-modal reconstruction loss function includes: From the N modes, determine the mode to be synthesized for synthesis; The initial merging model is used to determine the merged sample content vector corresponding to the initial content vector of the modality to be synthesized. Based on the merged sample content vector, the cross-modal reconstruction vector of the target modality is predicted using the single-modal prediction model corresponding to the target modality; The cross-modal reconstruction loss function is determined based on the difference between the initial content vector of the target modality and the cross-modal reconstruction vector.

5. The method according to claim 4, characterized in that, The step of determining the mode to be synthesized from the N modes includes: Determine the feature granularity of the N initial content vectors; The modality to be synthesized is determined from modalities with a finer feature granularity than the initial content vector of the target modality.

6. The method according to claim 1, characterized in that, The step of training the initial merging model using the intra-modal restoration loss function and the cross-modal restoration loss function includes: Determine the degree of difference identified by the intramodal reduction loss function; In response to the difference level being lower than the difference condition, the initial merging model is trained according to the intra-modal restoration loss function and the cross-modal restoration loss function; In response to the difference level being greater than or equal to the difference condition, the cross-modal restoration loss function is discarded, and the initial merged model is trained based on the intra-modal restoration loss function.

7. The method according to claim 1, characterized in that, The initial merging model includes an attention layer, which maps the input feature vector to an attention space, the dimension of which is lower than the dimension of the feature space of the input feature vector.

8. The method according to any one of claims 1-7, characterized in that, The method further includes: Based on the N initial content vectors, the merged sample vector of the content training samples is obtained through the initial merging model; The classification result corresponding to the merged sample vector is determined by the classification model corresponding to the classification task; Based on the difference between the classification results and the classification labels of the content training samples, determine the classification loss function; The step of training the initial merging model using the intra-modal restoration loss function and the cross-modal restoration loss function to obtain the vector merging model includes: The initial merging model is trained using the classification loss function, the intra-modal restoration loss function, and the cross-modal restoration loss function to obtain the vector merging model.

9. The method according to claim 8, characterized in that, When the classification task includes M types, M>1, determining the classification result corresponding to the merged sample vector using the classification model corresponding to the classification task includes: By using the classification models corresponding to the M classification tasks respectively, the classification results of the merged sample vector corresponding to the M classification tasks are determined. The step of determining the classification loss function based on the difference between the classification result and the classification labels of the content training samples includes: For each classification task, a subclassification loss function corresponding to the classification task is determined based on the classification result of the classification task and the difference between the subclassification labels of the content training samples corresponding to the classification task. The classification loss function is determined based on the sub-classification loss function for each classification task.

10. The method according to claim 9, characterized in that, The step of determining the classification loss function based on the sub-classification loss function for each classification task includes: Based on the sub-classification loss function of each classification task, determine the sub-loss weights corresponding to the sub-classification loss function respectively. The sub-loss weights are used to normalize the sub-classification loss function of each classification task to the same scale space. The classification loss function is determined based on the sub-classification loss function for each classification task and the corresponding sub-loss weights.

11. The method according to claim 9, characterized in that, For each classification task, the number of training samples called in a training batch is the same.

12. The method according to claim 8, characterized in that, The process of training the initial merging model using the classification loss function, the intra-modal restoration loss function, and the cross-modal restoration loss function to obtain the vector merging model includes: Configure a first loss weight for the intra-modal reconstruction loss function and the cross-modal reconstruction loss function, and configure a second loss weight for the classification loss function, wherein the first loss weight is greater than the second loss weight; The total loss function is determined based on the intra-modal restoration loss function, the cross-modal restoration loss function and the first loss weight, as well as the classification loss function and the second loss weight; The initial merging model is trained based on the total loss function to obtain the vector merging model.

13. The method according to claim 8, characterized in that, After obtaining the vector merging model, the method further includes: Obtain the initial classification model corresponding to the target classification task; Training samples are used based on the target content of the target classification task, and the corresponding target classification result is determined by the vector merging model and the initial classification model. Based on the difference between the target classification result and the target classification label of the target content training sample, the target classification loss function is determined; The initial classification model is trained using the target classification loss function to obtain the target classification model corresponding to the target classification task.

14. A model determining device, characterized in that, The device includes an acquisition unit, a prediction unit, and a training unit: The acquisition unit is used to acquire N initial content vectors that correspond one-to-one with N modalities for the content training samples, where N>1, and the content training samples are video training samples. The prediction unit is used to perform intra-modal reconstruction prediction based on the N initial content vectors using an initial merging model and a single-modal prediction model, to obtain an intra-modal reconstruction loss function. For the target modality among the N modalities, the corresponding intra-modal reconstruction loss function is determined based on the difference between the initial content vector of the target modality and the intra-modal reconstruction vector. The intra-modal reconstruction vector is obtained by partially discarding features from the initial content vector of the target modality and then reconstructing it from the initial content vector of the target modality with missing features using the initial merging model and the single-modal prediction model. The initial merging model is used to merge the input vectors and output a merged vector, and the single-modal prediction model is used to restore the merged vector to the complete content vector of the corresponding modality. The prediction unit is further configured to perform cross-modal reconstruction prediction based on the N initial content vectors, using the initial merging model and the single-modal prediction model, to obtain a cross-modal reconstruction loss function; for the target modality, the corresponding cross-modal reconstruction loss function is determined based on the difference between the initial content vector of the target modality and the cross-modal reconstruction vector, wherein the cross-modal reconstruction vector is obtained from the initial content vectors of the modal to be synthesized (excluding the target modality) among the N modalities using the initial merging model and the single-modal prediction model; The training unit is used to train the initial merging model using the intra-modal restoration loss function and the cross-modal restoration loss function to obtain a vector merging model. The vector merging model is used to obtain a merged content vector based on the content to be processed and the N content vectors corresponding one-to-one with the N modalities.

15. A computer device, characterized in that, The computer device includes a processor and memory: The memory is used to store computer programs and to transfer the computer programs to the processor; The processor is configured to perform the method according to any one of claims 1-13 according to the computer program.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program for performing the method according to any one of claims 1-13.

17. A computer program product comprising a computer program, which, when run on a computer device, causes the computer device to perform the method of any one of claims 1-13.

Images