Modal correlation prediction methods, devices, equipment, storage media, and program products

Abstract

The application discloses a modal correlation degree prediction method and device, equipment, a storage medium and a program product, and relates to the field of machine learning. The method comprises the following steps: acquiring a sample content set, extracting a second modal feature vector corresponding to second modal data of the sample content, determining a feature vector center corresponding to a plurality of sample classifications respectively, determining the distance between the second modal feature vector and the feature vector center as the correlation degree label between the second modal data and the first modal data in the sample content, training a candidate correlation degree identification model based on the correlation degree label to obtain a correlation degree identification model, using the model to identify the correlation degree between the first modal data and the second modal data in target content, and extracting the semantic feature representation of the target content based on the correlation degree. The correlation degree identification model can better learn the correlation between the modes in the multi-modal content, thereby assisting in enhancing the understanding ability of the article semantics.

Description

Technical Field

[0001] This application relates to the field of machine learning, and in particular to modal correlation prediction methods, apparatus, devices, storage media, and program products. Background Technology

[0002] News feed articles contain multiple modalities, including text and image modalities. As multimodal pre-trained models continue to develop, they are beginning to be applied to news feed articles to help enhance the understanding of article semantics and improve the accuracy of downstream tasks such as article classification, article tag extraction, and article quality prediction.

[0003] In related technologies, the training tasks of multimodal pre-trained models for information flow articles mainly include the task of masking and restoring text modalities and the task of matching text modalities with image modalities.

[0004] However, in the training task of the multimodal pre-trained model of the above-mentioned information flow articles, the matching task of text modality and image modality depends on cross-modal data with high correlation between text modality and image modality. Data with low correlation will affect the model's modeling ability and result in poor training effect. Summary of the Invention

[0005] This application provides a method, apparatus, device, storage medium, and program product for predicting modal correlation, which can improve the semantic correlation between modalities of multimodal content. The technical solution is as follows:

[0006] On the one hand, a modal correlation prediction method is provided, the method comprising:

[0007] Obtain a sample content set, wherein the sample content in the sample content set includes text modal data and image modal data, and the sample content is labeled with a classification label, which is used to indicate the sample category to which the sample content belongs;

[0008] Extract the second modality feature vector corresponding to the second modality data of the sample content;

[0009] Based on the second modal feature vectors corresponding to the sample content belonging to the same sample category, the feature vector centers corresponding to multiple sample categories are determined respectively.

[0010] The distance between the second modality feature vector corresponding to the second modality data and the feature vector center of the sample classification corresponding to the second modality data is determined and used as the correlation label between the second modality data and the first modality data in the sample content;

[0011] The candidate correlation recognition model is trained based on the sample content and the correlation labels corresponding to the sample content to obtain the correlation recognition model. The correlation recognition model is used to identify the correlation between the first modality data and the second modality data in the target content, and extracts the semantic feature representation of the target content based on the correlation. The semantic feature representation is used to characterize the semantics of the target content.

[0012] On the other hand, a modal correlation prediction device is provided, the device comprising:

[0013] The acquisition module is used to acquire a sample content set, wherein the sample content in the sample content set includes first modality data and second modality data, and the sample content is labeled with a classification label, which is used to indicate the sample category to which the sample content belongs;

[0014] The extraction module is used to extract the second modality feature vector corresponding to the second modality data of the sample content;

[0015] The determination module is used to determine the feature vector centers corresponding to multiple sample categories based on the second modality feature vectors corresponding to sample content belonging to the same sample category; and to determine the distance between the second modality feature vector corresponding to the second modality data and the feature vector center of the sample category corresponding to the second modality data, as a correlation label between the second modality data and the first modality data in the sample content.

[0016] The training module is used to train the candidate correlation recognition model based on the sample content and the correlation labels corresponding to the sample content to obtain the correlation recognition model. The correlation recognition model is used to identify the correlation between the first modality data and the second modality data in the target content, and to extract the semantic feature representation of the target content based on the correlation. The semantic feature representation is used to characterize the semantics of the target content.

[0017] On the other hand, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one instruction, at least one program, code set or instruction set, the at least one instruction, the at least one program, the code set or instruction set being loaded and executed by the processor to implement the modal correlation prediction method as described in any of the embodiments of this application above.

[0018] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction, at least one program, code set, or instruction set is stored therein, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the modal correlation prediction method as described in any of the embodiments of this application above.

[0019] On the other hand, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform any of the modal correlation prediction methods described in the above embodiments.

[0020] The beneficial effects of the technical solutions provided in this application include at least the following:

[0021] By leveraging classification information from multimodal content and combining it with cross-modal content corpus data, a candidate relevance recognition model is trained to obtain a relevance recognition model. This model identifies the relevance between first-modal and second-modal data within multimodal content and extracts semantic feature representations of the multimodal content based on these relevances. This enhances the semantic understanding of multimodal content, exemplified by news feed articles, and improves the accuracy of tasks such as article classification, article tag extraction, and article quality prediction. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of the training process of the correlation recognition model in the related art provided by an exemplary embodiment of this application;

[0024] Figure 2 This is a schematic diagram of the training process in the context of an information flow article in the related technology provided by an exemplary embodiment of this application;

[0025] Figure 3 This is a schematic diagram of an implementation environment provided by an exemplary embodiment of this application;

[0026] Figure 4 This is a flowchart illustrating the training process of an association recognition model in a related art provided by an exemplary embodiment of this application;

[0027] Figure 5 Based on Figure 4 The illustrated embodiment provides a flowchart of the iterative training process of the correlation recognition model;

[0028] Figure 6 This is a schematic diagram of a process for extracting and applying semantic feature representations of multimodal content, provided by another exemplary embodiment of this application;

[0029] Figure 7 This is a flowchart illustrating the process of constructing classification labels and the hierarchical construction process of multi-level classification sub-labels based on classification labels in the related technology provided in one embodiment of this application.

[0030] Figure 8 This is a schematic diagram illustrating the output of semantic vectors of multimodal content in a related technology provided by an exemplary embodiment of this application;

[0031] Figure 9 This is a structural block diagram of a modal correlation prediction device provided in an exemplary embodiment of this application;

[0032] Figure 10 This is a structural block diagram of a modal correlation prediction device provided in another exemplary embodiment of this application;

[0033] Figure 11 This is a structural block diagram of a computer device provided in an exemplary embodiment of this application. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0035] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0036] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, large feature extraction techniques, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0037] In this application embodiment, in the field of artificial intelligence, multimodal content refers to content containing multiple modalities of data. For example, taking a news feed article as an example, a news feed article typically includes text modal data and image modal data, where the correlation between text modal data and image modal data varies in strength. For instance, when the text modal data is about constellations and the image modal data is about animals, the correlation between the text modal data and the image modal data is weak; while when the text modal data is about constellations and the image modal data is about star charts, the correlation between the text modal data and the image modal data is strong.

[0038] In related technologies, when extracting features from multimodal content, multiple modal data from the multimodal content are input into a feature extraction network for feature extraction. The extracted features are then combined to analyze the multimodal content, such as predicting the topic corresponding to the multimodal content or analyzing the recommendation probability between the multimodal content and a specified user.

[0039] However, due to the weak correlation between the data of different modalities in multimodal content, the effectiveness of the features extracted from multimodal content is poor, which affects the downstream application of the extracted features, such as resulting in low accuracy in topic prediction and low accuracy in user recommendation.

[0040] In this embodiment, the correlation between modal data in multimodal content is predicted, and feature extraction is performed on multimodal content based on the correlation between modal data, so as to avoid the problem that the feature extraction effect of multimodal content is poor when the correlation between modal data is low.

[0041] Indicative, such as Figure 1 As shown, taking the analysis of information flow articles as an example, in this embodiment of the application, a sample corpus 100 is first obtained. The sample corpus 100 includes article corpus labeled with classification tags, and the article corpus includes text modal data and image modal data that meet the relevance requirements.

[0042] Image feature vectors 110 corresponding to the image modal data in each article corpus are extracted, and the image feature vectors in the feature space are clustered according to the classification labels labeled in the article corpus to obtain the feature vector center 120 corresponding to each category. Based on the distance between the image feature vector 110 and the feature vector center 120, the correlation degree 130 between the image modal data and the text modal data in the article corpus is determined.

[0043] Based on the correlation coefficient of 130 between image modal data and text modal data, a machine learning model for downstream applications is pre-trained. Downstream applications include, but are not limited to, article classification, article keyword extraction, article quality prediction, and article recommendation.

[0044] Schematically, such as Figure 2 shown, take the scenario of an information flow article with the title "Today's Horoscope for Capricorn" as an example. The information flow article 200 includes text modality content 210 and image modality content 220. For the pre-trained model, the input content of the text modality usually uses the article title and the article body content of the information flow article, and is truncated according to the pre-set model and the maximum length threshold. For example, in this example article, the threshold is set to retain the first 256 characters of the input text content.

[0045] Schematically, Figure 2 is a schematic diagram of the correlation analysis process proposed in this application. The text modality content 210 is "Today's Horoscope for Capricorn", that is, the title of the information flow article, and the image modality content 220 is a captured picture within the information flow article. Generally, the specified article illustration of the information flow article is selected, and the cover picture and other article illustrations of the information flow article can also be selected.

[0046] Such as Figure 2 shown, the left part is the input part of the text modality content 210, and the right part is the input part of the image modality content 220. The 7 characters "Today's Horoscope for Capricorn" in the text modality content 210 are split, and after passing through a feature transformation model 230, they become character features 240 "C, T1, T2, T3, T4, T5, T6". The second character in the text modality content 210, in order, the character "羯" is masked. Then, the masked character is predicted by the feature transformation model 230 and restored.

[0047] The right part is the input part of the image modality content 220. First, two representative pictures in the information flow article are passed through a pre-trained model 250 to obtain corresponding picture modality vectors 251 and picture modality vectors 252. After analysis by the feature transformation model 230, the correlation analysis results corresponding to the text modality content 210 and each image modality content 220 are obtained.

[0048] Secondly, the implementation environment involved in the embodiments of this application is described. Schematically, please refer to Figure 3 , in this implementation environment, a terminal 310 and a server 320 are involved, and the terminal 310 and the server 320 are connected through a communication network 330.

[0049] In some embodiments, terminal 310 is used to send multimodal content to server 320. In some embodiments, terminal 310 is equipped with an application that has multimodal content analysis capabilities (e.g., a news topic prediction function for information stream articles). For example, terminal 310 may be equipped with a search engine program, travel application, lifestyle assistance application, instant messaging application, video application, game application, news application, etc., but this embodiment does not limit the scope of the application.

[0050] After acquiring multimodal content, server 320 analyzes the correlation between the modal data in the multimodal content to obtain correlation analysis results. Based on the correlation analysis results, it extracts the feature vectors corresponding to the multimodal content, which are then applied to downstream multimodal content analysis tasks, such as multimodal content tag extraction, multimodal content classification, and multimodal content recommendation.

[0051] Among them, correlation analysis refers to analyzing the correlation between different modal data in multimodal content, such as conducting correlation analysis on text modal data and image modal data in information flow articles.

[0052] The aforementioned terminal can be various forms of terminal devices such as mobile phones, tablets, desktop computers, portable laptops, smart TVs, vehicle terminals, and smart home devices, and this application embodiment does not limit them.

[0053] It is worth noting that the aforementioned servers can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0054] Cloud technology refers to a hosting technology that unifies hardware, software, and network resources within a wide area network (WAN) or local area network (LAN) to achieve data computation, storage, processing, and sharing. Based on the cloud computing business model, cloud technology encompasses network technology, information technology, integration technology, management platform technology, and application technology. It can form resource pools, providing flexible and convenient on-demand access. Cloud computing technology will become a crucial support. Backend services of technical network systems require substantial computing and storage resources, such as video websites, image websites, and many portal websites. With the rapid development and application of the internet industry, every item may have its own identification mark in the future, requiring transmission to backend systems for logical processing. Data at different levels will be processed separately, and various industry data will require robust system support, which can only be achieved through cloud computing.

[0055] In some embodiments, the server described above can also be implemented as a node in a blockchain system.

[0056] Based on the above-described terminology and application scenarios, the modal correlation prediction method provided in this application will be explained. This method can be executed by a server or a terminal, or jointly by a server and a terminal. In this embodiment, the method is illustrated by being executed by a server. Figure 4 As shown, the method includes the following steps.

[0057] Step 401: Obtain the sample content set. The sample content set includes first modality data and second modality data, and the sample content is labeled with classification tags.

[0058] In some embodiments, the sample content set includes multiple sample contents, and each sample content consists of multimodal data, including first modal data and second modal data. Optionally, the first modal data and the second modal data belong to two different modalities.

[0059] Optionally, the sample content in the above sample content set can be implemented as an information flow article, wherein the information flow article includes text content as the main text and image content as the accompanying images, wherein the text content is text modal data and the image content is image modal data.

[0060] Optionally, the sample content in the above sample content set can also be social content published on social platforms, that is, public content published by social accounts on social platforms, such as: social videos and social texts published by social accounts on social platforms; or social images and social texts published by social accounts on social platforms; or social voice and social images published by social accounts on social platforms, etc., which are not limited in this application embodiment. Among them, social videos are composed of multiple image frames, and image frames or social images are used as image modal data; social text is used as text modal data; social voice can be used as voice modal data, or social voice can be converted to text and then used as text modal data.

[0061] It is worth noting that the above sample content is merely an illustrative example, and this embodiment does not limit it.

[0062] Optionally, the sample content in the sample content set includes both first modality data and second modality data. In this embodiment, taking an information flow article corpus as an example, the information flow article corpus includes multiple information flow articles, and each information flow article includes text modality data as first modality data and image modality data as second modality data.

[0063] In some embodiments, the sample content in the sample content set is pre-selected content, and the pre-selected sample content meets the modal correlation requirements. That is, among the pre-selected sample content, the first modal data and the second modal data meet the correlation requirements. For example, pre-selected sample content with a strong correlation between text modal data and image modal data constitutes the sample content and is added to the sample content set.

[0064] Optionally, when acquiring sample content, a classification label is assigned to the sample content. This classification label indicates the category to which the sample content belongs. Taking text modal data and image modal data as an example, when there is a strong correlation between text modal data and image modal data, the classification label can express both the text topic classification of the text modal data and the image content classification of the image modal data. For example, text content with the topic "constellation" is acquired as text modal data, and "constellation star chart" is acquired as image modal content. This text modal data and image modal data constitute sample content, and the classification label "constellation" is assigned, thus indicating that the text modal data and image modal data in this sample content are related to "constellation".

[0065] In this embodiment, a sample content set is pre-selected. The sample content in the sample content set not only includes first modality data and second modality data, but also has sample classification labels. These classification labels can clearly represent the classification of the sample content, helping to better understand the sample content and making it more convenient to study the relationship between the first modality data and the second modality data of the sample content during subsequent training.

[0066] Step 402: Extract the second modal feature vector corresponding to the second modal data of the sample content.

[0067] The second modality data in the sample content is input into the pre-trained feature extraction model, and the output is the second modality feature vector corresponding to the second modality data.

[0068] It is worth noting that in this embodiment, the second modality feature vector of the second modality data is extracted as an example. In some embodiments, the first modality feature vector of the first modality data can also be extracted, and the first modality feature vector can be clustered based on the classification label to obtain the vector center. This embodiment does not limit this.

[0069] In some embodiments, the feature extraction model is a pre-trained model for extracting features from the second modality data. Illustratively, taking image modality data as an example, the feature extraction model is an image feature extraction model. Optionally, the image feature extraction model is used to extract texture features, color features, shape features, spatial relationship features, etc., from the image modality data.

[0070] In this embodiment, the second modal feature vector corresponding to the second modal data of the sample content is extracted, and the second modal data of different forms in the sample content is converted into feature vectors. All features of the second modal data are converted into numbers and represented by numbers, which more intuitively and concisely represents the second modal data of the sample content.

[0071] Step 403: Based on the second modal feature vectors corresponding to the sample content belonging to the same sample category, determine the feature vector centers corresponding to the multiple sample categories respectively.

[0072] Optionally, after extracting the second modality feature vectors corresponding to the sample content, since the sample content itself is labeled with classification labels, which are used to characterize sample classification, based on the classification labels, the second modality feature vectors corresponding to the sample content labeled with the same classification label (i.e., belonging to the same sample category) are first clustered in the feature space to obtain the clustering results corresponding to each sample category. For each sample category's clustering result, the feature center corresponding to each second modality feature vector in the clustering result is obtained as the feature vector center corresponding to that sample category. That is, the second modality feature vectors corresponding to the sample content belonging to the same sample category are averaged to obtain the feature vector centers corresponding to multiple sample categories.

[0073] Indicatively, the sample content belonging to the category label "constellation" includes sample content A, sample content B, and sample content C. Feature 1 is extracted from image a in sample content A, feature 2 is extracted from image b in sample content B, and feature 3 is extracted from image c in sample content C. Features 1, 2, and 3 are averaged in the feature space, that is, the elements at the same position in the three feature vectors are averaged to obtain the feature vector center corresponding to the category label "constellation".

[0074] It is worth noting that the above example uses an image as an example of sample content. In some embodiments, the first modal data and the second modal data in the sample content can be one or more. This embodiment does not limit this. For example, an information flow article includes a piece of text content and multiple image content.

[0075] In this embodiment, the feature vector center corresponding to each sample category is determined, and the sample categories belonging to different fields are represented by numbers and presented in vector form, which more intuitively and concisely represents the sample category to which the sample content belongs.

[0076] Step 404: Determine the distance between the second modality feature vector corresponding to the second modality data and the feature vector center of the sample classification corresponding to the second modality data, and use it as the correlation label between the second modality data and the first modality data in the sample content.

[0077] Optionally, since all sample content in the sample content set is pre-selected, and each sample content belongs to its corresponding sample category, and the first modality data in the sample content is strongly correlated with the sample category, the correlation between the first modality data and the second modality data in each sample content is transformed into the correlation between the sample category and the second modality data.

[0078] In related technologies, when predicting whether there is a match between the first and second modal data in a multimodal model, a hard label with a correlation degree of 0 or 1 is assigned between the first and second modal data. Label 0 represents that the first and second modal data do not match, i.e., they are unrelated; while label 1 represents that the first and second modal data match, i.e., they are related. This method of assigning correlation labels to the first and second modal data directly determines whether they are related. However, in scenarios where the first and second modal data are strongly correlated but not strictly matched, errors can easily occur.

[0079] Therefore, in this embodiment, the first modality data and the second modality data in the sample content are not directly labeled with a correlation degree of 0 or 1. Instead, the correlation degree between the first modality data and the second modality data in the sample content is predicted. Specifically, the distance between the second modality feature vector corresponding to the second modality data of the sample content and the center of all second modality feature vectors in the sample category corresponding to the sample content is calculated as the correlation degree between the second modality data and the first modality data.

[0080] Optionally, after determining the feature vector center corresponding to each sample category in the feature space, for the target sample content, since the target sample content is labeled with a classification label, the distance between the feature vector center corresponding to the classification label of the target sample content and the second modality feature vector corresponding to the second modality data in the target sample content is calculated, and this distance is used as the correlation label between the first modality data and the second modality data in the target sample content.

[0081] To illustrate, the second modality feature vector corresponding to the second modality data of the target sample content is T1, and the sample category to which this second modality feature vector belongs is "constellation". In the sample content set, there are a total of 3 sample contents belonging to "constellation", and the second modality feature vectors corresponding to the second modality data of these 3 sample contents are T1, T2, and T3.

[0082] The vectors T1, T2, and T3 are summed and averaged to obtain the feature vector center T for the "constellation" category. The cosine similarity between T1 and T is calculated, yielding a cosine similarity of 0.8. In some embodiments, this cosine similarity can also be represented as the distance between T1 and T. This cosine similarity is used as the correlation between the first modality data and the second modality data in the target sample content.

[0083] Optionally, the first modal data is text modal data and the second modal data is image modal data. In some embodiments, the first modal data and the second modal data can be data of any modality, but the first modal data and the second modal data are data of different modalities.

[0084] It is worth noting that the second modality feature vector corresponding to the second modality data in the sample content is a multi-dimensional vector. The distance between the second modality feature vector corresponding to the second modality data in the sample content and the center of the feature vector of the sample classification corresponding to the second modality data can be any value, that is, the correlation label can be any value. This embodiment does not limit this.

[0085] In this embodiment, after pre-screening, all sample content in the sample content set has a strong correlation between its first modality data and sample classification. Therefore, the correlation between the first modality data and the second modality data of the sample content is transformed into the correlation between the second modality data and the sample classification of the sample content. The second modality data and the sample classification are represented in vector form. The distance between the center of the second modality feature vector corresponding to the second modality data and the center of the feature vector of the sample classification corresponding to the second modality data is used as the correlation label between the second modality data and the first modality data in the sample content. The correlation is presented in numerical form, which more intuitively and concisely represents the relationship between the first modality data and the second modality data in the sample content.

[0086] Step 405: Train the candidate correlation recognition model based on the sample content and the correlation labels corresponding to the sample content to obtain the correlation recognition model.

[0087] Optionally, the correlation recognition model obtained after the above training process can be used to identify the correlation between the first modality data and the second modality data in the target content. It can also extract the semantic feature representation of the target content based on the correlation, wherein the semantic feature representation is used to characterize the semantics of the target content.

[0088] Optionally, the functions of semantic feature representation include, but are not limited to, the following:

[0089] 1. Input the semantic feature representation of the target content into the keyword extraction model to extract the keywords of the target content;

[0090] 2. Input the semantic feature representation of the target content into the classification model, classify the target content, and output the classification result of the target content. The classification model includes a preset classification set. The semantic feature representation of the target content is matched with the classification in the classification set to obtain the classification result corresponding to the target content.

[0091] 3. Input the semantic feature representation of the target content into the recommendation model and match it with user features to recommend target content that meets the matching requirements to the user.

[0092] The aforementioned target content refers to multimodal content that includes first modal data and second modal data.

[0093] Optionally, after obtaining the correlation label (i.e., a numerical value) based on the sample content and its corresponding correlation, this correlation label is used as information content of the target sample. The target sample is input into the candidate correlation recognition model, which contains first modality data and second modality data. The candidate correlation recognition model analyzes the predicted correlation between the first modality data and the second modality data in the target sample and outputs the predicted correlation. The predicted correlation and the aforementioned correlation label are input into a preset loss function to obtain a correlation loss value. This correlation loss value is returned to the correlation recognition model for training the model parameters. To illustrate, the correlation label between the first modality data and the second modality data in the target sample content S is 0.8. After inputting the target sample content S into the candidate correlation recognition model M, a predicted correlation between the first modality data and the second modality data in the target sample is output as 0.7. The correlation label and the predicted correlation are input into a preset loss function F(x). The correlation loss value is calculated by the loss function F(x) to be 0.1. The correlation loss value of 0.1 is returned to the candidate correlation recognition model M to continue subsequent training.

[0094] It is worth noting that the predicted correlation degree output by the candidate correlation degree identification model can be any value, and the preset loss function F(x) can be any function that meets the actual needs. The correlation degree loss value calculated by the loss function F(x) can also be any value. This embodiment does not limit this.

[0095] In summary, the method provided in this embodiment trains the candidate correlation recognition model using sample content and the corresponding correlation labels. The sample content in the sample content set is content labeled with classification labels, that is, the first modality content and the second modality content are both content associated with the classification labels. Therefore, using the sample content in the sample content set to train the candidate correlation recognition model can make the predicted correlation results output by the candidate correlation recognition model more accurate.

[0096] Furthermore, by calculating the loss value of the predicted correlation degree and the correlation degree label between the first modality data and the second modality data, and returning the loss value to the candidate correlation degree recognition model for updating and continuing training, the predicted correlation degree output by the candidate correlation degree recognition model can be more accurate and have smaller errors compared with the correlation degree label, thus realizing the continuous optimization process of the model.

[0097] In an optional embodiment, the candidate correlation identification model is trained based on the loss value calculated from the aforementioned correlation labels and predicted correlations. For example... Figure 5 As shown, step 405 above can also be implemented as the following steps:

[0098] Step 4051: Obtain the target sample content in the sample content set. The target sample content includes the target first modality data and the target second modality data.

[0099] In some embodiments, the sample content set includes multiple sample contents, and each sample content consists of multimodal data, including first modal data and second modal data. Optionally, the first modal data and the second modal data belong to two different modalities.

[0100] Optionally, the sample content in the above sample content set can be implemented as an information flow article, wherein the information flow article includes text content as the main text and image content as the accompanying images, wherein the text content is text modal data and the image content is image modal data.

[0101] Optionally, the sample content in the sample content set includes both first modality data and second modality data. In this embodiment, taking an information flow article corpus as an example, the information flow article corpus includes multiple information flow articles, and each information flow article includes text modality data as first modality data and image modality data as second modality data.

[0102] It is worth noting that the first and second modal data of the above sample content are merely illustrative examples, and this embodiment does not limit them.

[0103] Step 4052: Input the target sample content into the candidate correlation recognition model and output the predicted correlation between the target first modality data and the target second modality data.

[0104] The candidate correlation recognition model is a pre-defined model to be trained. It takes the first and second modal data of the target sample content as input and outputs a cosine similarity score. This cosine similarity score represents the similarity between the first and second modal data in the target sample content, serving as the predicted correlation between them. Specifically, cosine similarity measures the similarity between two feature vectors by measuring the cosine of the angle between them. The two feature vectors are the feature vectors extracted from the first modal data and the feature vectors extracted from the second modal data, respectively.

[0105] Optionally, after the first modality data and the second modality data of the sample content are transformed by the candidate correlation recognition model, they are presented in vector form, and the cosine similarity between the second modality feature vectors corresponding to the first modality data content and the second modality data is calculated. This cosine similarity is used as the output content of the candidate correlation recognition model, that is, the predicted correlation between the first modality data and the second modality data of the target sample content.

[0106] Indicatively, the first modality of the target sample content is the text content of "Capricorn Today's Horoscope," and the second modality is the image content of "Image 1." The candidate correlation recognition model includes a first feature extraction network to transform the image content of "Image 1" into a corresponding second modality feature vector "IMG1." The model also includes a second feature extraction network, such as a transformer network, to extract the feature vector of the first modality and calculate the cosine similarity between the first and second modality data of the target sample content. Specifically, this involves calculating the cosine similarity between the text content of "Capricorn Today's Horoscope" and the second modality data "Image 1." This cosine similarity is the output, representing the predicted correlation between the first and second modality data of the target sample content.

[0107] It is worth noting that, in the above examples, the target sample content includes a piece of text and an image. In some embodiments, the first modal data and the second modal data in the target sample content can be one or more, and this embodiment does not limit this. Furthermore, in the above examples, the first modal data in the target sample content is text and the second modal data is an image. In some embodiments, the first modal data can be any type of content such as text, image, or video, and the second modal data can also be any type of content such as text, image, or video. This embodiment does not limit this. However, the first modal data and the second modal data must be modal data of different types. If the first modal data or the second modal data is video content, then keyframe extraction is performed on the video content, i.e., screenshots are taken of the video keyframes, converted into image content, and operated according to the method of the above embodiment.

[0108] Step 4053: Based on the relevance labels and predicted relevance of the target sample content, obtain the relevance loss value, which is used to represent the difference between the relevance labels and the predicted relevance.

[0109] Optionally, the correlation loss value refers to the correlation label between the first modality data and the second modality data in the target sample content, and the predicted correlation between the first modality data and the second modality data output by the correlation recognition model, which are input into a preset loss function F(x) to obtain an output result, which is the correlation loss value.

[0110] To illustrate, the correlation label between the first modality data and the second modality data in the target sample content is 0.8. The predicted correlation between the first modality data and the second modality data output by the correlation recognition model is 0.7. After inputting the correlation label and the predicted correlation into the preset loss function F(x), the correlation loss value is 0.1.

[0111] It is worth noting that, in the above example, taking the correlation label between the first modality data and the second modality data as 0.8 and the predicted correlation between the first modality data and the second modality data output by the correlation recognition model as 0.7 as an example, in some embodiments, the above correlation label and predicted correlation can be any value, and this embodiment does not limit this. Furthermore, the correlation loss value obtained based on the above correlation label and predicted correlation can be any value, and the preset loss function can be any function that meets the actual needs; this embodiment does not limit this.

[0112] Step 4054: Train the candidate correlation recognition model based on the correlation loss value to obtain the correlation recognition model.

[0113] Optionally, training the candidate correlation identification model based on the correlation loss value means updating and iteratively training the model parameters in the candidate correlation identification model based on the correlation loss value. Optionally, iteratively updating and training the model parameters of the candidate correlation identification model is performed using the gradient descent method.

[0114] In a schematic manner, a correlation loss value is obtained based on the correlation labels and predicted correlation of the target sample content. If the correlation loss value does not meet the training requirements, the model parameters in the candidate correlation recognition model are adjusted based on the correlation loss value, and further iterative training is performed on the candidate correlation recognition model with adjusted parameters until the loss value corresponding to the predicted correlation between the first modality data and the second modality data of the output target sample content meets the training requirements. Optionally, meeting the training requirements for the loss value includes: 1. loss value convergence; 2. loss value less than a preset threshold; 3. the number of iterations reaches the required number, etc., which are not limited in this embodiment.

[0115] It is worth noting that the correlation label between the first modality data and the second modality data of the target sample content, and the predicted correlation between the first modality data and the second modality data obtained by the candidate correlation identification model of the target sample content, can be any value based on the correlation label and the predicted correlation. Furthermore, the predicted correlation in the above example converges to a certain value or a preset threshold. The convergence value and the threshold can be any value, and this embodiment does not limit them.

[0116] In summary, the method provided in this embodiment is based on the correlation labels between the first and second modal data in the target sample content and the predicted correlation between the first and second modal data output by the candidate correlation recognition model. A correlation loss value is calculated using a preset loss function, and this loss value is returned to the candidate correlation recognition model. Taking into full account the error between the model's prediction and the actual result, the process of returning the loss value to the candidate correlation recognition model involves parameter updates and iterative training, allowing the model to continuously optimize and reduce the error between the predicted correlation and the correlation labels. This further enhances the model's ability to understand multimodal target sample content, better models the target sample content, and thus improves the accuracy of overall information flow article content comprehension.

[0117] In an optional embodiment, the correlation recognition model involved in this application embodiment can also extract semantic feature representations of multimodal content, output semantic vectors of multimodal content, and apply them. Figure 6 This is a flowchart illustrating the extraction and application of semantic feature representations of multimodal content according to an embodiment of this application. The specific process is as follows:

[0118] Step 601: Obtain the semantic similarity distribution corresponding to the classification tag library based on the semantic similarity relationship between each classification tag in the classification tag library.

[0119] Optionally, the category tag library contains category tags corresponding to the multimodal content in the multimodal content set. Classification prediction is performed on the target multimodal content, and the probability of the target multimodal content appearing in each dimension and each category tag is statistically analyzed to determine the topic representation content corresponding to each category tag. The topic representation content is used to represent the implicit semantics of the topic corresponding to the category tag. The target multimodal content includes first-modal data and second-modal data.

[0120] To illustrate, we selected articles from the news feed and used the five articles with the highest probability of appearing under each category tag as the thematic representation of that category tag, as shown in Table 1 below.

[0121] Table 1

[0122]

[0123] The above five articles are the five most frequently appearing articles in the information flow under the category tag "2000-dimensional topic - topic 1 oral health", and are used as the thematic representation content of the category tag "2000-dimensional topic - topic 1 oral health".

[0124] In this context, "2000-dimensional topic" refers to a topic at a 2000-dimensional granularity. The content of the category tags can be topic content at any granularity, and this embodiment does not impose any limitations on this. The topic representation content under each category tag corresponds to the category tag. In the example above, using "2000-dimensional topic - Topic 1 Oral Health" as the category tag, it can be seen from the title of its corresponding topic representation content that the topic representation content revolves around the content of the category tag "2000-dimensional topic - Topic 1 Oral Health". The titles of the topic representation content mentioned above are merely examples and can be any information flow article content corresponding to the category tag "2000-dimensional topic - Topic 1 Oral Health". This embodiment does not impose any limitations on this.

[0125] The semantic feature vectors of the topic representation content are averaged to obtain the semantic vectors of the classification labels corresponding to the classification labels. Specifically, the semantic feature vector of the topic representation content refers to the semantic feature vector output after the multimodal content of the topic representation content is input into the aforementioned association recognition model and transformed by the model.

[0126] The cosine similarity between the semantic vectors of each category tag is obtained to get the semantic similarity distribution corresponding to the category tag library. Specifically, the semantic vector of a category tag is obtained by weighting the semantic feature vectors representing the content of each topic under each category tag, which is represented by the average of the semantic feature vectors.

[0127] Schematic representation: The semantic feature vector of the above topic representation content is as follows:

[0128] X1(1,2,3), X2(1,1,1), X3(3,4,5), X4(1,1,2), X5(4,2,9); then the semantic vector of the classification label X is (X1+X2+X3+X4+X5) / 5=(2,2,4).

[0129] By calculating the cosine similarity between the semantic vectors of the classification labels in different dimensions, the semantic similarity distribution of each classification label can be obtained.

[0130] To illustrate, the cosine similarity between the semantic vector of the category label "2000-dimensional topic - topic 1 oral health" and the semantic vectors of the category labels "5000-dimensional topic - topic 1 dental caries treatment", "5000-dimensional topic - topic 2 dental knowledge", and "5000-dimensional topic - topic 3 King of Glory mage equipment" is calculated. The cosine similarity between different category label semantic vectors is used as the semantic similarity, and the corresponding semantic similarity distribution is obtained by combining them, as shown in Table 2 below.

[0131] Table 2

[0132] 2000-dimensional theme 5000-dimensional theme semantic similarity Theme 1 Oral Health Topic 1 Dental Health 0.92 Theme 1 Oral Health Topic 2: Knowledge about teeth 0.85 Theme 1 Oral Health Theme 3: King of Glory Mage Equipment Build 0.60

[0133] As shown in Table 2, the semantic similarity between "2000-dimensional topic - topic 1 oral health" and "5000-dimensional topic - topic 1 dental caries treatment" is 0.92, the semantic similarity between "2000-dimensional topic - topic 2 dental knowledge" is 0.85, and the semantic similarity between "5000-dimensional topic - topic 3 King of Glory mage build" is 0.60.

[0134] It is worth noting that in the examples above, the topic representation content is selected from the five most frequently occurring articles in the news feed under the category tag. In some embodiments, the topic representation content can be any number of target multimodal content that appears most frequently under the category tag; this embodiment does not limit this. Furthermore, in the examples above, both the semantic feature vector of the topic representation content and the semantic vector of the category tag are three-dimensional feature vectors, with each element being a positive integer. In some embodiments, the semantic feature vector of the topic representation content and the semantic vector of the category tag can be any multidimensional vector; this embodiment does not limit this. The semantic similarity distribution in Table 2 above is merely an example; the semantic similarity distribution between different category tags can be any value, and this embodiment does not limit this.

[0135] In this embodiment, the aforementioned association recognition model is applied to obtain the semantic similarity distribution corresponding to the classification tag library based on the semantic similarity relationship between each classification tag in the classification tag library. By inputting multimodal content into the association recognition model, the output is a semantic vector that represents the complete semantics of the multimodal content, improving the understanding of multimodal content and the accuracy of the semantic vector. It is not limited to inputting only one modality of data, such as only inputting text modality data.

[0136] Step 602: Based on the semantic features extracted from the target multimodal content using the correlation recognition model, obtain the probability distribution of the category label content corresponding to the target multimodal content. The probability distribution of the category label content is used to indicate the semantic association between the semantic features of the target multimodal content and each category label.

[0137] Optionally, the co-occurrence probability of classification tags at different granularities is calculated to obtain the probability distribution of classification tag content. The co-occurrence probability refers to the probability that the target multimodal content under the first granularity classification tag appears under the second granularity classification tag.

[0138] Granularity refers to the level of detail in data statistics within the same dimension. Both the first and second granularities serve to assist in the division of information flow article category tags. The first and second granularities can be arbitrary, but they must be different. This embodiment does not impose any limitations on this.

[0139] To illustrate, taking a first granularity of 2000 dimensions and a second granularity of 5000 dimensions as an example, Table 3 below is a table of the probability distribution of example category label content.

[0140] Table 3

[0141] 2000-dimensional theme 5000-dimensional theme Co-occurrence probability Theme 1 Oral Health Topic 1: Treatment of Dental Caries 0.60 Theme 1 Oral Health Topic 2: Knowledge about teeth 0.30 Theme 1 Oral Health Theme 3: Mage Item Build 0

[0142] Among them, there are 10,000 articles with the category label "Topic 1 Oral Health" under the 2000-dimensional topic, and 6,000 articles with the category label "Topic 1 Dental Caries Treatment" under the 5000-dimensional topic. The co-occurrence probability of the category label is 0.60.

[0143] Similarly, among the 10,000 articles categorized under the 2000-dimensional topic with the label "Topic 1 Oral Health", there are 3,000 articles categorized under the 5000-dimensional topic with the label "Topic 2 Dental Knowledge". Therefore, the co-occurrence probability of the category labels is 0.30.

[0144] If there are 10,000 articles categorized under the 2000-dimensional topic with the tag "Topic 1 Oral Health", and 0 articles categorized under the 5000-dimensional topic with the tag "Topic 3 Game Mage Equipment", then the co-occurrence probability of the category tags is 0. Combining these co-occurrence probabilities, we obtain the probability distribution of the category tag content.

[0145] It is worth noting that the above example uses 10,000 news feed articles as an example of multimodal content. In some embodiments, multimodal content can be in forms other than news feed articles, such as social content on social platforms, and the number of multimodal content can be arbitrary, but must be greater than or equal to 2. This embodiment does not limit this.

[0146] In this embodiment, the semantic features extracted from the target multimodal content based on the correlation recognition model are used as the probability distribution of the category label content by taking the co-occurrence probability of the target multimodal content in each category label. This further improves the semantic understanding of the multimodal content.

[0147] Step 603: Fuse the semantic similarity distribution and the probability distribution of the classification tag content to obtain the multi-level classification sub-tags corresponding to the target multimodal content.

[0148] Optionally, multi-level category sub-tags refer to category tags obtained by further dividing the existing category tags of the target multimodal content into levels.

[0149] The probability distribution of category label content and the semantic similarity distribution are weighted and fused to obtain the hierarchical category label probability distribution; wherein, when the value corresponding to the hierarchical category label probability distribution meets the preset probability, the multi-level category sub-labels corresponding to the target multimodal content have a hierarchical relationship.

[0150] The weighted fusion of the probability distribution of category tag content and the semantic similarity distribution refers to the normalization of the semantic similarity of category tags after weighting them using the co-occurrence probability of category tags.

[0151] It is worth noting that in the above embodiments, the target multimodal content corresponds to three category tags. In some embodiments, the number of category tags corresponding to the target multimodal content can be arbitrary. If there is only one category tag, there is no hierarchical relationship. This embodiment does not limit this. Furthermore, the probability distribution and semantic similarity distribution of the category tag content of the target multimodal content can be arbitrary values. This embodiment does not limit this.

[0152] In this embodiment, the multi-level sub-labels corresponding to the classification labels are obtained by the above method. These sub-labels serve as important identifiers of the target multimodal content, highly summarizing the semantic features of the target multimodal content. This can enhance the semantic understanding of the target multimodal content and improve the overall grasp of the multimodal content.

[0153] Figure 7 This is a flowchart of a modal correlation prediction method provided in an exemplary embodiment of this application. In this embodiment, the classification labels of the sample content include multi-level classification sub-labels, such as... Figure 7 As shown above, Figure 4 Before step 401 shown, the following steps are also included:

[0154] Step 701: Obtain the semantic similarity distribution corresponding to the classification label library based on the semantic similarity relationship between each classification label in the classification label library.

[0155] Optionally, the category tag library contains category tags corresponding to the multimodal content in the multimodal content set. The content in the multimodal content set is classified and predicted, and the probability of the target multimodal content appearing on each category tag is calculated. Multimodal content with a probability greater than a preset threshold under each category tag is used as the topic representation content corresponding to that category tag.

[0156] The topic representation content is input into a preset feature extraction model to extract the semantic feature vector corresponding to each topic representation content. Specifically, the semantic feature vector of the topic representation content refers to the semantic feature vector output after the title and body content of the topic representation content are input into the preset feature extraction model (e.g., the BERT model) and transformed by the model.

[0157] The semantic feature vectors of the topic representation content are averaged to obtain the category label semantic vectors corresponding to the category labels. The cosine similarity between the category label semantic vectors of each category label is then obtained to obtain the semantic similarity distribution corresponding to the category label library. Specifically, the category label semantic vector is obtained by weighting the semantic feature vectors of each topic representation content under each category label, which is represented by averaging the semantic feature vectors.

[0158] As an example, consider an article titled "Health Science Popularization, Children's Oral Health" from an information feed. This article, as the target multimodal content, is input into the feature extraction model described above. The information feed article includes text modal data. For example... Figure 8 As shown, firstly, the text modal data 810 of the news feed article is input. In this feature extraction model, there is a transformation network 820 that transforms the text modal data 810 into vectors of different dimensions: token embeddings, segment embeddings, and position embeddings. These vectors are then processed by a transformer network 830 to obtain a semantic representation vector 840, which represents the overall content of the news feed article. In other words, inputting the text modal content from the multimodal content into this feature extraction model will output a semantic representation vector.

[0159] It is worth noting that the semantic vectors of the above-mentioned topic representation articles are obtained by feature extraction using a preset BERT model. In some embodiments, the preset feature extraction model can be other models that meet actual needs, and this embodiment does not limit this.

[0160] In this embodiment, the semantic similarity distribution of the category tag library is obtained based on the semantic similarity relationship between the category tags in the category tag library. First, the topic representation articles of the category tags contained in the category tag library are obtained and represented in vector form. Further, vectors representing each category tag are obtained, and the cosine similarity between them is calculated, thus obtaining the semantic similarity relationship between the category tags and forming a semantic similarity distribution. This transforms the abstract similarity relationship into vector and numerical forms, making the relationship between each category tag intuitively clear and facilitating further hierarchical division of the category tags.

[0161] Step 702: Obtain the probability distribution of the category label content corresponding to the target multimodal content. The probability distribution of the category label content is used to indicate the semantic relationship between the semantic features of the target multimodal content and each category label.

[0162] Optionally, the target multimodal content is classified and predicted, and the classification labels that correspond to the target multimodal content and whose probability distributions meet the probability requirements are retained. Meeting the probability requirements means that after the target multimodal content is predicted by the preset model, the probability value obtained for each classification label reaches a pre-preset threshold.

[0163] To illustrate, after the target multimodal content is predicted by the preset model, the probability value obtained on the classification label Q1 is 0.6, and the probability value obtained on the classification label Q2 is 0.1. The preset threshold is 0.5. The probability value obtained on the classification label Q1 meets the probability requirement, while the probability value obtained on the classification label Q2 does not meet the probability requirement. Therefore, the target multimodal content belongs to the classification label Q1. The above process completes the classification prediction of the target multimodal content.

[0164] It is worth noting that in the above examples, 0.5 is used as the preset threshold. In some embodiments, the preset threshold can be any non-negative value, but it must be less than or equal to one and greater than or equal to zero. This embodiment does not limit this. Furthermore, the above examples only show two classification labels for classification prediction. In some embodiments, any number of classification labels can be used for classification prediction. This embodiment does not limit this.

[0165] The probability distribution of category label content is obtained by calculating the co-occurrence probability of category labels at different granularities. The co-occurrence probability refers to the probability that the target multimodal content under the first granularity category label appears under the second granularity category label.

[0166] In this embodiment, the frequency of multimodal content appearing at the second granularity under the first granularity is used as the co-occurrence probability, which is used as the probability distribution of the classification label content. This further indicates the semantic relationship between the semantic features of the target multimodal content and each classification label, which facilitates the hierarchical division of the classification labels corresponding to the target multimodal content and enhances the semantic understanding of the multimodal content.

[0167] Step 703: Fuse the semantic similarity distribution and the probability distribution of the classification label content to obtain the multi-level classification sub-labels corresponding to the target multimodal content.

[0168] The semantic similarity of the category tags is weighted and normalized using the co-occurrence probability of the category tags to obtain the hierarchical category tag probability distribution. When the value corresponding to the hierarchical category tag probability distribution meets the preset probability, multi-level category sub-tags with hierarchical relationships in the target multimodal content are obtained.

[0169] It is worth noting that the number and type of multi-level category sub-labels can be arbitrary, and this embodiment does not limit this.

[0170] Step 704: Based on the multi-level classification sub-labels corresponding to the target multimodal content, construct a sample content set by using the target multimodal content as sample content.

[0171] Optionally, the sample content set consisting of the aforementioned target multimodal content includes, but is not limited to, first modality data, second modality data, classification labels of multimodal content, and multi-level classification sub-labels corresponding to multimodal content.

[0172] Indicatively, the sample content set consisting of the target multimodal content includes multiple news feed articles. Each news feed article includes: the content of the first modality data, which is text modality data; the content of the second modality data, which is image modality data; the classification label corresponding to the content of the news feed article; and the multi-level classification sub-labels based on the content of the news feed article.

[0173] It is worth noting that in the above examples, the sample content set composed of target multimodal content uses news feed articles as sample multimodal content, and each sample multimodal content includes two modalities, corresponding category tags, and multi-level category sub-tags. In some embodiments, the target multimodal content can be any form of content, such as articles, videos, social content published on social platforms, etc., and each sample multimodal content can have any type of modal data; this embodiment does not limit this. The first modal data and the second modal data can also be any form of modal data, such as text modal data, image modal data, video modal data, etc.; this embodiment does not limit this. However, the types of the first modal data and the second modal data must be different. The category tags and multi-level category sub-tags included in the target multimodal content can also be of any type; this embodiment does not limit this.

[0174] In summary, the method provided in this embodiment, based on the semantic similarity distribution and co-occurrence probability distribution of classification tags, completes the construction of hierarchical relationships between classification tags at different granularities. It decouples the classification tag construction process, is applicable to common classification tag methods, and effectively ensures semantic consistency and multimodal content distribution consistency among hierarchical classification tags. The hierarchical classification tags constructed using this method not only achieve very high offline evaluation accuracy but also show a significant improvement in online classification tag retrieval performance through hierarchical classification tag relationships.

[0175] Figure 9 This is a structural block diagram of a modal correlation prediction device provided in an exemplary embodiment of this application, as shown below. Figure 9 As shown, the device includes:

[0176] The acquisition module 910 is used to acquire a sample content set, wherein the sample content in the sample content set includes first modality data and second modality data, and the sample content is labeled with a classification label, which is used to indicate the sample category to which the sample content belongs;

[0177] Extraction module 920 is used to extract the second modality feature vector corresponding to the second modality data of the sample content;

[0178] The determination module 930 is used to determine the feature vector centers corresponding to multiple sample categories based on the second modality feature vectors corresponding to sample content belonging to the same sample category; and to determine the distance between the second modality feature vector corresponding to the second modality data and the feature vector center of the sample category corresponding to the second modality data, as a correlation label between the second modality data and the first modality data in the sample content.

[0179] The training module 940 is used to train the candidate correlation recognition model based on the sample content and the correlation label corresponding to the sample content to obtain the correlation recognition model. The correlation recognition model is used to identify the correlation between the first modality data and the second modality data in the target content, and to extract the semantic feature representation of the target content based on the correlation. The semantic feature representation is used to characterize the semantics of the target content.

[0180] In an optional embodiment, the acquisition module 910 is further configured to acquire target sample content in the sample content set, the target sample content including target first modality data and target second modality data; input the target sample content into the candidate correlation recognition model, and output the predicted correlation between the target first modality data and the target second modality data;

[0181] The training module 940 is further configured to obtain a correlation loss value based on the correlation label and predicted correlation of the target sample content, wherein the correlation loss value is used to represent the difference between the correlation label and the predicted correlation; and to train the candidate correlation recognition model based on the correlation loss value to obtain the correlation recognition model.

[0182] In an optional embodiment, the training module 940 is further configured to iteratively train the candidate correlation recognition model based on the correlation loss values ​​corresponding to the sample content in the sample content set, so as to obtain the correlation recognition model.

[0183] In an optional embodiment, the determining module 930 is further configured to perform average processing on the second modal feature vectors corresponding to the sample content belonging to the same sample category, so as to obtain the feature vector centers corresponding to multiple sample categories respectively.

[0184] In an optional embodiment, the acquisition module 910 is further configured to acquire the semantic similarity distribution corresponding to the classification tag library based on the semantic similarity relationship between each classification tag in the classification tag library;

[0185] The acquisition module 910 is further configured to obtain the probability distribution of the category tag content corresponding to the target multimodal content based on the semantic features extracted from the target multimodal content by the correlation recognition model, and the probability distribution of the category tag content is used to indicate the semantic association relationship between the semantic features of the target multimodal content and each category tag;

[0186] like Figure 10 As shown, the device also includes:

[0187] The fusion module 950 is used to fuse the semantic similarity distribution and the probability distribution of the classification label content to obtain the multi-level classification sub-labels corresponding to the target multimodal content.

[0188] In an optional embodiment, the acquisition module 910 is further configured to perform classification prediction on the target multimodal content, determine the topic representation content corresponding to each category label, the topic representation content being used to represent the implicit semantics of the topic corresponding to the category label; take the average value of the semantic feature vector of the topic representation content to obtain the category label semantic vector corresponding to the category label; and obtain the cosine similarity between the category label semantic vectors of each category label to obtain the semantic similarity distribution.

[0189] In an optional embodiment, the acquisition module 910 is further configured to perform classification prediction on the target multimodal content, retain classification labels whose probability distributions corresponding to the target multimodal content meet the probability requirements; calculate the co-occurrence probability of classification labels at different granularities to obtain the probability distribution of the classification label content, wherein the co-occurrence probability refers to the probability that content under the first granularity classification label appears under the second granularity classification label.

[0190] In an optional embodiment, the fusion module 950 is further configured to perform weighted fusion of the probability distribution of the classification tag content and the semantic similarity distribution to obtain a hierarchical classification tag probability distribution; wherein, when the value corresponding to the hierarchical classification tag probability distribution is higher than a preset threshold, the multi-level classification sub-tags corresponding to the target multimodal content have a hierarchical relationship.

[0191] In summary, the device provided in this embodiment trains the candidate correlation recognition model using sample content and the correlation labels corresponding to the sample content. The sample content in the sample content set is content labeled with classification labels, that is, the first modality content and the second modality content are both content associated with the classification labels. Therefore, using the sample content in the sample content set to train the candidate correlation recognition model can make the predicted correlation results output by the candidate correlation recognition model more accurate.

[0192] It should be noted that the modal correlation prediction device provided in the above embodiments is only an example of the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0193] Figure 11This illustration shows a structural block diagram of a computer device 1100 provided in an exemplary embodiment of this application. The computer device 1100 may be a smartphone, tablet computer, MP3 player (Moving Picture Experts Group Audio Layer III), MP4 player (Moving Picture Experts Group Audio Layer IV), laptop computer, or desktop computer. The computer device 1100 may also be referred to as a user device, portable terminal, laptop terminal, desktop terminal, or other names.

[0194] Typically, computer device 1100 includes a processor 1101 and a memory 1102.

[0195] Processor 1101 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1101 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1101 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1101 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 1101 may also include an AI processor for handling computational operations related to machine learning.

[0196] The memory 1102 may include one or more computer-readable storage media, which may be non-transitory. The memory 1102 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1102 are used to store at least one instruction, which is executed by the processor 1101 to implement the virtual object interaction method provided in the method embodiments of this application.

[0197] In some embodiments, the computer device 1100 also includes other components, as those skilled in the art will understand. Figure 11 The structure shown does not constitute a limitation on terminal 1100 and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0198] Embodiments of this application also provide a computer device, which includes a processor and a memory. The memory stores at least one instruction, at least one program, code set, or instruction set. The at least one instruction, at least one program, code set, or instruction set is loaded and executed by the processor to implement the modal correlation prediction method provided in the above-described method embodiments.

[0199] Embodiments of this application also provide a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the modal correlation prediction method provided in the above-described method embodiments.

[0200] Embodiments of this application also provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform any of the modal correlation prediction methods described in the above embodiments.

[0201] Optionally, the computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), solid-state drives (SSDs), or optical discs, etc. The random access memory may include resistive random access memory (ReRAM) and dynamic random access memory (DRAM). The sequence numbers of the embodiments in this application are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0202] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0203] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A modal correlation prediction method, characterized in that, The method includes: Obtain a sample content set, wherein the sample content in the sample content set includes first modality data and second modality data, and the sample content is labeled with a classification label, wherein the classification label is used to indicate the sample category to which the sample content belongs, and the first modality data in the sample content is strongly correlated with the sample category; Extract the second modality feature vector corresponding to the second modality data of the sample content; Based on the second modal feature vectors corresponding to the sample content belonging to the same sample category, the feature vector centers corresponding to multiple sample categories are determined respectively. The distance between the second modality feature vector corresponding to the second modality data and the feature vector center of the sample classification corresponding to the second modality data is determined and used as the correlation label between the second modality data and the first modality data in the sample content; The candidate correlation recognition model is trained based on the sample content and the corresponding correlation labels to obtain the correlation recognition model. The correlation recognition model is used to identify the correlation between the first modality data and the second modality data in the target content, and extract the semantic feature representation of the target content based on the correlation. Specifically, the second modality feature vectors corresponding to the second modality data in each sample content are extracted, and the second modality feature vectors in the feature space are clustered according to the classification labels to obtain the feature vector center corresponding to each category. The correlation between the first modality data and the second modality data in the target content is determined based on the distance between the second modality feature vector of the second modality data of the target content and the feature vector center corresponding to the classification label of the target content. The semantic feature representation is used to characterize the semantics of the target content.

2. The method according to claim 1, characterized in that, The step of training the candidate correlation recognition model based on the sample content and the corresponding correlation labels to obtain the correlation recognition model includes: Obtain the target sample content from the sample content set, wherein the target sample content includes target first modality data and target second modality data; The target sample content is input into the candidate correlation recognition model, and the predicted correlation between the first modality data of the target and the second modality data of the target is output. Based on the relevance labels and predicted relevance of the target sample content, a relevance loss value is obtained, which is used to represent the difference between the relevance labels and the predicted relevance. The candidate correlation recognition model is trained based on the correlation loss value to obtain the correlation recognition model.

3. The method according to claim 2, characterized in that, The step of training the candidate correlation recognition model based on the correlation loss value to obtain the correlation recognition model includes: The candidate correlation recognition model is iteratively trained based on the correlation loss values ​​corresponding to the sample content in the sample content set to obtain the correlation recognition model.

4. The method according to any one of claims 1 to 3, characterized in that, The determination of feature vector centers corresponding to multiple sample categories based on the second modality feature vectors corresponding to sample content belonging to the same sample category includes: The second modality feature vectors corresponding to the content of samples belonging to the same sample category are averaged to obtain the feature vector centers corresponding to multiple sample categories.

5. The method according to any one of claims 1 to 3, characterized in that, After training the candidate correlation recognition model based on the sample content and the corresponding correlation labels to obtain the correlation recognition model, the method further includes: The semantic similarity distribution corresponding to the classification tag library is obtained based on the semantic similarity relationship between each classification tag in the classification tag library; Based on the semantic features extracted from the target multimodal content by the correlation recognition model, the probability distribution of the category tag content corresponding to the target multimodal content is obtained. The probability distribution of the category tag content is used to indicate the semantic association between the semantic features of the target multimodal content and each category tag. By fusing the semantic similarity distribution and the probability distribution of the classification label content, multi-level classification sub-labels corresponding to the target multimodal content are obtained.

6. The method according to claim 5, characterized in that, The step of obtaining the semantic similarity distribution corresponding to the classification tag library based on the semantic similarity relationship between each classification tag in the classification tag library includes: The target multimodal content is classified and predicted to determine the topic representation content corresponding to each classification label. The topic representation content is used to represent the implicit semantics of the topic corresponding to the classification label. The semantic feature vectors representing the content of the topic are averaged to obtain the semantic vector of the classification label corresponding to the classification label; Obtain the cosine similarity between the semantic vectors of each category label to obtain the semantic similarity distribution.

7. The method according to claim 5, characterized in that, The acquisition of the probability distribution of the category tag content corresponding to the target multimodal content includes: The target multimodal content is classified and predicted, and the classification labels that meet the probability requirements are retained according to the probability distribution corresponding to the target multimodal content. The probability distribution of the category tag content is obtained by calculating the co-occurrence probability of the category tags at different granularities. The co-occurrence probability refers to the probability that the content under the first granularity category tag appears under the second granularity category tag.

8. The method according to claim 5, characterized in that, The process of fusing the semantic similarity distribution and the classification tag content probability distribution to obtain multi-level classification sub-tags corresponding to the target multimodal content includes: The probability distribution of the category label content and the semantic similarity distribution are weighted and fused to obtain the hierarchical category label probability distribution; wherein, when the value corresponding to the hierarchical category label probability distribution is higher than a preset threshold, the multi-level category sub-labels corresponding to the target multimodal content have a hierarchical relationship.

9. A modal correlation prediction device, characterized in that, The device includes: An acquisition module is used to acquire a sample content set, wherein the sample content in the sample content set includes first modality data and second modality data, and the sample content is labeled with a classification label, wherein the classification label is used to indicate the sample category to which the sample content belongs, and the first modality data in the sample content is strongly correlated with the sample category; The extraction module is used to extract the second modality feature vector corresponding to the second modality data of the sample content; The determination module is used to determine the feature vector centers corresponding to multiple sample categories based on the second modality feature vectors corresponding to sample content belonging to the same sample category; and to determine the distance between the second modality feature vector corresponding to the second modality data and the feature vector center of the sample category corresponding to the second modality data, as a correlation label between the second modality data and the first modality data in the sample content. The training module is used to train a candidate correlation recognition model based on the sample content and the correlation labels corresponding to the sample content, thereby obtaining a correlation recognition model. The correlation recognition model is used to identify the correlation between the first modality data and the second modality data in the target content, and to extract the semantic feature representation of the target content based on the correlation. Specifically, the second modality feature vectors corresponding to the second modality data in each sample content are extracted, and the second modality feature vectors in the feature space are clustered according to the classification labels to obtain the feature vector center corresponding to each classification. The correlation between the first modality data and the second modality data in the target content is determined based on the distance between the second modality feature vector of the second modality data of the target content and the feature vector center corresponding to the classification label of the target content. The semantic feature representation is used to characterize the semantics of the target content.

10. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one instruction, at least one program, a code set, or an instruction set, the at least one instruction, the at least one program, the code set, or the instruction set being loaded and executed by the processor to implement the modal correlation prediction method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The storage medium stores at least one program segment, which is loaded and executed by a processor to implement the modal correlation prediction method as described in any one of claims 1 to 8.

12. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the modal correlation prediction method as described in any one of claims 1 to 8.

Images