Method, device, equipment, medium and program product for training neural network model

Abstract

The application provides a neural network model training method and device, equipment, medium and program product; wherein, the method comprises: obtaining the characteristics of multiple modalities of an image; performing fusion processing based on the image characteristics and title text characteristics to obtain first fusion characteristics of cross modalities; calling a first neural network model based on the characteristics of multiple modalities to perform multiple single-modality prediction tasks to obtain corresponding single-modality prediction results and determine corresponding single-modality loss; calling the first neural network model based on the first fusion characteristics to perform multiple cross-modality prediction tasks to obtain corresponding cross-modality prediction results and determine corresponding cross-modality loss; and performing back propagation based on the single-modality loss and the cross-modality loss to update the parameters of the first neural network model. Through the application, one model can be used to perform multiple different modality tasks, thereby improving modeling efficiency.

Description

Technical Field

[0001] This application relates to artificial intelligence technology, and more particularly to a method, apparatus, device, medium, and program product for training a neural network model. Background Technology

[0002] Artificial Intelligence (AI) is a comprehensive technology within computer science that studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities. AI technology is a multidisciplinary field encompassing a wide range of areas, including natural language processing and machine learning / deep learning. With technological advancements, AI will be applied in more fields and play an increasingly important role.

[0003] In related technologies, the iterative process of information flow content processing algorithm models typically includes the following steps: requirement determination, data collection, model training, model testing, and deployment. During model training, existing pre-trained models are usually used for fine-tuning for the corresponding tasks. Since different tasks correspond to different types of labeled sample data, it is necessary to collect a large amount of labeled sample data for each type of task and then train them separately, resulting in low modeling efficiency and excessive costs. Summary of the Invention

[0004] This application provides a method, apparatus, electronic device, computer-readable storage medium, and computer program product for training a neural network model. By performing training tasks of various modalities on the neural network model, the neural network model can be applied to various different scenarios, improving modeling efficiency and reducing modeling costs.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides a method for training a neural network model, including:

[0007] The features of multiple modalities of an image are obtained; wherein the features of the multiple modalities include image features, title text features, and label text features;

[0008] Based on the image features and the title text features, a first cross-modal fusion feature is obtained;

[0009] Based on the features of the multiple modalities, the first neural network model is invoked to perform multiple single-modal prediction tasks, obtain the corresponding single-modal prediction results, and determine the corresponding single-modal loss based on the prediction results in each modality.

[0010] Based on the first fusion feature, the first neural network model is invoked to perform multiple cross-modal prediction tasks to obtain the corresponding cross-modal prediction results, and the corresponding cross-modal loss is determined based on each cross-modal prediction result.

[0011] The parameters of the first neural network model are updated by backpropagating based on the single-modal loss and the cross-modal loss.

[0012] The above methods also include:

[0013] The second neural network model is invoked to perform the following processing:

[0014] The image is divided into blocks, and linear projection is performed on the resulting multiple image blocks to obtain multiple corresponding linear projection features.

[0015] Each linear projection feature is summed with its corresponding positional feature, and self-attention calculation is performed based on the summation result to obtain the first self-attention feature corresponding to each linear projection feature.

[0016] Fully connected computation is performed based on multiple first self-attention features to obtain the first image features corresponding to the image.

[0017] The above methods also include:

[0018] The third neural network model is invoked to perform the following processing:

[0019] The word segmentation results of the tag include multiple word segments, which are then encoded to obtain the context features corresponding to each word segment; wherein, the encoding process includes multiple cascaded encoding processes;

[0020] The context features corresponding to multiple word segments are determined as the tag text features.

[0021] The above methods also include:

[0022] The multiple cross-modal prediction tasks include: a cross-modal similarity prediction task between the image and the title, a cross-modal matching prediction task between the image and the title, a cross-modal matching prediction task between the image, the title, and the tags, a cross-modal multi-tag prediction task between the image and the title, and a cross-modal masking language prediction task between the image and the title.

[0023] The above methods also include:

[0024] The image features included in each combination are concatenated with the title text features to obtain the first concatenated feature;

[0025] The first splicing feature is downsampled to obtain the first fusion feature corresponding to each combination.

[0026] The above methods also include:

[0027] The first neural network model is invoked to perform the following processing:

[0028] Based on the first fusion feature, self-attention calculation processing is performed to obtain the second self-attention feature;

[0029] Based on the second self-attention feature, a fully connected computation is performed to obtain the corresponding fully connected feature;

[0030] The fully connected features are normalized to obtain the corresponding prediction results for the multiple cross-modal features.

[0031] This application provides a training apparatus for a neural network model, comprising:

[0032] The acquisition module is used to acquire features of multiple modalities of an image; wherein the features of the multiple modalities include image features, title text features, and label text features;

[0033] The fusion module is used to perform fusion processing based on the image features and the title text features to obtain a first fusion feature across modalities;

[0034] The first determining module is used to call the first neural network model based on the features of the multiple modalities to perform multiple single-modal prediction tasks, obtain the corresponding single-modal prediction results, and determine the corresponding single-modal loss based on the prediction results in each modality.

[0035] The second determining module is used to call the first neural network model based on the first fusion feature to perform multiple cross-modal prediction tasks, obtain the corresponding cross-modal prediction results, and determine the corresponding cross-modal loss based on each cross-modal prediction result.

[0036] An update module is used to perform backpropagation based on the single-modal loss and the cross-modal loss to update the parameters of the first neural network model.

[0037] This application provides an electronic device, including:

[0038] Memory, used to store executable instructions;

[0039] The processor, when executing executable instructions stored in the memory, implements the training method of the neural network model provided in the embodiments of this application.

[0040] This application provides a computer-readable storage medium storing executable instructions for implementing the neural network model training method provided in this application when executed by a processor.

[0041] This application provides a computer program product including computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the neural network model training method described above in this application.

[0042] The embodiments of this application have the following beneficial effects:

[0043] By training the first neural network model with various modalities and updating its parameters based on the loss values ​​corresponding to these different training tasks, the first neural network model can acquire sufficient learning capabilities. This allows the trained model to support a variety of application scenarios, thereby expanding its applicability, effectively improving its modeling efficiency, and reducing its modeling costs. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the architecture of the neural network model training system 100 provided in the embodiments of this application;

[0045] Figure 2 This is a schematic diagram of the structure of the server 200 provided in the embodiments of this application;

[0046] Figures 3A-3D This is a flowchart illustrating the training method for the neural network model provided in an embodiment of this application;

[0047] Figure 4 This is a schematic diagram of the structure of the neural network model provided in the embodiments of this application;

[0048] Figure 5 This is a schematic diagram of the training method for the neural network model provided in the embodiments of this application;

[0049] Figure 6 A comparative schematic diagram of the loss function curves provided in the embodiments of this application;

[0050] Figure 7 This is a schematic diagram of the shielding process provided in the embodiments of this application. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0052] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0053] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0055] It is understood that in the embodiments of this application, data such as user information are involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0056] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0057] 1) Machine Learning (ML): This is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, and many other disciplines. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance.

[0058] 2) Deep Learning (DL): Originating from research on artificial neural networks, it uses multilayer perceptrons with multiple hidden layers to combine low-level features to form more abstract high-level representations of attribute categories or features, thereby discovering distributed feature representations of data. It can learn the inherent patterns and representation levels of sample data, with the ultimate goal of enabling machines to have analytical and learning capabilities like humans, and to recognize data such as text, images, and sound.

[0059] 3) Natural Language Processing (NLP): This is an important area within computer science and artificial intelligence. It studies various theories and methods that enable effective communication between humans and computers using natural language, and is a science that integrates linguistics, computer science, and mathematics.

[0060] 4) Loss Function: One of the two essential elements for compiling a neural network model. The loss function is a function that calculates the difference between the label value and the predicted value. In machine learning, various loss functions are available, typically distance vectors and absolute value vectors. The loss function must be a scalar because vectors cannot be compared in magnitude (vectors themselves need to be compared using scalars such as norms).

[0061] 5) Feeds: Also known as news sources, these are continuously updated content streams presented to users. Feeds aggregate content from several news sources that users actively subscribe to, helping them continuously access the latest content from these sources. These sources are typically news websites and blogs. Feeds can be displayed in various ways, primarily timelines and ranks. A timeline displays content in the order it was updated; a rank assigns weights to content based on certain factors, determining the order in which it is displayed.

[0062] 6) Content: The information recommended by the client to users can be videos, pictures, text, and graphic content. Graphic content can be actively edited and published by self-media, including vertical or horizontal short graphic content. Videos can be professionally produced content (PGC) or user-generated content (UGC) provided in the form of feeds.

[0063] 7) Short video: also known as short video, is a way of disseminating internet content. It generally refers to video content with a duration of less than 5 minutes that is disseminated on internet media.

[0064] 8) Model fine-tuning: This refers to adding a shallow model (such as a classifier model) based on the output of the last layer of a pre-trained model, and then training the pre-trained model with the added shallow model using sample data from a specific domain, so that the resulting model is more suitable for a specific application scenario. As an example, in the embodiments of this application, the pre-trained model is a first neural network model.

[0065] With the rapid development of the internet and the lowering of barriers to content production, the volume of various types of content being published is growing exponentially. Text and image content (including image galleries) is distributed through algorithms and displayed in feeds. Because internet content is so diverse, including articles, images, and videos, the quality of this content varies greatly and requires review before distribution.

[0066] The review process also requires judging the completeness of the image. This judgment can only be made accurately by considering the context of the image content. In other words, a more granular understanding of the subject in the image is needed. For example, for beauty content, a cover image with only half a face is appropriate. However, for other content, a cover image with only half a face is incomplete.

[0067] In related technologies, content is manually reviewed and flagged before distribution, and images with quality issues are filtered out. However, manual review is inefficient when dealing with a large volume of content, so machine learning models are usually used to assist humans in judging the content.

[0068] In related technologies, the iterative process of information flow content processing algorithm models typically includes the following steps: requirement determination, data collection, model training, model testing, and deployment. The model training step usually includes model structure design and the corresponding pre-trained model. Because pre-trained models require a large amount of data collection and training time, they are generally not trained additionally. Instead, existing pre-trained models (such as those obtained from open-source platforms like ImageNet / COCO) are directly used for fine-tuning the corresponding tasks.

[0069] The applicant found that supervised learning modeling involves sample collection. In information flow business, many models have very sparse samples within the business content, making large-scale collection difficult and costly. Therefore, the current main sources of samples for algorithm modeling are collecting negative user feedback and then manually reviewing it to obtain negative samples; and collecting corresponding samples through manual inspection of online content. However, the overall collection efficiency is very low, and it is all post-event collection, one step behind the problem. Furthermore, it is inefficient in dealing with variations of sample quality issues. Different quality problems require different models, and the models cannot adapt to multiple tasks and business domains, resulting in high modeling costs. At the same time, the models cannot cope with the problem of modality missing.

[0070] Furthermore, the applicant found that the approach of directly utilizing existing pre-trained models for task fine-tuning in related technologies lacks relevant image knowledge for the actual business domain. The extraction of image representation information does not consider multi-modal content processing scenarios, and the fine-tuning process still requires a large amount of classification and annotation data. Therefore, the pre-training stage does not significantly help reduce the sample size in the subsequent fine-tuning stage. For tasks related to image contextual text, such as image integrity and quality assessment, a large amount of sample data still needs to be collected. Thus, modeling and development efficiency and modeling costs are not optimized.

[0071] This application provides a method, apparatus, electronic device, storage medium, and program product for training a neural network model. By performing various modal training tasks on the neural network model, the model can be applied to various scenarios, improving modeling efficiency and reducing modeling costs. The following describes exemplary applications of the electronic device provided in this application for training neural network models. This electronic device can be implemented as various types of user terminals such as laptops, tablets, desktop computers, set-top boxes, and mobile devices (e.g., mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), or as a server. The following describes exemplary applications when the electronic device is implemented as a server.

[0072] The neural network model training method provided in this application can be executed by the terminal or the server alone, or by the terminal and the server working together. See [link to relevant documentation]. Figure 1 , Figure 1 This is a schematic diagram of the architecture of a neural network model training system 100 provided in this application embodiment, including a server 200 and a terminal 400. The terminal 400 is connected to the server 200 through a network 300, which can be a wide area network, a local area network, or a combination of both.

[0073] As an application example of this application in the image and text retrieval scenario, after obtaining the trained neural network model (also referred to as the first neural network model, or pre-trained model, in this application), some image and text sample pairs are obtained from the information flow scenario as training data. For example, image and text sample pairs are obtained from the cover image and title corresponding to the video content, or from the image and title corresponding to the image and text content. Then, the trained neural network model is fine-tuned based on the training data to make the model more suitable for the image and text retrieval scenario. After obtaining the fine-tuned neural network model, the user can operate the human-computer interaction interface (such as a search engine client) of the terminal 400 to input search keywords. The terminal 400 sends the search keywords input by the user to the server 200 through the network 300. The server 200 calls the fine-tuned neural network model to perform corresponding image and text matching processing based on the search keywords transmitted by the terminal 400 through the network 300, thereby finding the image that accurately corresponds to the search keywords. Afterwards, the server 200 returns the found image to the terminal 400 through the network 300 so that the terminal 400 can display the image returned by the server on the human-computer interaction interface for the user to view or save.

[0074] As an application example of this application for image quality judgment, after obtaining the trained neural network model, some image-text sample pairs are obtained from the information flow scenario as training data, such as obtaining image-text sample pairs from the cover image and title corresponding to the video content, or obtaining image-text sample pairs from the image and title corresponding to the image-text content. Then, the trained neural network model is fine-tuned based on the training data to make the model more suitable for the image quality judgment scenario. After obtaining the fine-tuned neural network model, the user can input the image to be judged from the information flow through the human-computer interaction interface of the terminal 400. The terminal 400 sends the image to be judged input by the user to the server 200 through the network 300. The server 200 calls the fine-tuned neural network model to perform corresponding image-text matching processing based on the image to be judged transmitted by the terminal 400 through the network 300, thereby determining the text description of the quality problem corresponding to the image to be judged (such as incomplete image). Afterwards, the server 200 returns the determined text description of the corresponding quality problem to the terminal 400 through the network 300, so that the terminal 400 can display the text description of the quality problem corresponding to the image returned by the server on the human-computer interaction interface, thereby assisting the user in judging the image quality.

[0075] As an application example of this application in content classification, after obtaining the trained neural network model, some image-text sample pairs and associated tags are obtained from the information flow scenario as training data. For example, image-text sample pairs are obtained from the cover image and title corresponding to the video content, and the tags or categories corresponding to the video content are obtained. Then, the trained neural network model is fine-tuned based on the training data to make the model more suitable for the content classification scenario. After obtaining the fine-tuned neural network model, the user can operate the human-computer interaction interface of the terminal 400 to input the image to be classified and the corresponding title. The terminal 400 sends the image to be classified and the corresponding title input by the user to the server 200 through the network 300. The server 200 calls the fine-tuned neural network model to perform tag matching processing based on the image to be classified and the corresponding title transmitted by the terminal 400 through the network 300, thereby determining the tags corresponding to the image to be classified and the corresponding title, and classifying the image to be classified and the corresponding title according to the determined tags to obtain the corresponding classification results. Afterwards, the server 200 returns the determined classification results to the terminal 400 via the network 300, so that the terminal 400 can display the classification results returned by the server on the human-computer interaction interface, thereby assisting the user in classifying the text and image content.

[0076] As an application example of this application in a question-and-answer scenario, after obtaining the trained neural network model, some image-text sample pairs are obtained from the information flow scenario, and the titles corresponding to the images are masked. The images and their corresponding masked titles are used as training data. Then, the trained neural network model is fine-tuned based on the training data to make the model more suitable for the question-and-answer scenario. After obtaining the fine-tuned neural network model, the user can input incomplete question text through the human-computer interaction interface (such as a search engine client) of terminal 400. Terminal 400 sends the incomplete question text input by the user to server 200 through network 300. Server 200, based on the incomplete question text transmitted by terminal 400 through network 300, calls the fine-tuned neural network model to perform masked language prediction processing, thereby predicting the complete text corresponding to the incomplete question text, and determining the corresponding answer text based on the complete text. Afterwards, server 200 returns the determined answer text to terminal 400 through network 300, so that terminal 400 can display the answer text returned by the server on the human-computer interaction interface, allowing the user to view the answer to the question.

[0077] As an application example of this application in content recommendation, after obtaining the trained neural network model, some image-text sample pairs and associated tags are obtained from the information flow scenario as training data. For example, image-text sample pairs are obtained from the cover image and title corresponding to the video content, and the tags or categories corresponding to the video content are obtained. Then, the trained neural network model is fine-tuned based on the training data to make the model more suitable for the content recommendation scenario. After obtaining the fine-tuned neural network model, the user can operate the human-computer interaction interface of the terminal 400 to input preferred tags, titles, and other information. The terminal 400 sends the user-inputted tags, titles, and other information to the server 200 through the network 300. Based on the tags, titles, and other information transmitted by the terminal 400 through the network 300, the server 200 calls the fine-tuned neural network model to perform cross-modal similarity prediction processing, thereby determining video content or image-text content with tags and titles similar to those input by the user. Afterwards, the server 200 returns the determined video or graphic content to the terminal 400 via the network 300, so that the terminal 400 can display the video or graphic content returned by the server on the human-computer interaction interface and recommend it to the user for browsing.

[0078] In some embodiments, server 200 may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Terminal 400 may be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, etc., but is not limited thereto. Terminals and servers can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment.

[0079] See below Figure 2 , Figure 2 This is a schematic diagram of the structure of the server 200 provided in the embodiments of this application. Figure 2 The server 200 shown includes at least one processor 210, memory 230, and at least one network interface 220. The various components of server 200 are coupled together via a bus system 240. It is understood that the bus system 240 is used to implement communication between these components. In addition to a data bus, the bus system 240 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 240.

[0080] Processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0081] The memory 230 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 230 may optionally include one or more storage devices physically located away from the processor 210.

[0082] The memory 230 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 230 described in this application embodiment is intended to include any suitable type of memory.

[0083] In some embodiments, memory 230 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0084] Operating system 231 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks.

[0085] The network communication module 232 is used to reach other computing devices via one or more (wired or wireless) network interfaces 220, such as Bluetooth, WiFi, and Universal Serial Bus (USB).

[0086] In some embodiments, the training apparatus for the neural network model provided in this application can be implemented in software. Figure 2 A training device 233 for a neural network model stored in memory 230 is shown. It can be software in the form of programs and plug-ins, including the following software modules: acquisition module 2331, fusion module 2332, first determination module 2333, second determination module 2334, and update module 2335. These modules are logical and can therefore be arbitrarily combined or further split according to the functions they implement.

[0087] The training method for the neural network model provided in this application will be described below with reference to exemplary applications and implementations of the electronic device provided in the embodiments of this application. It will be understood that the following method can be executed individually or collaboratively by the terminal 400 or server 200 described above.

[0088] Before describing the training method of the neural network model provided in the embodiments of this application, the structure of the neural network model involved in the embodiments of this application will be described first. See [link to documentation]. Figure 4 , Figure 4 This is a schematic diagram of the structure of the neural network model provided in the embodiments of this application.

[0089] like Figure 4 As shown, this application embodiment involves a first neural network model, a second neural network model, and a third neural network model. The first neural network model is used for single-modal similarity prediction and cross-modal prediction tasks based on image features, text features, and a first fusion feature. The cross-modal prediction tasks include: cross-modal similarity prediction of images and titles, cross-modal matching prediction of images and titles, cross-modal matching prediction of images, titles, and tags, cross-modal multi-label prediction of images and titles, and cross-modal masked language prediction of images and titles. The second neural network model is used to perform image feature extraction processing on images to obtain image features. The third neural network model is used to perform text feature extraction processing on text to obtain text features.

[0090] The training method of the neural network model provided in the embodiments of this application will be described below.

[0091] See Figure 3A , Figure 3A This is a flowchart illustrating the training method for the neural network model provided in this application embodiment, which will be combined with... Figure 3A The steps shown are explained.

[0092] In step 101, features of multiple modalities of the image are acquired.

[0093] As an example, we first obtain the features of multiple modalities of the image. These features include image features, title text features, and label text features.

[0094] Here, the title text features and label text features are determined based on the title corresponding to the image and the label corresponding to the image, respectively. The image and its corresponding title can constitute an image-text sample pair.

[0095] As an example, multiple image-text sample pairs can be obtained in the following way:

[0096] 1) Obtaining image-text sample pairs based on video and text content. Typically, video or text content uploaded to internet media has a cover image and a title. Since the entity information in the title is strongly correlated with the cover image, the cover image and title of each video or text content can be selected as an image-text sample pair. These resulting image-text sample pairs can be considered weakly supervised information and do not require further manual labeling.

[0097] If the video or text content does not have a title, optical character recognition (OCR) can be used to process the image to extract text information from the image, and the extracted text information can be combined with the image to form a text-image sample pair.

[0098] Meanwhile, each piece of text or video content usually has tag information marked by the author during publication or during the link processing. The tag information can be considered as a condensed description of the content. Therefore, the tag information of each text sample pair is collected at the same time.

[0099] These data belong to specific business areas and can be collected from the information flow chain.

[0100] It's important to note that there are many forms of weak supervision, and the code and costs of different weak supervision methods vary significantly. In the information flow business, when content is published, author accounts provide text descriptions, and authors also publish keywords such as the content's category, tags, and thematic terms. When a user searches for a specific thematic term, such as "cats fighting," if the content contains this tag, it will appear in the most recent search results for that tag. Therefore, these tags can serve as weak supervision information for the sample data.

[0101] 2) Obtaining image-text sample pairs from public image-text sample datasets. The ConceptualCaptions (CC) dataset is a dataset containing (image, caption) pairs, which can be used for training and evaluating machine learning image captioning systems. The CC dataset includes two versions: CC3M (containing approximately 3.3 million images) and CC12M (containing approximately 12 million images). These are weakly related image-text descriptions automatically collected from the network through a simple filtering procedure. Because the images and their original descriptions in the CC dataset come from the network and have a wider range of styles, approximately 12 million image-text sample pairs can be obtained from CC12M.

[0102] In addition, image-text sample pairs can be obtained from datasets such as SBU, Flickr30K, and COCO. These datasets include labeled classification data, so the images and labels themselves constitute weakly supervised data.

[0103] 3) Crawling image-text sample pairs from the Internet. For example, through content distributed via information feeds, the entity words corresponding to the content tags are counted as search keywords. A large number of images are collected through search engines and vertical websites, and the crawled images and their corresponding keywords are used as image-text sample pairs.

[0104] As an example, image-text sample pairs can be crawled in the following way: For all video tags in the information flow recommendation library, statistical analysis is performed on the tags corresponding to all videos within 200 days. After deduplication, the frequency of tag occurrence is calculated, resulting in a total of 520,000(w) tag texts. For image-text tags in the information flow content library, the tags of all image-text content within the past year can be statistically analyzed. After deduplication, the frequency of tag occurrence is calculated and then sorted. The overall tags consist of entity nouns and motion words, with a total of approximately 5.2 million words. The top 250,000 basically cover common entity words and frequently used motion words. The video and image-text tag information is merged and deduplicated to obtain the final tag set. Then, using this tag information as search keywords, images are retrieved from the search engine, resulting in a large number of corresponding images. Next, images with a resolution lower than 200x200 are removed. To save on image storage costs, downloaded images can be converted to 224x224 size before storage. Images corresponding to similar or identical tags can be considered similar, while images corresponding to other different tags are dissimilar.

[0105] In some cases, if the image corresponding to the keyword cannot be found, the image portion can be filled with a blank image with all pixels being 0.

[0106] Obtaining image and text sample pairs from multiple sources can increase the generalization of sample data and greatly help reduce the sample requirements for subsequent actual prediction tasks.

[0107] After obtaining a large number of image-text sample pairs through the above method, feature extraction processing can be performed on each image-text sample pair to obtain the features of multiple modalities corresponding to the image in each image-text sample pair.

[0108] See Figure 3B , Figure 3B This is a flowchart illustrating a training method for a neural network model provided in an embodiment of this application. Based on Figure 3A , Figure 3B Step 101 shown can be implemented through steps 1011-1015. The following will combine... Figure 3B Steps 1011-1015 are described below.

[0109] In step 1011, the image is subjected to data augmentation processing to obtain the enhanced image.

[0110] As an example, when performing feature extraction on an image, data augmentation can be performed on the original image. This includes processes such as rotation, cropping, adding Gaussian noise, masking, grayscale processing, and adding filters to obtain an enhanced image. The enhanced image can then be used for self-supervised learning; for instance, the original image and its corresponding grayscale image can be used for self-supervised learning.

[0111] As an example, images can be processed using Application Programming Interface (API) augmentations, which are specifically designed for data augmentation, to obtain an enhanced image.

[0112] In step 1012, the second neural network model is invoked based on the image to perform image feature extraction processing, thereby obtaining the first image feature corresponding to the image.

[0113] As an example, after obtaining the enhanced image, feature extraction can be performed on both the original image and the enhanced image.

[0114] Here, a second neural network model is invoked based on the original image to perform image feature extraction processing, obtaining the first image features corresponding to the original image. As an example, the second neural network model can be a visual neural network (VisionTransformer, ViT) model.

[0115] In some embodiments, step 1012 can be implemented by calling the second neural network model to perform the following processing: dividing the image into blocks, performing linear projection processing based on the obtained multiple image blocks to obtain multiple corresponding linear projection features; summing each linear projection feature with its corresponding position feature, performing self-attention calculation processing based on the obtained summation result to obtain a first self-attention feature corresponding to each linear projection feature; and performing fully connected calculation processing based on the multiple first self-attention features to obtain a first image feature corresponding to the image.

[0116] As an example, the ViT model is called to perform the following processing to divide the original image into blocks, resulting in multiple image blocks. For example, the original image of size 224*224 can be divided into image blocks of a fixed size (such as 16*16), thus obtaining (224*224) / (16*16), or 196 image blocks. Each image block has a dimension of 16*16*3, or 768 dimensions.

[0117] After obtaining multiple image patches, linear projection is performed on these patches to obtain linear projection features for each patch. Next, the linear projection features of each patch are added to the positional features of that patch to obtain a summation result for each patch. Then, self-attention calculation is performed on the summation result for each patch to obtain the first self-attention feature corresponding to each linear projection feature. This self-attention calculation can be implemented using a multi-head self-attention mechanism. Finally, fully connected computation is performed based on the obtained first self-attention features to obtain the first image feature corresponding to the original image.

[0118] It should be noted that the first image feature here is the feature output from the penultimate layer of the ViT model, not the final output of the ViT model. This is because the granularity of the image at the abstract semantic level increases with the number of layers. Using the semantic granularity of the penultimate layer makes it easier to fine-tune during the application phase.

[0119] The first image feature corresponding to the original image can be accurately obtained through the above method.

[0120] In step 1013, the second neural network model is invoked based on the enhanced image to perform image feature extraction processing, thereby obtaining the second image features corresponding to the enhanced image.

[0121] As an example, after obtaining the enhanced image, a second neural network model is invoked to perform image feature extraction processing based on the enhanced image, thereby obtaining the second image features. The method for obtaining the second image features is similar to that for obtaining the first image features, and will not be described again here.

[0122] In step 1014, the title corresponding to the image is segmented into words. Based on the segmentation results of the title, the third neural network model is called to perform two text feature extraction processes to obtain the first title text feature and the second title text feature.

[0123] As an example, when obtaining title text features, the title corresponding to the image in the image-text sample pair is first segmented into words to obtain the word segmentation result. For example, the title can be segmented into words using the tokenizer (length 64) in the Bidirectional Encoder Representations from Transformers (BERT) model. Each word in the title segmentation result is the sum of the position embedding, segmentation embedding, and token embedding of the corresponding word.

[0124] After obtaining the word segmentation results for the title, the third neural network model is invoked to perform two text feature extraction processes based on these results, thereby obtaining the first and second title text features. Furthermore, each time the third neural network model performs text feature extraction, a first proportion of neurons in the model are randomly discarded.

[0125] As an example, the third neural network model can be the BERT model, which has an R-Drop regularization mechanism. The R-Drop mechanism means that during each training process of a deep learning network, for multiple neural network units (i.e., neurons), a certain proportion (such as the first proportion) of neurons is temporarily dropped from the network. In other words, the parameters of some neurons are randomly set to 0. The model after dropping some neurons is called a sub-model, and the model without dropping any neurons is called the large model. By training the sub-models, the large model can eventually contain multiple sub-models that solve the problem, thereby improving the model's generalization ability.

[0126] As an example, a BERT model with R-Drop mechanism can be called on the same sample to perform two text feature extraction processes, and then the cross-entropy loss of the two outputs can be calculated. The R-Drop mechanism is used to ensure that the prediction results corresponding to the two outputs are the same, and the information divergence of the two outputs is low, thereby ensuring that the output of the same sample under different sub-models is as consistent as possible.

[0127] Therefore, in this embodiment of the application, as an example, a BERT model with R-Drop mechanism can be called based on the word segmentation results of the title to perform two text feature extraction processes, thereby obtaining the features output by the BERT model twice, namely the first title text feature and the second title text feature. Obtaining the title text features in this way can make full use of the original corpus information and improve the generalization effect of the model.

[0128] In step 1015, the labels corresponding to the image are segmented into words. Based on the segmentation results of the obtained labels, the third neural network model is called to perform text feature extraction to obtain the label text features.

[0129] As an example, after obtaining the image-text sample pair, the associated tags of the image-text sample pair are segmented, and based on the segmentation results of the obtained tags, a third neural network model is called to perform text feature extraction to obtain the tag text features.

[0130] In some embodiments, step 1015 can be implemented by calling a third neural network model to perform the following processing: encoding the multiple words included in the word segmentation result of the tag to obtain the context features corresponding to each word; wherein, the encoding processing includes multiple cascaded encoding processes; and determining the context features corresponding to the multiple words as tag text features.

[0131] As an example, the following explanation uses the BERT model as the third neural network model.

[0132] After obtaining the word segmentation results of the tags, the word segmentation results are encoded to obtain the corresponding context features of each word in the word segmentation results. Since the BERT model includes multiple cascaded Transformer model encoders, multiple cascaded encoding processes will be performed here.

[0133] After obtaining the context features corresponding to each word segment, these context features are determined as the label text features. For example, if the word segmentation result corresponding to label A is (a1, a2, a3, a4), after processing the text features using the BERT model, the context features F1, F2, F3, and F4 corresponding to each word are obtained. Therefore, (F1, F2, F3, F4) are determined as the text features of label A. Using this method, the label text features can be accurately determined.

[0134] It should be noted that the method for obtaining title text features is similar to that for obtaining tag text features, and will not be repeated here.

[0135] In step 102, the image features and title text features are fused together to obtain the first fused feature across modalities.

[0136] As an example, after obtaining the features of multiple modalities of an image, a fusion process is performed based on the image features and the title text features to obtain the first fused feature across modalities.

[0137] See Figure 3C , Figure 3C This is a flowchart illustrating a training method for a neural network model provided in an embodiment of this application. Based on Figure 3A , Figure 3C Step 102 shown can be implemented through steps 1021-1024. The following will combine... Figure 3C Steps 1021-1024 are described below.

[0138] In step 1021, the title corresponding to the image is masked to obtain the masked title.

[0139] As an example, after obtaining the title corresponding to the image, the title is masked to obtain the masked title.

[0140] As an example, the masking here can be achieved by adding a mask [mask] to the text. For example, if the title of the image is "Do you know how two cats fight?", if we want to mask the cats and the fight, we can add [mask] to the corresponding position to get the masked title "Do you know how two [mask] [mask][mask]?".

[0141] As an example, the masking process here includes masking at different granularities, such as character-level masking, word-level masking, and mixed character-word masking. For instance, if the image's title is "ABC is a car brand," character-level masking would mask each character individually; word-level masking would mask words like "ABC," "a," "car," and "brand"; and mixed character-word masking would mask text such as "ABC," "car," and "brand" within the title. By using various masking methods at different granularities, the title text can be broken down and recombined from the bottom up, providing more supervisory information.

[0142] In step 1022, the masked title is segmented into words. Based on the segmentation results of the masked title, the third neural network model is called to extract text features to obtain the text features of the masked title.

[0143] As an example, after obtaining the masked title, the masked title is segmented into words to obtain the segmentation results. Based on the segmentation results, a third neural network model is called to perform text feature extraction to obtain the text features of the masked title.

[0144] It should be noted that the method for obtaining the features of the masked title text is similar to the method for obtaining the features of the title text, and will not be repeated here.

[0145] In step 1023, the image features and the masked title text features are concatenated to obtain the second concatenated features.

[0146] As an example, after obtaining the masked title text features, the image features and the masked title text features are concatenated to obtain the second concatenated feature. For example, if both the image features and the masked title text features have a dimension of 768, then the dimension of the second concatenated feature will be 1536.

[0147] In step 1024, the second splicing feature is downsampled to obtain the first fusion feature across modalities.

[0148] As an example, after obtaining the second concatenated feature, it is downsampled to obtain the first fused feature across modalities. For instance, the 1536-dimensional second concatenated feature can be downsampled by a factor of 6 to obtain the 256-dimensional first fused feature.

[0149] It should be noted that the first fusion feature can be obtained not only in the manner described in steps 1021-1024 above, i.e., based on image features and masked title text features, but also directly based on image features and title text features. This method will be explained below.

[0150] In step 103, the first neural network model is invoked based on the features of multiple modalities to perform multiple single-modal prediction tasks, thereby obtaining the corresponding single-modal prediction results, and the corresponding single-modal loss is determined based on the prediction results within each modality.

[0151] As an example, after obtaining features from multiple modalities, a first neural network model can be invoked based on these features to perform multiple single-modal prediction tasks, thereby obtaining the prediction result for each single modality. Then, the corresponding single-modal loss is determined based on the prediction result within each modality. As an example, the first neural network here can be the Encoder of a Transformer model.

[0152] In some embodiments, when the single-modal prediction task is an image modality similarity prediction task, step 103 can be implemented in the following way: based on the first image features and the second image features, the first neural network model is called to perform the image modality similarity prediction task, to obtain the first predicted similarity between the first image features and the second image features, and the first predicted similarity is determined as the loss of the image modality.

[0153] As an example, the features of multiple modalities include image features, which include first image features and second image features. The first and second image features are obtained by performing image feature extraction processing on the original image and the enhanced image respectively using a second neural network model.

[0154] Here, when the single-modal prediction task is an image modality similarity prediction task, the features of multiple modalities are image features. Therefore, based on the first image features and the second image features, the first neural network model is invoked to perform the image modality similarity prediction task, thereby obtaining the first predicted similarity between the first image features and the second image features.

[0155] As an example, a first neural network model can be invoked, and the first and second image features can be substituted into a specific loss function for calculation to obtain the first predicted similarity. This specific loss function can be the loss function corresponding to the Simple Contrastive Learning of Sentence Embeddings (SimCSE) model. By substituting the first and second image features into the SimCSE loss function, the SimCSE loss, i.e., the first predicted similarity, can be obtained, and this first predicted similarity can be determined as the loss of the image modality.

[0156] In some embodiments, when the single-modal prediction task is the similarity prediction task of the title modality, step 103 can also be implemented in the following way: based on the first title text features and the second title text features, the first neural network model is called to perform the title modality similarity prediction task, to obtain the second predicted similarity of the first title text features and the second title text features, and the second predicted similarity is determined as the loss of the title modality.

[0157] As an example, when the unimodal prediction task is a title modality similarity prediction task, the features of multiple modalities are title text features, which include first title text features and second title text features. The first and second title text features are obtained by calling a third neural network model twice to extract text features based on the title corresponding to the image.

[0158] Here, when the unimodal prediction task is the similarity prediction task of the title modality, the features of the multiple modalities are the title text features. Therefore, based on the first title text features and the second title text features, the first neural network model is invoked to perform the title modality similarity prediction task, thereby obtaining the second predicted similarity between the first title text features and the second title text features.

[0159] As an example, the first neural network model can be called, and the first and second title text features can be substituted into the loss function corresponding to SimC SE to obtain the SimCSE loss, which is the second predicted similarity. The second predicted similarity can then be determined as the loss of the title modality.

[0160] In some embodiments, when the prediction task of a single modality is the similarity prediction task of a label modality, step 103 can also be implemented in the following way: calling the first neural network model based on different label text features to perform the label modality similarity prediction task, obtaining the label prediction similarity of different label text features, and determining the label prediction similarity as the loss of the label modality.

[0161] As an example, the first neural network model can be called, and the different label text features can be substituted into the loss function corresponding to SimCSE to obtain the SimCSE loss, that is, the label prediction similarity, and the label prediction similarity can be determined as the loss of the label modality.

[0162] In step 104, the first neural network model is invoked based on the first fusion feature to perform multiple cross-modal prediction tasks, obtain the corresponding cross-modal prediction results, and determine the corresponding cross-modal loss based on each cross-modal prediction result.

[0163] As an example, see Figure 4 After obtaining the first fusion feature through step 102, the first neural network model can be invoked based on the first fusion feature to perform multiple cross-modal prediction tasks, thereby obtaining the corresponding cross-modal prediction results. Then, the corresponding cross-modal loss is determined based on the cross-modal prediction results.

[0164] In some embodiments, calling a first neural network model based on a first fusion feature to perform multiple cross-modal prediction tasks and obtain corresponding multiple cross-modal prediction results can be achieved in the following way: calling the first neural network model to perform the following processing: performing self-attention calculation processing based on at least one of the first fusion feature and the first label text feature to obtain a second self-attention feature; performing fully connected calculation processing based on the second self-attention feature to obtain a corresponding fully connected feature; and performing normalization calculation processing on the fully connected feature to obtain the corresponding multiple cross-modal prediction results.

[0165] As an example, the following explanation uses the Transformer model's encoder as the first neural network model. The Transformer model's encoder mainly includes a self-attention layer and a feed-forward neural network (FNN) layer, which is a fully connected layer.

[0166] Therefore, based on the first fusion feature, a self-attention layer is invoked for self-attention calculation to obtain the second self-attention feature; then, based on the second self-attention feature, a fully connected layer is invoked for fully connected calculation to obtain the fully connected feature; finally, the fully connected feature is normalized using a normalization function to obtain the prediction results for multiple cross-modalities. It should be noted that the values ​​of the multiple cross-modal prediction results here range from [0, 1]. Through the above method, the prediction result for each cross-modality can be accurately obtained.

[0167] See Figure 3D , Figure 3DThis is a flowchart illustrating the training method for the neural network model provided in this application. In some embodiments, based on... Figure 3A When the cross-modal prediction task is a cross-modal similarity prediction task between an image and a title, Figure 3D Step 104 shown can be implemented through steps 1041-1043. (This will be combined with...) Figure 3D Steps 1041-1043 are described below.

[0168] In step 1041, multiple different combinations are generated.

[0169] As an example, after calling the second neural network model to perform image feature extraction processing based on the original image and the enhanced image respectively, obtaining the first image feature and the second image feature respectively, and calling the third neural network model twice to perform text feature extraction processing based on the title corresponding to the image, obtaining the first title text feature and the second title text feature respectively, different combinations are generated based on the first image feature, the second image feature, the first title text feature and the second title text feature.

[0170] Each combination includes an image feature and a title text feature. The image feature is either the first image feature F1 or the second image feature F2, and the title text feature is either the first title text feature T1 or the second title text feature T2. In this way, four different combinations can be obtained: (F1, T1), (F1, T2), (F2, T1), and (F2, T2).

[0171] In step 1042, a fusion process is performed based on the image features and title text features included in each combination to obtain the first fused feature corresponding to each combination.

[0172] As an example, after obtaining four different combinations, a fusion process is performed based on the image features and title features included in each combination to obtain the first fused feature corresponding to each combination.

[0173] See Figure 3D Step 1042 can be achieved through steps 10421-10422. The following will combine... Figure 3D Steps 10421-10422 are described below.

[0174] In step 10421, the image features and title text features included in each combination are concatenated to obtain the first concatenated feature.

[0175] As an example, after obtaining four different combinations, the image features and title text features included in each combination are concatenated to obtain the first concatenated feature. For example, a combination is (F1, T1), which includes the first image feature and the first title text feature. Then, F1 and T1 are concatenated. If the dimensions of F1 and T1 are both 768, then the first concatenated feature obtained is 1536-dimensional.

[0176] In step 10422, the first splicing feature is downsampled to obtain the first fusion feature corresponding to each combination.

[0177] As an example, after obtaining the first concatenated feature, it is downsampled to obtain the first fusion feature corresponding to each combination. Continuing the example above, the 1536-dimensional first concatenated feature can be downsampled by a factor of 6 to obtain the 256-dimensional first fusion feature. Using this method, the first fusion feature can be accurately determined.

[0178] In step 1043, any two first fusion features are selected from multiple first fusion features. Based on the selected two first fusion features, the first neural network model is called to perform a cross-modal similarity prediction task between the image and the title, and the third predicted similarity between the two first fusion features is obtained. The third predicted similarity is then determined as the first cross-modal loss.

[0179] As an example, since four different combinations can be obtained in step 1041, and each combination can determine a first fusion feature in step 1042, four first fusion features can be obtained accordingly. In the case where the cross-modal prediction task is the cross-modal similarity prediction task between an image and a title, two first fusion features are arbitrarily selected from these four. Based on these two selected first fusion features, the first neural network model is invoked to perform the cross-modal similarity prediction task between the image and the title, thereby obtaining the third predicted similarity of these two first fusion features.

[0180] As an example, the first neural network model can be called, and the two selected first fusion features can be substituted into the loss function corresponding to SimCSE each time to obtain the SimCSE loss, which is the third predicted similarity. The third predicted similarity is then determined as the loss of the title modality.

[0181] In some embodiments, when the cross-modal prediction task is a cross-modal matching prediction task of an image and a title, step 104 can be implemented in the following way: based on the first fusion feature, the first neural network model is called to perform the cross-modal matching prediction task of the image and the title to obtain the first prediction result; the first prediction result and the corresponding first label value are substituted into the first loss function for calculation and processing to obtain the second cross-modal loss.

[0182] As an example, when the cross-modal prediction task is the cross-modal matching prediction task of image and title, based on the first fusion feature determined in step 1042, the first neural network model is called to perform the cross-modal matching prediction task of image and title. That is, the first neural network model is used to determine whether the image features corresponding to the first fusion feature match the title text features, which is to determine whether the image and title match.

[0183] The matching prediction task here is performed in batch processing. Each batch includes positive and negative samples. Positive samples include the first fusion feature determined based on the image features and title text features corresponding to the image and the corresponding title in the image-text sample pair, respectively. The first label value of the positive sample is 1. Negative samples include the first fusion feature determined based on the image features and title text features corresponding to the image and the corresponding title (which do not correspond to the image). The first label value of the negative sample is 0.

[0184] Based on the first fusion feature in a batch, the first neural network model is called to perform a cross-modal matching and prediction task between the image and the title, and a first prediction result is obtained. The first prediction result represents the matching degree between the image features corresponding to the first fusion feature and the title text features. If the first fusion feature is a positive sample, the value of the first prediction result should be as close to 1 as possible; if the first fusion feature is a negative sample, the value of the first prediction result should be as close to 0 as possible.

[0185] After obtaining the first prediction result, the first prediction result and the corresponding first label value are substituted into the first loss function for calculation, thereby obtaining the first cross-modal loss. If the first fusion feature is a positive sample, the first prediction result and the corresponding first label value 1 are substituted into the first loss function for calculation; if the first fusion feature is a negative sample, the first prediction result and the corresponding first label value 0 are substituted into the first loss function for calculation.

[0186] As an example, the first loss function can be the cross-entropy loss function, which is calculated as follows:

[0187]

[0188] Where L1 represents the cross-entropy loss function, m represents the number of the first fused features in a batch, and y i Indicates the first label value, y i The value of y is either 0 or 1. If the first fused feature is a positive sample, then y i The value is 1; if the first fusion feature is a negative sample, then y iThe value is 0; p i This indicates the first prediction result.

[0189] The second transmodal loss can be calculated using Formula 1 above.

[0190] In some embodiments, a second number of first fusion features are determined from a first number of first fusion features according to a second ratio; the second number of first fusion features are replaced with corresponding image features; a first neural network model is invoked based on each image feature to perform a cross-modal matching prediction task between the image and the title, and a second prediction result is obtained; the second prediction result and the first label value are substituted into a first loss function for calculation and processing to obtain a second cross-modal loss.

[0191] As an example, there are situations where not all images have corresponding titles; some may only have a single image (i.e., only the image modality). In such cases, it's necessary to predict the corresponding text context for that single image. For instance, if we need to detect quality issues with the cover image of a short video, and the short video lacks a title or the uploader hasn't provided tags or categories, we can only use the image for quality assessment. If modality loss and corresponding handling aren't considered during pre-training, the generalization effect of the pre-trained model will be poor, and we might even need to construct different pre-trained models for the text modality and the image modality separately. Therefore, this embodiment of the application simulates the situation of missing text modalities during training to better handle inference scenarios.

[0192] As an example, a second number of first fusion features are randomly determined from the first number of first fusion features included in each batch, according to a second ratio. For example, according to a 50% ratio, 100*50%, or 50, first fusion features are randomly determined from the 100 first fusion features included in each batch. Then, these 50 determined first fusion features are replaced with corresponding image features. For example, if the first fusion features are determined based on the first image features and the first title text features, then the first fusion features are replaced with the first image features. In this way, the first fusion features, which include two modalities of image and title, are replaced with image features that only have the image modality.

[0193] After replacing the 50 first fusion features with their corresponding image features, 50 image features are obtained. Then, based on each image feature, the first neural network model is invoked to perform a cross-modal matching and prediction task between the image and the title, yielding a second prediction result. This second prediction result represents the probability that an image feature corresponds to a predicted title text. In this case, the first neural network model predicts the possible title text corresponding to the image, and the probability of each predicted title text.

[0194] After obtaining the second prediction result, the second prediction result and the first label value are substituted into the first loss function for calculation to obtain the second cross-modal loss.

[0195] The above method can simulate the situation where the title modality is missing in actual application scenarios, which can increase the robustness of the first neural network model.

[0196] In some embodiments, when the cross-modal prediction task is a cross-modal matching prediction task of image, title and label, step 104 can be implemented in the following way: based on the first fusion feature and the first label text feature, call the first neural network model to perform the cross-modal matching prediction task of image, title and label to obtain the third prediction result; substitute the third prediction result and the corresponding second label value into the second loss function for calculation and processing to obtain the third cross-modal loss.

[0197] As an example, when the cross-modal prediction task is a cross-modal matching prediction task of image, title, and label, the first neural network model is invoked based on the first fused feature and the first label text feature to perform prediction processing, thereby obtaining the third prediction result. Here, the first label text feature is obtained by downsampling the label text feature, and the dimension of the first label text feature is the same as the dimension of the first fused feature. The third prediction result is the matching degree between the first fused feature and the first label text feature.

[0198] The matching prediction task here is also performed in batch processing. Each batch includes positive sample pairs and negative sample pairs. A positive sample pair includes a first fusion feature determined based on the image-text sample pair, and a first label text feature determined based on the label corresponding to the image-text sample pair. The second label value for the positive sample pair is 1. A negative sample pair includes a first fusion feature determined based on the image-text sample pair, and a first label text feature determined based on a label that does not correspond to the image-text sample pair. The second label value for the negative sample pair is 0.

[0199] Based on positive and negative sample pairs in a batch, the first neural network model is called to perform a cross-modal matching and prediction task of image, title, and label, and a third prediction result is obtained. If the first fused feature and the first label text feature form a positive sample pair, the value of the third prediction result should be as close to 1 as possible; if the first fused feature and the first label text feature form a negative sample pair, the value of the third prediction result should be as close to 0 as possible.

[0200] After obtaining the third prediction result, the third prediction result and the corresponding second label value are substituted into the second loss function for calculation to obtain the third cross-modal loss. The second loss function here can also be the cross-entropy loss function. As an example, the third prediction result and the corresponding second label value can be substituted into Formula 1 above to obtain the third cross-modal loss.

[0201] In some embodiments, when the cross-modal prediction task is a cross-modal multi-label prediction task of image and title, step 104 can be implemented in the following way: based on the first fusion feature, the first neural network model is called to perform a cross-modal multi-label prediction task of image and title, and multiple fourth prediction results are obtained; each fourth prediction result and the corresponding third label value are substituted into the third loss function for calculation and processing to obtain the cross-modal loss corresponding to each fourth prediction result; the cross-modal losses corresponding to each fourth prediction result are summed and averaged to obtain the fourth cross-modal loss.

[0202] As an example, when the cross-modal prediction task is a cross-modal multi-label prediction task of image and title, the first neural network model is called based on the first fusion feature to perform prediction processing, and multiple fourth prediction results are obtained. Each fourth prediction result represents the probability of the first fusion feature corresponding to a prediction label.

[0203] After obtaining multiple fourth prediction results, each fourth prediction result and its corresponding third label value are substituted into the third loss function to obtain the cross-modal loss corresponding to each fourth prediction result. Then, the cross-modal losses corresponding to multiple fourth prediction results are summed and averaged, and the average value is used as the fourth cross-modal loss.

[0204] Using the above method, a multi-label prediction task can be transformed into a binary classification problem on each label. Since the labels are not mutually exclusive, it is only necessary to determine whether each label exists. Ultimately, each label prediction task can be regarded as a binary classification task, and the loss calculation on the corresponding label can follow the binary classification calculation method. For example, based on the first fusion feature, the first neural network model is called for prediction processing to obtain 5 fourth prediction results (p1, p2, p3, p4, p5), and the corresponding third label values ​​are (y1, y2, y3, y4, y5). When the third loss function is the cross-entropy loss function, each fourth prediction result and the corresponding third label value can be substituted into Formula 1 for calculation to obtain 5 cross-modal losses. Finally, these 5 cross-modal losses are summed and averaged to obtain the fourth cross-modal loss.

[0205] In the above approach, multi-label prediction tasks require multiple binary classifiers, resulting in extremely imbalanced samples. Here, alternative loss functions can be used, such as replacing the cross-entropy loss function with asymmetric loss, to overcome the imbalance between positive and negative samples in multi-label prediction tasks. Alternatively, focal loss can be used, which reduces the focus on simple negative samples while attenuating a small number of positive samples.

[0206] As an example, the formula for calculating the focus loss function is as follows:

[0207]

[0208] Where p represents the fourth prediction result, (1-p) γ and p Υ These are the adjustment factors for the positive sample loss function and the negative sample loss function, respectively. These adjustment factors can reduce the weight of the loss function for high-confidence samples, allowing the first neural network model to focus more on difficult-to-classify samples during training; Υ is the aggregation parameter p. m This can make the loss function smoother.

[0209] As an example, the formula for calculating the asymmetric loss function is as follows:

[0210]

[0211] Where, p m =max(pm,0), where m is a learnable hyperparameter, and the meanings of the other parameters are the same as in Formula 2 above. This loss function can decouple the descent rates of the positive and negative sample loss functions, thereby better learning positive sample features when positive samples are scarce.

[0212] In some embodiments, when the cross-modal prediction task is a cross-modal masked language prediction task of image and title, step 104 can be implemented in the following way: based on the first fusion feature, the first neural network model is called to perform the cross-modal masked language prediction task of image and title to obtain the fifth prediction result; the fifth prediction result and the corresponding fourth label value are substituted into the fourth loss function for calculation and processing to obtain the fifth cross-modal loss.

[0213] As an example, when the cross-modal prediction task is a cross-modal masked language prediction task between an image and a title, the first neural network model is invoked based on the first fusion feature to perform prediction processing, resulting in a fifth prediction result. Here, the first fusion feature is determined through steps 1021-1024 above. The fifth prediction result is the probability of the predicted text corresponding to the masked portion of the title after masking.

[0214] After obtaining the fifth prediction result, the fifth prediction result and the corresponding fourth label value are substituted into the fourth loss function for calculation and processing, thereby obtaining the fifth cross-modal loss.

[0215] In step 105, backpropagation is performed based on the single-modal loss and cross-modal loss to update the parameters of the first neural network model.

[0216] As an example, after obtaining multiple single-modal losses and multiple cross-modal losses, backpropagation is performed based on all single-modal losses and all cross-modal losses to update the parameters of the first neural network model.

[0217] Repeat steps 101-105 above until all unimodal losses and all multimodal losses are minimized, or the set number of training iterations are reached, then end the training process of the first neural network model and obtain the trained first neural network model.

[0218] As an example, after obtaining the first trained neural network model, a small amount of sample data can be processed to fine-tune the final model based on the first trained neural network model. Typically, a shallow model such as a Logistic Regression (LR) model or a Support Vector Machine (SVM) is added to the output of the last layer of the first trained neural network model. The fine-tuning process is then completed using a small amount of sample data, allowing the fine-tuned neural network model to be deployed in real-world applications.

[0219] In this embodiment, by performing training tasks of various modalities based on the first neural network model and updating the parameters of the first neural network model based on the loss values ​​corresponding to the various training tasks, the first neural network model can be fully learned, enabling the trained first neural network model to support multiple application scenarios, thereby expanding the applicability of the trained first neural network model, effectively improving the modeling efficiency of the neural network model, and reducing the modeling cost.

[0220] The following will describe an exemplary application of the embodiments of this application in a real-world neural network model application scenario.

[0221] In this embodiment of the application, after obtaining the trained first neural network model, the trained first neural network model (i.e., the pre-trained model) can be fine-tuned based on sample data from a portion of the business domain, so that the fine-tuned first neural network model can be applied to the corresponding scenario.

[0222] As an example, data such as images, titles, and tags corresponding to video or text content can be obtained from information flow business scenarios. This data serves as weakly supervised sample data, which is then used to fine-tune the trained first neural network model. The fine-tuning process is similar to the training process, aiming to make the model more adaptable to the corresponding information flow business scenario. In this embodiment, since the first neural network model can perform various prediction tasks based on the same or similar sample data, the fine-tuned first neural network model can also perform various prediction tasks, thus allowing for flexible application in various business scenarios.

[0223] As an example, the fine-tuned first neural network model can be applied to image and text retrieval scenarios, accurately finding the corresponding images based on the user's input search keywords; it can also be applied to image quality assessment scenarios, accurately describing and judging the quality problems of the image to be judged based on the user's input; it can also be applied to image and text classification scenarios, accurately determining the tag information and classification information corresponding to the image and title based on the user's input image and title; it can also be applied to question-answering scenarios, predicting the corresponding complete question text based on the user's incomplete question text, and determining the corresponding answer text; and it can also be applied to recommendation scenarios, recommending video content or image and text content with similar titles and tags to the user based on the images, titles, tags, and other information corresponding to the video content or image and text content the user is currently browsing.

[0224] The training method of the neural network model provided in the embodiments of this application will be described below. See also Figure 5 , Figure 5 This is a schematic diagram of the training method for the neural network model provided in the embodiments of this application. For example... Figure 5 As shown, the image is processed by a second neural network model to extract image features, and then processed by a third neural network model to extract text features from the corresponding title and tag text information, resulting in title text features and tag text features. Next, the title text features and image features are fused together to obtain the first fused feature.

[0225] The system can use image features to call the first neural network model to perform image modality similarity prediction tasks, or it can use title text features to call the first neural network model to perform title modality similarity prediction tasks, or it can use tag text features to call the first neural network model to perform tag modality similarity prediction tasks.

[0226] Furthermore, the first neural network model can be invoked based on the first fusion feature to perform cross-modal similarity prediction tasks, cross-modal image-text matching tasks, cross-modal multi-label prediction tasks, and cross-modal masked language prediction tasks. It can also be invoked based on the first fusion feature and the labeled text features to perform cross-modal label matching tasks.

[0227] The following will describe the method for collecting sample data required for training the neural network model in the embodiments of this application.

[0228] In this application embodiment, sample data can be obtained in the following ways:

[0229] Method 1): Obtain image-text sample pairs based on video and text content. Typically, video or text content uploaded to internet media has a cover image and a title. Since the entity information in the title is strongly correlated with the cover image, the cover image and title of each video or text content can be selected as an image-text sample pair. The resulting image-text sample pairs can be considered weakly supervised information and do not require further manual labeling.

[0230] If the video or text content does not have a title, you can perform OCR processing on the image to extract the text information from the image, and then combine the extracted text information with the image to form a text-image sample pair.

[0231] Meanwhile, each piece of text or video content usually has tag information marked by the author during publication or during the link processing. The tag information can be considered as a condensed description of the content. Therefore, the tag information of each text sample pair is collected at the same time.

[0232] These data belong to specific business areas and can be collected from the information flow chain.

[0233] It's important to note that there are many forms of weak supervision, and the code and costs of different weak supervision methods vary significantly. In the information flow business, when content is published, author accounts have text descriptions, and authors also publish keywords such as the content's category, tags, and themes. When a user searches for a specific theme, such as "cats fighting," if the content contains this tag, then that content will appear in the most recent search results for that tag. These tags can serve as weak supervision information for the sample data.

[0234] Method 2): Obtain image-text sample pairs from public image-text sample pairs datasets. The CC dataset is a dataset containing (image, caption) pairs that can be used for training and evaluating machine learning image captioning systems. The CC dataset includes two versions: CC3M (containing approximately 3.3 million images) and CC12M (containing approximately 12 million images). These are weakly related image-text descriptions automatically collected from the network through a simple filtering procedure. Since the images and their original descriptions in the CC dataset come from the network and have a wider range of styles, approximately 12 million image-text sample pairs can be obtained from CC12M.

[0235] In addition, image-text sample pairs can be obtained from datasets such as SBU, Flickr30K, and COCO. These datasets include labeled classification data, so the images and labels themselves constitute weakly supervised data.

[0236] Method 3): Crawling image-text sample pairs from the Internet. For example, through content distributed via information feeds, the entity words corresponding to the content tags are counted as search keywords. A large number of images are collected through search engines and vertical websites, and the crawled images and their corresponding keywords are used as an image-text sample pair.

[0237] Image and text sample pairs can be crawled in the following way: For all video tags in the information flow recommendation library, statistical analysis of the tags corresponding to all videos within 200 days is performed. After deduplication, the tag frequency is calculated, resulting in a total of 520,000 tag texts. For image and text tags in the information flow content library, the tags of all image and text content within the past year can be statistically analyzed. After deduplication, the tag frequency is calculated and then sorted. The overall tags consist of entity nouns and motion words, with a total of 5.2 million words. The top 250,000 basically cover common entity words and frequently used motion words. The video and image / text tag information is merged and deduplicated to obtain the final tag set. Then, using this tag information as search keywords, images are retrieved from the search engine, resulting in a large number of corresponding images. Next, images with a resolution lower than 200x200 are removed. Images corresponding to similar or identical tags can be considered similar, while images corresponding to other different tags are dissimilar. To save on image storage costs, downloaded images can be converted to 224x224 size before storage.

[0238] In some cases, if the image corresponding to the keyword cannot be found, the image portion can be filled with a blank image with all pixels being 0.

[0239] Obtaining image and text sample pairs from multiple sources can increase the generalization of sample data and greatly help reduce the sample requirements for subsequent actual prediction tasks.

[0240] After obtaining a large number of image-text sample pairs through the above method, each image-text sample pair can be processed to obtain the features of multiple modalities corresponding to the image in each image-text sample pair.

[0241] As an example, the `albumentations` package is used to perform data augmentation on the original image in each image-text sample pair. This augmentation includes processes such as rotation, cropping, adding Gaussian noise, masking, color transformation, and adding filters, resulting in an augmented image. Then, a second neural network model (e.g., the ViT model) is called to extract image features from both the original and augmented images, yielding the first image features corresponding to the original image and the second image features corresponding to the augmented image. For this example, the image features here are the features output from the penultimate layer of the ViT model, not the final output. This is because the semantic granularity of the image increases with each subsequent layer; using the semantic granularity of the penultimate layer allows for better fine-tuning for final business applications.

[0242] As an example, using the tokenizer (64 characters long) of a third neural network model (such as BERT), the text information corresponding to the title, tags, and author account description of each image-text sample pair is segmented. Based on the segmentation results, the BERT model is called to extract text features, resulting in corresponding title text features, tag text features, etc. Specifically, the BERT model with R-Drop mechanism is called twice for feature extraction based on each title, thus obtaining the first title text features and the second title text features.

[0243] The R-Drop mechanism refers to the practice of temporarily discarding a portion of neurons (i.e., neurons) from the network during each training process of a deep learning network, according to a certain ratio (such as the first ratio). In other words, some neurons' parameters are randomly set to 0. The model after discarding some neurons is called a sub-model, while the model without discarding any neurons is called the main model. By training the sub-models, the main model can ultimately contain multiple sub-models that solve the problem, thereby improving the model's generalization ability. Here, by calling the BERT model with the R-Drop mechanism twice for two feature extraction processes, we can fully utilize the original corpus information and also improve the generalization effect of the final model.

[0244] After obtaining the first image feature, second image feature, first title text feature and second title text feature corresponding to each image and text sample pair, fusion processing can be performed based on different image features and different title text features to obtain multiple first fused features.

[0245] Having obtained these features, each prediction task will be explained below.

[0246] 1) Image modality similarity prediction task.

[0247] As an example, a first neural network model can be invoked, and the first and second image features can be substituted into a specific loss function for calculation to obtain the first predicted similarity. This specific loss function can be the loss function corresponding to the SimCSE model. By substituting the first and second image features into the SimCSE loss function, the SimCSE loss can be obtained, which is the predicted similarity of the image modality. This predicted similarity of the image modality is then determined as the loss of the image modality.

[0248] 2) Title modality similarity prediction task.

[0249] As an example, the first neural network model can be invoked, and the first and second title text features can be substituted into the loss function corresponding to SimC SE to obtain the SimCSE loss, which is the predicted similarity of the title modality. The predicted similarity of the title modality is then determined as the loss of the title modality.

[0250] 3) Tag modality similarity prediction task.

[0251] As an example, the first neural network model can be called, and the different label text features can be substituted into the loss function corresponding to SimCSE to obtain the SimCSE loss, that is, the label prediction similarity, and the label prediction similarity can be determined as the loss of the label modality.

[0252] 4) Cross-modal similarity prediction task.

[0253] As an example, the first neural network model can be called, and the two first fusion features selected from multiple first fusion features each time can be substituted into the loss function corresponding to SimCSE to obtain the SimCSE loss, which is the cross-modal prediction similarity. The cross-modal prediction similarity is then determined as the first cross-modal loss.

[0254] 5) Cross-modal image-text matching task.

[0255] As an example, the first neural network model can be called based on the first fusion feature in a batch to perform a cross-modal image-text task involving images and titles, obtaining a first prediction result. The first prediction result represents the matching degree between the image features corresponding to the first fusion feature and the title text features. If the first fusion feature is a positive sample, the value of the first prediction result should be as close to 1 as possible; if the first fusion feature is a negative sample, the value of the first prediction result should be as close to 0 as possible.

[0256] After obtaining the first prediction result, the first prediction result and the corresponding first label value are substituted into the first loss function for calculation, thereby obtaining the first cross-modal loss. If the first fusion feature is a positive sample, the first prediction result and the corresponding first label value 1 are substituted into the first loss function for calculation; if the first fusion feature is a negative sample, the first prediction result and the corresponding first label value 0 are substituted into the first loss function for calculation. Here, the first loss function can be the cross-entropy loss function, and the second cross-modal loss is determined in this way.

[0257] As an example, the formula for calculating the cross-entropy loss function is as follows:

[0258]

[0259] Where L1 represents the cross-entropy loss function, m represents the number of the first fused features in a batch, and y i Indicates the first label value, y i The value of y is either 0 or 1. If the first fused feature is a positive sample, then y i The value is 1; if the first fusion feature is a negative sample, then y i The value is 0; p i This represents the first prediction result. The second transmodal loss can be calculated using Formula 1 above.

[0260] As an example, in real-world business scenarios, not all images have corresponding titles; some may only have a single, plain image (i.e., only the image modality). This necessitates predicting the corresponding text context for that plain image. For instance, if we need to detect quality issues with the cover image of a short video, and the short video lacks a title or the uploader hasn't provided tags or categories, we can only use the image for quality assessment. If modality loss and corresponding handling aren't considered during pre-training, the generalization performance of the pre-trained model will be poor. We might even need to construct different pre-trained models for text modalities and image modalities separately. Therefore, this embodiment of the application needs to simulate the situation of missing text modalities during training to better handle inference scenarios.

[0261] As an example, a second number of first fusion features are randomly determined from the first number of first fusion features included in each batch, according to a second ratio. For example, according to a 50% ratio, 100*50%, or 50, first fusion features are randomly determined from the 100 first fusion features included in each batch. Then, these 50 determined first fusion features are replaced with corresponding image features. For example, if the first fusion features are determined based on the first image features and the first title text features, then the first fusion features are replaced with the first image features. In this way, the first fusion features, which include two modalities of image and title, are replaced with image features that only have the image modality.

[0262] After replacing the 50 first fusion features with their corresponding image features, 50 image features are obtained. Then, based on each image feature, the first neural network model is invoked to perform a cross-modal matching and prediction task between the image and the title, yielding a second prediction result. This second prediction result represents the probability that an image feature corresponds to a predicted title text. In this case, the first neural network model predicts the possible title text corresponding to the image, and the probability of each predicted title text.

[0263] After obtaining the second prediction result, the second prediction result and the first label value are substituted into the first loss function for calculation to obtain the second cross-modal loss.

[0264] 6) Cross-modal label matching task.

[0265] As an example, a first neural network model can be invoked based on the first fused feature and the first label text feature to perform prediction processing, thereby obtaining a third prediction result. Here, the first label text feature is obtained by downsampling the label text feature, and the dimension of the first label text feature is the same as the dimension of the first fused feature. The third prediction result is the matching degree between the first fused feature and the first label text feature.

[0266] The matching prediction task here is also performed in batch processing. Each batch includes positive sample pairs and negative sample pairs. A positive sample pair includes a first fusion feature determined based on the image-text sample pair, and a first label text feature determined based on the label corresponding to the image-text sample pair. The second label value for the positive sample pair is 1. A negative sample pair includes a first fusion feature determined based on the image-text sample pair, and a first label text feature determined based on a label that does not correspond to the image-text sample pair. The second label value for the negative sample pair is 0.

[0267] Based on positive and negative sample pairs in a batch, the first neural network model is called to perform a cross-modal matching and prediction task of image, title, and label, and a third prediction result is obtained. If the first fused feature and the first label text feature form a positive sample pair, the value of the third prediction result should be as close to 1 as possible; if the first fused feature and the first label text feature form a negative sample pair, the value of the third prediction result should be as close to 0 as possible.

[0268] After obtaining the third prediction result, the third prediction result and the corresponding second label value are substituted into the second loss function for calculation to obtain the third cross-modal loss. The second loss function here can also be the cross-entropy loss function. As an example, the third prediction result and the corresponding second label value can be substituted into Formula 4 above to obtain the third cross-modal loss.

[0269] 7) Cross-modal multi-label prediction task.

[0270] As an example, a first neural network model can be invoked based on the first fusion feature to perform prediction processing, resulting in multiple fourth prediction results. Each fourth prediction result represents the probability of a prediction label corresponding to the first fusion feature.

[0271] After obtaining multiple fourth prediction results, each fourth prediction result and its corresponding third label value are substituted into the third loss function to obtain the cross-modal loss corresponding to each fourth prediction result. Then, the cross-modal losses corresponding to multiple fourth prediction results are summed and averaged, and the average value is used as the fourth cross-modal loss.

[0272] Using the above method, a multi-label prediction task can be transformed into a binary classification problem on each label. Since the labels are not mutually exclusive, it is only necessary to determine whether each label exists. Ultimately, each label prediction task can be regarded as a binary classification task, and the loss calculation on the corresponding label can follow the binary classification calculation method. For example, based on the first fusion feature, the first neural network model is called for prediction processing to obtain 5 fourth prediction results (p1, p2, p3, p4, p5), and the corresponding third label values ​​are (y1, y2, y3, y4, y5). When the third loss function is the cross-entropy loss function, each fourth prediction result and the corresponding third label value can be substituted into Formula 4 for calculation to obtain 5 cross-modal losses. Finally, these 5 cross-modal losses are summed and averaged to obtain the fourth cross-modal loss.

[0273] In the above approach, multi-label prediction tasks require multiple binary classifiers, resulting in extremely imbalanced samples. Here, alternative loss functions can be used, such as replacing the cross-entropy loss function with an asymmetric loss function, to overcome the imbalance between positive and negative samples in multi-label prediction tasks. Alternatively, a focus loss function can be used, which reduces the focus on simple negative samples while attenuating a small number of positive samples.

[0274] As an example, the formula for calculating the focus loss function is as follows:

[0275]

[0276] Where p represents the fourth prediction result, (1-p) Υ and p Υ These are the adjustment factors for the positive sample loss function and the negative sample loss function, respectively. These adjustment factors can reduce the weight of the loss function for high-confidence samples, allowing the first neural network model to focus more on difficult-to-classify samples during training; Υ is the aggregation parameter p. m This can make the loss function smoother.

[0277] As an example, the formula for calculating the asymmetric loss function is as follows:

[0278]

[0279] Where, p m =max(pm,0), where m is a learnable hyperparameter, and the meanings of the other parameters are the same as in Formula 5 above. This loss function can decouple the descent rates of the positive and negative sample loss functions, thereby better learning positive sample features when positive samples are scarce.

[0280] See Figure 6 , Figure 6 This is a comparative diagram of the loss function curves provided in the embodiments of this application. For example... Figure 6 As shown, 601 represents the cross-entropy loss function, 602 represents the focus loss function, and 603 represents the asymmetric loss function. It can be seen that, compared to the cross-entropy loss function, the focus loss function changes more gradually; and compared to the focus loss function, the asymmetric loss function changes more smoothly.

[0281] 8) Cross-modal masked language prediction task.

[0282] As an example, after obtaining the title corresponding to the image, the title is masked to obtain the masked title. This masking can be achieved by adding a mask [mask] to the text. For example, if the title of the image is "Do you know how two cats fight?", if the cats and fighting are masked, [mask] can be added to the corresponding positions to obtain the masked title "Do you know how two [mask] [mask][mask]?".

[0283] As an example, the masking process here includes masking at different granularities, such as character-level masking, word-level masking, mixed character-word masking, and phrase-level masking (phrases can be selected from a custom phrase table defined by the business domain; a phrase represents an indivisible whole, such as the hero Li Bai in the game "Honor of Kings"). Words (or phrases) are composed of characters at the smallest granularity. During feature fusion, to ensure efficient computation, the number of repetitions of character codes implicitly indicates the number of characters a word comprises, indirectly telling the language model which parts are an indivisible whole. This ensures good performance while reducing computational cost.

[0284] See Figure 7 , Figure 7 This is a schematic diagram of the shielding process provided in an embodiment of this application. For example... Figure 7 As shown, assuming the caption corresponding to the image is "ABC is a car brand", such as Figure 7 As shown on the left, if character-level masking is performed, each character in the title can be masked individually to achieve the desired masking effect; for example... Figure 7 As shown on the right, if word-level masking is performed, words like "ABC," "one," "car," and "brand" in the title can be masked to achieve masking. If mixed word-level masking is performed, text such as "ABC," "car," and "brand" in the title can be masked to achieve masking. By using masking methods of different granularities, the title text can be decomposed and combined from the bottom up, providing more supervisory information, thereby improving the model's feature extraction and representation capabilities. It also avoids out-of-vocabulary words, allowing direct representation at the character level without the need for a separate vocabulary or training separate word vectors, thus improving training efficiency.

[0285] As an example, the first neural network model can be invoked based on the first fusion feature to perform prediction processing and obtain the fifth prediction result. The first fusion feature here is determined through steps 1022-1024 described above. The fifth prediction result is the probability of the predicted text corresponding to the masked portion of the title after masking.

[0286] After obtaining the fifth prediction result, the fifth prediction result and the corresponding fourth label value are substituted into the fourth loss function for calculation and processing, thereby obtaining the fifth cross-modal loss.

[0287] After obtaining the corresponding losses through the above seven tasks, the parameters of the first neural network model are updated during backpropagation based on the losses corresponding to each task.

[0288] Repeat the training process until all losses are minimized or the set number of training iterations are reached, then end the training process of the first neural network model and obtain the trained first neural network model.

[0289] As an example, after obtaining the first trained neural network model, a small amount of sample data can be processed to fine-tune the final model based on the first trained neural network model. Typically, a shallow model such as an LR model or SVM is added to the output of the last layer of the first trained neural network model, and the fine-tuning process is completed using a small amount of sample data. In this way, the fine-tuned neural network model can be put into practical application scenarios.

[0290] In the embodiments of this application, multiple pre-training tasks of different modalities are implemented through a single neural network model. By constraining the loss of multiple pre-training tasks, various semantic information in the corpus can be learned from multiple perspectives, which can significantly improve the comprehensive ability of the trained neural network model.

[0291] Furthermore, in this embodiment, specialized data supplemented by business domains can be fully utilized for prior learning, effectively accelerating the development progress of multimodal business algorithms and improving the model modeling effect. In particular, it reduces the need for supervised samples, effectively reducing the amount of labeled sample data and labor costs required for model iteration, effectively shortening the algorithm model development time, and can cope with the situation of missing modalities. Whether it is a one-sided input of text or image modality, it can achieve a good overall cost-performance effect, thereby expanding the applicability of the trained neural network model.

[0292] The functions of each service module in the training system for the information flow-based neural network model provided in the embodiments of this application will be described below.

[0293] I. Content Production and Content Consumption

[0294] (1) Content production ends such as PGC, UGC, Multi-Channel Network (MCN), and Professional User Generated Content (PUGC) provide local or photographed text, video or image collections through mobile terminals or backend application interfaces. These are the main sources of content for distribution.

[0295] (2) The content production end obtains the upload server interface address through communication with the upstream and downstream content interface services, and then uploads local files (content). The source of text and image content is usually the lightweight publishing end and the content editing entry. The video content publishing end is usually the shooting and photography end. During the shooting process, local video content can choose to match the music, filter templates and video beautification functions, etc.

[0296] (3) The content consumer communicates with the upstream and downstream content interface services, obtains the index information of the accessed content through recommendations, and then communicates with the content storage server to obtain the corresponding content, including recommended content and content subscribed to in specific topics. The content storage server stores content entities such as video source files and image source files, while the content's metadata, such as title, author, cover image, category, tag information, etc., is stored in the content database;

[0297] (4) The content consumption terminal simultaneously reports user playback behavior data, stuttering, loading time, playback clicks, etc. during the upload and download process to the server;

[0298] (5) Content consumption is done by browsing content data through feeds and can be refreshed continuously to obtain more recommended information from the server.

[0299] II. Uplink and Downlink Content Interface Server

[0300] (1) Communicate directly with the content production end and store the content submitted by the front end, usually the title, publisher, summary, cover image, publication time and other information into the content database;

[0301] (2) Write the metadata of the text and image content, such as file size, cover image link, title, publication time, author, etc., into the content database;

[0302] (3) Synchronize the published content to the scheduling center server for subsequent content processing and transfer.

[0303] III. Content Database

[0304] (1) The core database of content, in which all the metadata of the content published by producers is stored. The focus is on the metadata of the content itself, such as file size, cover image link, bitrate, file format, title, publication time, author, video file size, whether it is original or first publication, and also includes the classification of content during the manual review process (including first, second and third level classification and tag information, such as an article explaining XX brand mobile phone, the first level classification is technology, the second level classification is smartphone, the third level classification is domestic mobile phone, and the tag information is XX brand, YY model).

[0305] (2) During the manual review process, information in the content database will be read, and the results and status of the manual review will also be sent back to the content database.

[0306] (3) The content processing of the dispatch center mainly includes machine processing and manual review. Here, machine processing checks various quality judgments, such as low-quality filtering, content tags such as classification and tag information, and content deduplication. These results will be written into the content database. Completely duplicate content will not be given to manual secondary processing.

[0307] IV. Dispatch Center and Manual Review System

[0308] (1) Responsible for the entire scheduling process of content flow, receiving and storing content through the upstream and downstream content interface servers, and then obtaining the content's metadata from the content database;

[0309] (2) Schedule the manual review system and the machine processing system, and control the scheduling order and priority;

[0310] (3) The content is enabled through the manual review system and then directly provided to the terminal content consumers through the content export distribution service (usually a recommendation engine or search engine or operation), that is, the content index information obtained by the consumer end;

[0311] (4) The manual review system is the carrier of human service capabilities. It is mainly used to review content that the machine cannot determine, and also to label and confirm the video content.

[0312] V. Content Storage Service

[0313] (1) Content entity information other than the metadata of the stored content, such as video source files and image source files of text and image content;

[0314] (2) Communicate with the content consumption terminal. The terminal directly downloads the corresponding file from the content storage server through the content access address for playback or displays the image.

[0315] VI. Content Processing Sample Library

[0316] Image features extracted from various atomic models in image processing, such as clarity scores, aesthetic scores, inappropriate image content subcategories, mosaic markers, watermark markers, and integrity markers. Corresponding atomic models include: clarity, aesthetics, stitched images, inappropriate images, image fingerprints, and image integrity. Identifying and modeling these image processing issues requires labeled samples. This section stores a small number of sample images needed for the final modeling stage, used to fine-tune the final model based on a large-scale pre-trained model.

[0317] VII. Content Processing Business Model and Services

[0318] (1) In the process of information flow image processing, there are various atomic capability models, generating atomic feature libraries of various images. The corresponding atomic models include: clarity, aesthetics, stitched image, unsuitable image, image fingerprint, image integrity, etc.

[0319] (2) At the same time, the business model obtained based on the large-scale pre-trained model and fine-tuning learning will be serviced and communicated with the scheduling center service to complete the generation of image atomic features;

[0320] (3) In the main process of information flow content distribution, the service of image processing from various sources is implemented, such as cover image selection, cover image cropping, cover image review, etc. It is necessary to call the content processing business model and service to obtain the results of various atomic feature recognition.

[0321] (4) Accept the scheduling of the internal dispatch center server, complete various processing of image content business applications, and save the processing results in the content database.

[0322] VIII. Multimodal Pre-trained Model for Text and Images

[0323] (1) Collect pre-trained image data from various channels according to the method described above, and perform necessary cleaning, filtering and enhancement processing on the data.

[0324] (2) Based on the sample data, the pre-trained model of the image and text multimodal model is constructed by combining self-supervision and weak supervision, in accordance with the pre-trained model structure and pre-training task construction method described above.

[0325] (3) Construct an image atomic service model. Based on this image pre-trained model, fine-tune it. The large model can be distilled to obtain the final model. During the fine-tuning process, a small amount of supervised sample data from actual business scenarios can be used for fine-tuning to accelerate the development of the model.

[0326] IX. Image and Text Pre-training Content Library

[0327] (1) Save the pre-training data corpus of the corresponding images crawled from the Internet. This mainly involves searching publicly available image data through search engines by searching keywords, and also includes sample data pairs collected through the various methods described above.

[0328] (2) Save the video frame data and metadata (classification, tag information) extracted from the video content by the main information distribution channel as the pre-training data corpus.

[0329] 10. Crawling and Data Preprocessing System

[0330] (1) Following the method described above, retrieve corresponding image data from the Internet by constructing search keywords from tags discovered through information flow content.

[0331] (2) Images with similar or identical labels can be considered similar, while other dissimilar images can be used to construct weakly supervised sample data for comparative machine learning. Here, the image and text sample pairs are mainly used to learn the alignment between the intrinsic information of the image and the semantic information of reality.

[0332] In summary, the training system for the information flow-based neural network model provided in this application has the following beneficial effects:

[0333] (1) It can provide a large-scale pre-trained model for general image and text matching (i.e., the first neural network model). Based on this, the business sample model can be fine-tuned, which can effectively reduce the amount of sample data and manual cost required for business model iteration, especially the need for supervised samples, effectively shorten the algorithm model development time, and well support the construction of a large number of content processing business models in information flow business.

[0334] (2) It can cope with the situation of partial modality missing. Whether it is a one-sided input text or image modality, it can achieve a good overall cost performance and expand the applicability of the pre-trained model.

[0335] The following description continues to illustrate the exemplary structure of the neural network model training device 233 provided in the embodiments of this application as a software module. In some embodiments, such as... Figure 2As shown, the software modules in the training device 233 of the neural network model stored in the memory 230 may include: an acquisition module 2331, used to acquire features of multiple modalities of an image; wherein the features of multiple modalities include image features, title text features, and label text features; a fusion module 2332, used to perform fusion processing based on image features and title text features to obtain a first fusion feature across modalities; a first determination module 2333, used to call the first neural network model to perform multiple single-modal prediction tasks based on the features of multiple modalities, obtain the corresponding single-modal prediction results, and determine the corresponding single-modal loss based on the prediction results within each modality; a second determination module 2334, used to call the first neural network model to perform multiple cross-modal prediction tasks based on the first fusion feature, obtain the corresponding cross-modal prediction results, and determine the corresponding cross-modal loss based on each cross-modal prediction result; and an update module 2335, used to perform backpropagation based on the single-modal loss and the cross-modal loss to update the parameters of the first neural network model.

[0336] In the above scheme, the acquisition module 2331 is used to perform data enhancement processing on the image to obtain an enhanced image; based on the image, a second neural network model is called to perform image feature extraction processing to obtain the first image feature corresponding to the image; based on the enhanced image, a second neural network model is called to perform image feature extraction processing to obtain the second image feature corresponding to the enhanced image; the title corresponding to the image is segmented into words, and based on the word segmentation result of the obtained title, a third neural network model is called to perform two text feature extraction processes to obtain the first title text feature and the second title text feature; wherein, when the third neural network model is called to perform text feature extraction processing, a first proportion of neurons in the third neural network model are discarded; the tag corresponding to the image is segmented into words, and based on the word segmentation result of the obtained tag, a third neural network model is called to perform text feature extraction processing to obtain the tag text feature.

[0337] In the above scheme, module 2331 is used to call the second neural network model to perform the following processing:

[0338] The image is divided into blocks, and linear projection is performed on the resulting image blocks to obtain multiple corresponding linear projection features. Each linear projection feature is summed with its corresponding location feature, and self-attention calculation is performed on the summation result to obtain the first self-attention feature corresponding to each linear projection feature. Fully connected computation is performed on the multiple first self-attention features to obtain the first image feature corresponding to the image.

[0339] In the above scheme, the acquisition module 2331 is used to call the third neural network model to perform the following processing: encode the multiple words included in the word segmentation result of the tag to obtain the context features corresponding to each word; wherein, the encoding processing includes multiple cascaded encoding processes; and determine the context features corresponding to the multiple words as the tag text features.

[0340] In the above scheme, when the single-modal prediction task is the image modality similarity prediction task, the features of multiple modalities include image features, which include first image features and second image features. The first image features and second image features are obtained by calling the second neural network model to perform image feature extraction processing based on the image and the enhanced image. When the single-modal prediction task is the image modality similarity prediction task, the first determining module 2333 is used to call the first neural network model based on the first image features and second image features to perform the image modality similarity prediction task, obtain the first predicted similarity of the first image features and second image features, and determine the first predicted similarity as the loss of the image modality.

[0341] In the above scheme, when the single-modal prediction task is the similarity prediction task of the title modality, the features of multiple modalities include title text features, which include first title text features and second title text features. The first title text features and second title text features are obtained by calling the third neural network model twice to perform text feature extraction processing based on the title corresponding to the image. When the single-modal prediction task is the similarity prediction task of the title modality, the first determining module 2333 is used to call the first neural network model based on the first title text features and second title text features to perform the similarity prediction task of the title modality, obtain the second predicted similarity of the first title text features and second title text features, and determine the second predicted similarity as the loss of the title modality.

[0342] In the above scheme, multiple cross-modal prediction tasks include: cross-modal similarity prediction of images and titles, cross-modal matching prediction of images and titles, cross-modal matching prediction of images, titles and tags, cross-modal multi-tag prediction of images and titles, and cross-modal masked language prediction of images and titles.

[0343] In the above scheme, when the cross-modal prediction task is the cross-modal similarity prediction task between an image and a title, the second determining module 2334 is used to generate multiple different combinations; wherein, each combination includes an image feature and a title text feature, the image feature is any one of the first image feature and the second image feature, and the title text feature is any one of the first title text feature and the second title text feature; the first image feature and the second image feature are obtained by calling the second neural network model to perform image feature extraction processing based on the image and the enhanced image; the first title text feature and the second title text feature are obtained by calling the third neural network model twice to perform text feature extraction processing based on the title corresponding to the image; the image features and title text features included in each combination are fused to obtain the first fused feature corresponding to each combination; any two first fused features are selected from multiple first fused features, and based on the selected two first fused features, the first neural network model is called to perform the cross-modal similarity prediction task between the image and the title to obtain the third predicted similarity of the two first fused features, and the third predicted similarity is determined as the first cross-modal loss.

[0344] In the above scheme, the second determining module 2334 is used to concatenate the image features and title text features included in each combination to obtain the first concatenated feature; and to downsample the first concatenated feature to obtain the first fused feature corresponding to each combination.

[0345] In the above scheme, when the cross-modal prediction task is a cross-modal matching prediction task between an image and a title, the second determining module 2334 is used to call the first neural network model based on the first fusion feature to perform the cross-modal matching prediction task between the image and the title, and obtain the first prediction result; wherein, the first prediction result represents the matching degree between the image feature corresponding to the first fusion feature and the title text feature; the first prediction result and the corresponding first label value are substituted into the first loss function for calculation and processing to obtain the second cross-modal loss.

[0346] In the above scheme, the second determining module 2334 is used to determine a second number of first fusion features from a first number of first fusion features according to a second ratio; replace the second number of first fusion features with corresponding image features; call a first neural network model based on each image feature to perform a cross-modal matching prediction task between the image and the title, and obtain a second prediction result; wherein the second prediction result represents the probability that the image feature corresponds to the predicted title text; substitute the second prediction result and the first label value into a first loss function for calculation and processing to obtain a second cross-modal loss.

[0347] In the above scheme, when the cross-modal prediction task is a cross-modal matching prediction task of image, title and label, the second determining module 2334 is used to call the first neural network model based on the first fusion feature and the first label text feature to perform the cross-modal matching prediction task of image, title and label, and obtain the third prediction result; wherein, the third prediction result is the matching degree of the first fusion feature and the first label text feature, and the first label text feature is obtained by downsampling the label text feature; the third prediction result and the corresponding second label value are substituted into the second loss function for calculation to obtain the third cross-modal loss.

[0348] In the above scheme, when the cross-modal prediction task is a cross-modal multi-label prediction task of image and title, the second determining module 2334 is used to call the first neural network model based on the first fusion feature to perform the cross-modal multi-label prediction task of image and title, and obtain multiple fourth prediction results; wherein, each fourth prediction result represents the probability of the first fusion feature corresponding to a prediction label; each fourth prediction result and the corresponding third label value are substituted into the third loss function for calculation to obtain the cross-modal loss corresponding to each fourth prediction result; the cross-modal losses corresponding to each fourth prediction result are summed and averaged to obtain the fourth cross-modal loss.

[0349] In the above scheme, the fusion module 2332 is used to perform masking processing on the title corresponding to the image to obtain a masked title; wherein, the masking processing includes at least one of the following: character-level masking processing, word-level masking processing, and character-word mixed-level masking processing; the masked title is segmented into words, and based on the segmentation results of the obtained masked title, a third neural network model is called to perform text feature extraction processing to obtain the masked title text features; the image features and the masked title text features are concatenated to obtain a second concatenated feature; the second concatenated feature is downsampled to obtain a cross-modal first fusion feature.

[0350] In the above scheme, when the cross-modal prediction task is a cross-modal masked language prediction task of image and title, the second determining module 2334 is used to call the first neural network model based on the first fusion feature to perform the cross-modal masked language prediction task of image and title and obtain the fifth prediction result; wherein, the fifth prediction result is the probability of the predicted text corresponding to the masked part in the title after masking; the fifth prediction result and the corresponding fourth label value are substituted into the fourth loss function for calculation and processing to obtain the fifth cross-modal loss.

[0351] In the above scheme, the above device further includes a computational processing module, which is used to call the first neural network model to perform the following processing: perform self-attention computation processing based on the first fusion feature to obtain the second self-attention feature; perform fully connected computation processing based on the second self-attention feature to obtain the corresponding fully connected feature; and perform normalization computation processing on the fully connected feature to obtain the corresponding prediction results for multiple cross-modal features.

[0352] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the neural network model training method described above in this application.

[0353] This application provides a computer-readable storage medium storing executable instructions, wherein the executable instructions are stored and, when executed by a processor, will cause the processor to execute the training method of the neural network model provided in this application.

[0354] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0355] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0356] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0357] As an example, executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0358] In summary, the embodiments of this application can perform training tasks of various modalities based on the first neural network model, and update the parameters of the first neural network model based on the loss values ​​corresponding to various training tasks. This enables the first neural network model to learn sufficiently, allowing the trained first neural network model to support multiple application scenarios, thereby expanding the applicability of the trained first neural network model, effectively improving the modeling efficiency of the neural network model, and reducing the modeling cost.

[0359] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A method for training a neural network model, characterized in that, The method includes: The features of multiple modalities of an image are obtained; wherein the features of the multiple modalities include image features, title text features, and label text features; The title corresponding to the image is segmented into words. Based on the segmentation result of the title, a third neural network model is called to perform two text feature extraction processes to obtain the first title text feature and the second title text feature. When the third neural network model is called to perform the text feature extraction process, a first proportion of neurons in the third neural network model are discarded. Based on the image features and the title text features, a first cross-modal fusion feature is obtained; Based on the features of the multiple modalities, the first neural network model is invoked to perform multiple single-modal prediction tasks, obtain the corresponding single-modal prediction results, and determine the corresponding single-modal loss based on the prediction results in each modality. Based on the first fusion feature, the first neural network model is invoked to perform multiple cross-modal prediction tasks. When the cross-modal prediction task is a cross-modal matching prediction task between the image and the title, the following processing is performed: According to a second ratio, a second number of first fusion features are determined from a first number of first fusion features; the second number of first fusion features are replaced with the corresponding image features; the first neural network model is called based on each image feature to perform a cross-modal matching prediction task between the image and the title, and a second prediction result is obtained; the second prediction result and the first label value are substituted into the first loss function for calculation and processing to obtain the second cross-modal loss; When the cross-modal prediction task is a cross-modal matching prediction task of the image, the title, and the label, the following processing is performed: Based on the first fusion feature and the first tag text feature, the first neural network model is invoked to perform a cross-modal matching prediction task of the image, the title and the tag, and a third prediction result is obtained; wherein, the third prediction result is the matching degree between the first fusion feature and the first tag text feature, the first tag text feature is obtained by downsampling the tag text feature, and the dimension of the first tag text feature is the same as the dimension of the first fusion feature. The third prediction result and the corresponding second label value are substituted into the second loss function for calculation to obtain the third cross-modal loss. Backpropagation is performed based on the single-modal loss, the second cross-modal loss, and the third cross-modal loss to update the parameters of the first neural network model.

2. The method according to claim 1, characterized in that, The acquisition of features of multiple modalities of the image includes: The image is subjected to data augmentation processing to obtain an enhanced image; Based on the image, a second neural network model is invoked to perform image feature extraction processing to obtain the first image feature corresponding to the image; Based on the enhanced image, the second neural network model is invoked to perform the image feature extraction process to obtain the second image features corresponding to the enhanced image; The image is segmented into words based on its corresponding tags. The third neural network model is then invoked to extract the text features of the tags based on the segmentation results.

3. The method according to claim 1, characterized in that, When the single-modal prediction task is an image modality similarity prediction task, the features of the multiple modalities include the image features, which include a first image feature and a second image feature. The first image feature and the second image feature are obtained by calling a second neural network model to perform image feature extraction processing based on the image and the enhanced image. When the single-modal prediction task is the image modality similarity prediction task, the step of calling the first neural network model based on the features of the multiple modalities to perform multiple single-modal prediction tasks, obtaining the corresponding prediction results of the multiple single modalities, and determining the corresponding single-modal loss based on the prediction results within each modality includes: Based on the first image features and the second image features, the first neural network model is invoked to perform the image modality similarity prediction task, and a first predicted similarity between the first image features and the second image features is obtained. The first predicted similarity is then determined as the loss of the image modality.

4. The method according to claim 1, characterized in that, When the single-modal prediction task is a title modality similarity prediction task, the features of the multiple modalities include the title text features, which include a first title text feature and a second title text feature. The first title text feature and the second title text feature are obtained by calling a third neural network model to perform two text feature extraction processes based on the title corresponding to the image. When the single-modal prediction task is the similarity prediction task of the title modality, the step of calling the first neural network model based on the features of the multiple modalities to perform multiple single-modal prediction tasks, obtaining the corresponding prediction results of the multiple single modalities, and determining the corresponding single-modal loss based on the prediction results within each modality includes: Based on the first title text features and the second title text features, the first neural network model is invoked to perform the title modality similarity prediction task, and a second predicted similarity between the first title text features and the second title text features is obtained, and the second predicted similarity is determined as the loss of the title modality.

5. The method according to claim 1, characterized in that, The method further includes: Based on the first fusion feature, the first neural network model is invoked to perform multiple cross-modal prediction tasks. When the cross-modal prediction task is the cross-modal similarity prediction task between the image and the title, the following processing is performed: Generate multiple different combinations; Each of the combinations includes one image feature and one title text feature. The image feature is either a first image feature or a second image feature, and the title text feature is either a first title text feature or a second title text feature. The first image feature and the second image feature are obtained by performing image feature extraction processing using a second neural network model based on the image and the enhanced image. The first title text feature and the second title text feature are obtained by performing text feature extraction processing using a third neural network model twice based on the title corresponding to the image. Based on the image features and title text features included in each combination, a fusion process is performed to obtain a first fusion feature corresponding to each combination; Select any two of the first fusion features from a plurality of first fusion features, and based on the selected two first fusion features, call the first neural network model to perform the cross-modal similarity prediction task between the image and the title, obtain the third predicted similarity between the two first fusion features, and determine the third predicted similarity as the first cross-modal loss.

6. The method according to claim 1, characterized in that, The method further includes: The first neural network model is invoked based on the first fusion feature to perform multiple cross-modal prediction tasks. When the cross-modal prediction task is a cross-modal matching prediction task between the image and the title, the following processing is performed: Based on the first fusion feature, the first neural network model is invoked to perform a cross-modal matching and prediction task between the image and the title, and a first prediction result is obtained; wherein, the first prediction result represents the degree of matching between the image feature corresponding to the first fusion feature and the title text feature; The first prediction result and the corresponding first label value are substituted into the first loss function for calculation to obtain the second cross-modal loss.

7. The method according to claim 1, characterized in that, The method further includes: The first neural network model is invoked based on the first fusion feature to perform multiple cross-modal prediction tasks. When the cross-modal prediction task is a cross-modal multi-label prediction task between the image and the title, the following processing is performed: Based on the first fusion feature, the first neural network model is invoked to perform a cross-modal multi-label prediction task between the image and the title, resulting in multiple fourth prediction results; wherein each fourth prediction result represents the probability of the first fusion feature corresponding to a prediction label; Substitute each of the fourth prediction results and the corresponding third label value into the third loss function for calculation to obtain the cross-modal loss corresponding to each of the fourth prediction results; The cross-modal losses corresponding to each of the fourth prediction results are summed and averaged to obtain the fourth cross-modal loss.

8. The method according to claim 1, characterized in that, The fusion process based on the image features and the title text features to obtain the first cross-modal fusion feature includes: The title corresponding to the image is masked to obtain a masked title; wherein the masking process includes at least one of the following: character-level masking, word-level masking, and mixed character-word masking. The blocked title is segmented into words, and based on the segmentation results, a third neural network model is called to extract text features to obtain the text features of the blocked title. The image features and the masked title text features are concatenated to obtain a second concatenated feature; The second splicing feature is downsampled to obtain the first fusion feature across modalities.

9. The method according to claim 8, characterized in that, The method further includes: The first neural network model is invoked based on the first fusion feature to perform multiple cross-modal prediction tasks. When the cross-modal prediction task is a cross-modal masked language prediction task between the image and the title, the following processing is performed: Based on the first fusion feature, the first neural network model is invoked to perform a cross-modal masked language prediction task between the image and the title, and a fifth prediction result is obtained; wherein, the fifth prediction result is the probability of the predicted text corresponding to the masked part in the masked title; The fifth prediction result and the corresponding fourth label value are substituted into the fourth loss function for calculation to obtain the fifth cross-modal loss.

10. A training device for a neural network model, characterized in that, The device includes: The acquisition module is used to acquire features of multiple modalities of an image; wherein the features of the multiple modalities include image features, title text features, and label text features; The acquisition module is used to perform word segmentation on the title corresponding to the image, and call the third neural network model to perform two text feature extraction processes based on the obtained word segmentation results of the title to obtain the first title text feature and the second title text feature; wherein, when the third neural network model is called to perform the text feature extraction process, a first proportion of neurons in the third neural network model are discarded; The fusion module is used to perform fusion processing based on the image features and the title text features to obtain a first fusion feature across modalities; The first determining module is used to call the first neural network model based on the features of the multiple modalities to perform multiple single-modal prediction tasks, obtain the corresponding single-modal prediction results, and determine the corresponding single-modal loss based on the prediction results in each modality. The second determining module is used to call the first neural network model based on the first fusion feature to perform multiple cross-modal prediction tasks. When the cross-modal prediction task is a cross-modal matching prediction task between the image and the title, the following processing is performed: determining a second number of first fusion features from a first number of first fusion features according to a second ratio; replacing the second number of first fusion features with the corresponding image features; calling the first neural network model based on each image feature to perform a cross-modal matching prediction task between the image and the title to obtain a second prediction result; and substituting the second prediction result and the first label value into a first loss function for calculation to obtain a second cross-modal loss. The second determining module is used to call the first neural network model based on the first fusion feature to perform multiple cross-modal prediction tasks. When the cross-modal prediction task is a cross-modal matching prediction task of the image, the title, and the label, the following processing is performed: the first neural network model is called based on the first fusion feature and the first label text feature to perform a cross-modal matching prediction task of the image, the title, and the label, and a third prediction result is obtained; wherein, the third prediction result is the matching degree between the first fusion feature and the first label text feature, the first label text feature is obtained by downsampling the label text feature, and the dimension of the first label text feature is the same as the dimension of the first fusion feature; the third prediction result and the corresponding second label value are substituted into the second loss function for calculation to obtain the third cross-modal loss. An update module is used to perform backpropagation based on the single-modal loss, the second cross-modal loss, and the third cross-modal loss to update the parameters of the first neural network model.

11. The apparatus according to claim 10, characterized in that, The acquisition module is also used for: The image is subjected to data augmentation processing to obtain an enhanced image; Based on the image, a second neural network model is invoked to perform image feature extraction processing to obtain the first image feature corresponding to the image; Based on the enhanced image, the second neural network model is invoked to perform the image feature extraction process to obtain the second image features corresponding to the enhanced image; The image is segmented into words based on its corresponding tags. The third neural network model is then invoked to extract the text features of the tags based on the segmentation results.

12. The apparatus according to claim 10, characterized in that, When the single-modal prediction task is an image modality similarity prediction task, the features of the multiple modalities include the image features, which include a first image feature and a second image feature. The first image feature and the second image feature are obtained by calling a second neural network model to perform image feature extraction processing based on the image and the enhanced image. When the single-modal prediction task is the image modality similarity prediction task, the first determining module is further configured to: Based on the first image features and the second image features, the first neural network model is invoked to perform the image modality similarity prediction task, and a first predicted similarity between the first image features and the second image features is obtained. The first predicted similarity is then determined as the loss of the image modality.

13. The apparatus according to claim 10, characterized in that, When the single-modal prediction task is a title modality similarity prediction task, the features of the multiple modalities include the title text features, which include a first title text feature and a second title text feature. The first title text feature and the second title text feature are obtained by calling a third neural network model to perform two text feature extraction processes based on the title corresponding to the image. When the prediction task of the single modality is the similarity prediction task of the title modality, the first determining module is further configured to: Based on the first title text features and the second title text features, the first neural network model is invoked to perform the title modality similarity prediction task, and a second predicted similarity between the first title text features and the second title text features is obtained, and the second predicted similarity is determined as the loss of the title modality.

14. An electronic device, characterized in that, The electronic device includes: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the training method of the neural network model according to any one of claims 1 to 9.

15. A computer-readable storage medium, characterized in that, The device stores executable instructions that, when executed by a processor, implement the training method for the neural network model according to any one of claims 1 to 9.

16. A computer program product comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the training method of the neural network model according to any one of claims 1 to 9.

Images