Label identification model training method based on artificial intelligence and label identification method

Abstract

The invention provides a tag identification model training method based on artificial intelligence, a tag identification method and device based on artificial intelligence, electronic equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: carrying out feature interaction processing on content features of extracted sample content and tag features of each candidate tag; performing matching probability prediction processing according to the interaction characteristics of each candidate tag; determining the co-occurrence probability of different candidate tags according to the interaction feature of each candidate tag, and determining the pseudo tag of the sample content according to the co-occurrence probability of different candidate tags and the sample tag of the sample content; and training a label identification model according to the matching probability of each candidate label, the sample label of the sample content, the pseudo label of the sample content and the co-occurrence probability of different candidate labels. According to the method and the device, the model training effect of the label identification model and the model reasoning effect based on the trained label identification model can be improved.

Description

Technical Field

[0001] The present application relates to artificial intelligence technology, and in particular to an artificial intelligence-based label recognition model training method, an artificial intelligence-based label recognition method, device, electronic device, computer-readable storage medium and computer program product. Background Art

[0002] Content products are a form of product that provides services in the form of media such as graphics, video / live broadcast, and audio. The design chain of content products involves content production, content review, content distribution and other links. During the content review link, the content needs to be labeled for better content distribution. Currently, machine labeling is usually used to reduce the cost of manual content review.

[0003] In the solutions provided by related technologies, a label recognition model is usually trained based on a certain scale of labeled data sets (including several pieces of content and manually annotated sample labels for each piece of content), and machine labeling is performed using the trained label recognition model. However, due to the diversity of content itself and the huge size of the label library (which may contain millions or even tens of millions of labels), it is difficult to fully annotate all the labels of a product manually, resulting in poor model training and model inference performance of the label recognition model. Summary of the Invention

[0004] The present application provides an artificial intelligence-based label recognition model training method, an artificial intelligence-based label recognition method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product, which can improve the model training effect of the label recognition model and the model reasoning effect of the trained label recognition model.

[0005] The technical solution of this application is achieved as follows:

[0006] This application provides an artificial intelligence-based label recognition model training method, including:

[0007] Obtaining sample content and a sample label of the sample content; wherein the sample label is one of a plurality of candidate labels;

[0008] Performing content feature extraction processing on the sample content through a content feature extraction network in a tag recognition model to obtain content features of the sample content;

[0009] Perform label feature extraction on each candidate label through the label feature extraction network in the label recognition model to obtain the label features of each candidate label;

[0010] Performing feature interaction processing on the content features of the sample content and the tag features of each candidate tag through a feature interaction network in the tag recognition model to obtain an interaction feature of each candidate tag;

[0011] The matching probability prediction network in the tag recognition model performs matching probability prediction processing based on the interaction characteristics of each candidate tag to obtain the matching probability between each candidate tag and the sample content;

[0012] Determining the co-occurrence probability of different candidate tags according to the interaction feature of each candidate tag, and determining the pseudo label of the sample content according to the co-occurrence probability of different candidate tags and the sample label of the sample content;

[0013] The tag recognition model is trained according to the matching probability of each candidate tag, the sample tag of the sample content, the pseudo tag of the sample content, and the co-occurrence probability of different candidate tags.

[0014] This application provides an artificial intelligence-based tag recognition method, including:

[0015] Obtain the content to be identified and multiple candidate labels;

[0016] Performing content feature extraction processing on the content to be identified by a content feature extraction network in a tag recognition model to obtain content features of the content to be identified;

[0017] Perform label feature extraction on each candidate label through the label feature extraction network in the label recognition model to obtain the label features of each candidate label;

[0018] Performing feature interaction processing on the content features of the content to be identified and the tag features of each candidate tag through a feature interaction network in the tag recognition model to obtain an interaction feature of each candidate tag;

[0019] The matching probability prediction network in the tag recognition model performs matching probability prediction processing based on the interaction characteristics of each candidate tag to obtain the matching probability between each candidate tag and the content to be recognized;

[0020] Determine the candidate tag with a matching probability greater than a matching probability threshold as the target tag that matches the content to be identified;

[0021] Among them, the label recognition model is trained according to the label recognition model training method based on artificial intelligence.

[0022] This application provides an artificial intelligence-based label recognition model training device, comprising:

[0023] A first acquisition module is configured to acquire sample content and a sample label of the sample content; wherein the sample label is one of a plurality of candidate labels;

[0024] a first content feature extraction module, configured to perform content feature extraction processing on the sample content through a content feature extraction network in a tag recognition model to obtain content features of the sample content;

[0025] A first label feature extraction module is used to perform label feature extraction processing on each candidate label through the label feature extraction network in the label recognition model to obtain the label feature of each candidate label;

[0026] A first feature interaction module is configured to perform feature interaction processing on the content features of the sample content and the tag features of each candidate tag through a feature interaction network in the tag recognition model to obtain an interaction feature of each candidate tag;

[0027] A first matching probability prediction module is configured to perform matching probability prediction processing based on the interaction features of each candidate tag through a matching probability prediction network in a tag recognition model to obtain a matching probability between each candidate tag and the sample content;

[0028] a pseudo-label screening module, configured to determine the co-occurrence probability of different candidate labels based on the interaction characteristics of each candidate label, and determine the pseudo-label of the sample content based on the co-occurrence probability of different candidate labels and the sample label of the sample content;

[0029] The training module is used to train a tag recognition model according to the matching probability of each candidate tag, the sample tag of the sample content, the pseudo-tag of the sample content, and the co-occurrence probability of different candidate tags.

[0030] This application provides an artificial intelligence-based label recognition device, comprising:

[0031] The second acquisition module is used to obtain the content to be identified and multiple candidate tags;

[0032] A second content feature extraction module is configured to perform content feature extraction processing on the content to be identified through a content feature extraction network in a tag recognition model to obtain content features of the content to be identified;

[0033] The second label feature extraction module is used to perform label feature extraction processing on each candidate label through the label feature extraction network in the label recognition model to obtain the label feature of each candidate label;

[0034] A second feature interaction module is configured to perform feature interaction processing on the content features of the content to be identified and the tag features of each candidate tag through a feature interaction network in the tag recognition model to obtain an interaction feature of each candidate tag;

[0035] A second matching probability prediction module is configured to perform matching probability prediction processing based on the interaction features of each candidate tag through a matching probability prediction network in the tag recognition model to obtain a matching probability between each candidate tag and the content to be recognized;

[0036] A target tag screening module is used to determine a candidate tag with a matching probability greater than a matching probability threshold as a target tag matching the content to be identified;

[0037] Among them, the label recognition model is trained according to the label recognition model training method based on artificial intelligence.

[0038] The present application provides an electronic device, including:

[0039] a memory for storing executable instructions;

[0040] The processor is used to implement the artificial intelligence-based label recognition model training method or artificial intelligence-based label recognition method provided in this application when executing the executable instructions stored in the memory.

[0041] The present application provides a computer-readable storage medium storing executable instructions for causing a processor to execute instructions to implement the artificial intelligence-based label recognition model training method or artificial intelligence-based label recognition method provided in the present application.

[0042] The present application provides a computer program product, which includes executable instructions for causing a processor to execute and implement the artificial intelligence-based label recognition model training method or artificial intelligence-based label recognition method provided in the present application.

[0043] This application has the following beneficial effects:

[0044] In the model training phase of the label recognition model provided in the present application, the sample content and the sample label of the sample content are first obtained, and the sample content is subjected to content feature extraction processing by the content feature extraction network in the label recognition model, thereby extracting content features that can represent the overall semantic information of the sample content. At the same time, the label feature extraction network in the label recognition model is used to perform label feature extraction processing on each candidate label to obtain the label features of each candidate label. Then, the content features of the sample content and the label features of each candidate label are subjected to feature interaction processing by the feature interaction network in the label recognition model to obtain the interaction features of each candidate label. In this way, the overall semantic information of the sample content can be decoupled from each candidate label, so that the interaction features combine the semantic information of the sample content and the semantic information of the candidate label, so that the matching probability between each candidate label and the sample content can be predicted separately according to the interaction features of each candidate label, thereby improving the accuracy of the prediction. In addition, the co-occurrence probability of different candidate labels is determined based on the interactive features of each candidate label, and the pseudo-label of the sample content is determined based on the co-occurrence probability of different candidate labels and the sample label of the sample content. In this way, the sample label (i.e., pseudo-label) of the sample that is omitted is predicted by the co-occurrence probability, thereby supplementing and correcting the sample label. Finally, the label recognition model is trained based on the matching probability of each candidate label, the sample label of the sample content, the pseudo-label of the sample content, and the co-occurrence probability of different candidate labels. Since the label recognition ability of the label recognition model can be reflected by the matching probability of each candidate label, the sample label of the sample content, and the pseudo-label of the sample content, and the co-occurrence probability of different candidate labels and the sample label of the sample content can reflect the co-occurrence relationship prediction ability of the label recognition model, the above training method can seek a balance between the label recognition ability and the co-occurrence relationship prediction ability, improve the model training effect, and then improve the label recognition effect (i.e., model inference effect) of the trained label recognition model. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0046] Figure 1 This is a schematic diagram of the architecture of the artificial intelligence-based tag recognition system provided in an embodiment of the present application;

[0047] Figure 2A This is a first structural diagram of a server provided in an embodiment of the present application;

[0048] Figure 2BThis is a second structural diagram of the server provided in an embodiment of the present application;

[0049] Figure 3A This is a first flow chart of the artificial intelligence-based label recognition model training method provided in an embodiment of the present application;

[0050] Figure 3B This is a second flow chart of the artificial intelligence-based label recognition model training method provided in an embodiment of the present application;

[0051] Figure 3C This is a third flow chart of the artificial intelligence-based label recognition model training method provided in an embodiment of the present application;

[0052] Figure 3D This is a fourth flow chart of the artificial intelligence-based label recognition model training method provided in an embodiment of the present application;

[0053] Figure 4 This is a flow chart of the artificial intelligence-based tag recognition method provided in an embodiment of the present application;

[0054] Figure 5 This is a flowchart from content production to content review and then to content distribution provided by an embodiment of the present application;

[0055] Figure 6 This is a flow chart of the model training phase of the tag recognition model provided in the embodiment of the present application;

[0056] Figure 7 It is a flowchart of the multimodal feature extraction process provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0058] In the following description, references to "some embodiments" describe a subset of all possible embodiments. However, it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict. In the following description, the term "plurality" refers to at least two.

[0059] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0061] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0062] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0063] 1) Artificial Intelligence (AI): It is the theory, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0064] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, pre-trained models, operating / interaction systems, and mechatronics. Pre-trained models, also known as large models or basic models, can be fine-tuned and widely applied to downstream tasks across various AI disciplines. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0065] In the embodiment of the present application, the tag recognition model involved is a model constructed based on the principles of artificial intelligence.

[0066] 2) Computer Vision (CV): Computer vision is the science of enabling machines to "see." Specifically, it refers to the use of cameras and computers to replace the human eye in identifying, tracking, and measuring objects. Further image processing is performed to transform the computer's image into something more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, aiming to build artificial intelligence systems capable of extracting information from images or multidimensional data. Large model technology has revolutionized the development of computer vision. Pre-trained models in the field of vision, such as the Swin Transformer, ViT, V-MOE, and MAE, can be fine-tuned to quickly and broadly apply to specific downstream tasks. Computer vision technologies generally include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / action recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and other technologies. Common biometric recognition technologies include facial recognition and fingerprint recognition.

[0067] In an embodiment of the present application, computer vision technology can be used to extract image modality content from specific content (referring to sample content or content to be identified), and image feature extraction processing can also be performed on the image modality content.

[0068] 3) Speech Technology: Key technologies include automatic speech recognition (ASR), text-to-speech (TTS), and voiceprint recognition. Speech technology enables computers to hear, see, speak, and feel, and is the future direction of human-computer interaction. Speech is one of the most promising methods of human-computer interaction. Large model technology has revolutionized the development of speech technology. Pre-trained models such as WavLM and UniSpeech, which use the Transformer architecture, have strong generalization and versatility, and can effectively handle a wide range of speech processing tasks.

[0069] In an embodiment of the present application, voice technology can be used to extract content in voice mode from specific content (referring to sample content or content to be identified), and further convert the content in voice mode into content in text mode for text feature extraction processing; alternatively, voice feature extraction processing can be performed directly on the content in voice mode.

[0070] 4) Natural Language Processing (NLP): This is an important field in the fields of computer science and artificial intelligence. It studies various theories and methods that enable effective communication between humans and computers using natural language. Natural language processing involves natural language, which is the language people use in daily life, and is closely related to linguistics research; it also involves computer science and mathematics. Similarly, large model technology can be used in the field of NLP. For example, pre-trained models in the field of NLP, such as large language models (LLM), can be quickly and widely applied to specific downstream tasks after fine-tuning. Natural language processing technology generally includes text processing, semantic understanding, machine translation, robot question answering, knowledge graphs, and other technologies.

[0071] In an embodiment of the present application, natural language processing technology can be used to extract text modal content from specific content (referring to sample content or content to be identified), and text feature extraction processing can also be performed on the text modal content.

[0072] 5) Machine Learning (ML): This is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning. Pretrained models are the latest development in deep learning, integrating these techniques.

[0073] In an embodiment of the present application, the training of the label recognition model can be implemented based on the principle of machine learning. For example, in the model training stage of the label recognition model, the loss value is calculated by the loss function, the loss value is back-propagated (BP) in the label recognition model, and the model parameters of the label recognition model are updated along the gradient descent direction during the back-propagation process. The model parameters may include weight parameters (usually written as w) and bias parameters (usually written as b).

[0074] 6) Pre-training Model (PTM): Also known as a cornerstone model or large model, this refers to a large-parameter deep neural network (DNN). It is trained on massive amounts of unlabeled data. Leveraging the function approximation capabilities of large-parameter DNNs, the PTM extracts common features from the data. Through techniques such as fine tuning, efficient parameter fine tuning (PEFT), and prompt-tuning, it is then adapted for downstream tasks. Therefore, pre-trained models can achieve ideal results in few-shot or zero-shot scenarios. Based on the data modality processed, PTMs can be categorized into language models (ELMO, BERT, GPT), vision models (swin-transformer, ViT, V-MOE), speech models (VALL-E), and multimodal models (ViBERT, CLIP, Flamingo, Gato). Pre-trained models are a crucial tool for AIGC and can also serve as a universal interface for connecting multiple task-specific models.

[0075] In an embodiment of the present application, the label recognition model used (or a specific network layer in the label recognition model) can be a pre-trained model, which can improve the training efficiency, that is, a better model inference effect can be achieved after a small amount of training.

[0076] 7) Content: Also known as media content, used as input to the tag recognition model in the embodiments of this application, can have at least one modality. It is worth noting that in the field of machine learning, modality refers to the source or form of information. It can be understood as different data input methods or data processing methods. For example, humans have multiple sensory modes such as touch, hearing, vision, and smell, which can all be considered different modalities. In terms of information media, voice, images, text, etc. are also different modalities.

[0077] The content lifecycle involves content production, content review, and content distribution. Content production refers to the process of creating and providing new content; content review refers to further processing of generated content, such as tagging, summary extraction, semantic analysis, content deduplication, content analysis, error correction, and image matching, among other algorithmic processes, to better organize, store, and recommend content; and content distribution refers to the process of distributing reviewed content to users. Tags assigned to content during the content review process can be used to achieve precise push notifications.

[0078] 8) Label: Also known as category. The essence of label recognition is to predict the label that matches specific content from a number of pre-set candidate labels. Different candidate labels can be divided by certain specific attributes or features.

[0079] The embodiments of this application involve candidate labels, sample labels, pseudo labels, and target labels. A candidate label refers to a pre-set label, such as a label in a label library; a sample label refers to a candidate label annotated (e.g., manually annotated) for the sample content; a pseudo label is a predicted sample label that should have been annotated but was omitted; and a target label is a predicted candidate label that matches the content to be identified.

[0080] The embodiments of the present application provide an artificial intelligence-based label recognition model training method, an artificial intelligence-based label recognition method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product, which can improve the model training effect of the label recognition model and the model reasoning effect of the trained label recognition model. The following describes an exemplary application of the electronic device provided by the embodiment of the present application. The electronic device provided by the embodiment of the present application can be implemented as various types of terminal devices or as a server.

[0081] See also Figure 1 , Figure 1 This is an architectural diagram of an artificial intelligence-based tag recognition system 100 provided in an embodiment of the present application. The terminal device 400 is connected to the server 200 through the network 300, and the server 200 is connected to the database 500. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0082] In some embodiments, taking the electronic device as a server as an example, the artificial intelligence-based label recognition model training method and the artificial intelligence-based label recognition method provided in the embodiments of the present application can both be implemented by the server. For example, the server 200 can obtain sample content and sample labels of the sample content from the database 500, and train the label recognition model based on the sample content and the sample labels. The storage location of the sample content and the sample labels is not limited to the database 500, and can also be stored in a distributed file system, blockchain, or other location of the server 200.

[0083] After completing the training of the tag recognition model, the server 200 can obtain the content to be recognized (such as content that is not labeled with a sample label) in the database 500, predict the target label that matches the content to be recognized through the trained tag recognition model, and store the target label in the database 500, that is, label the content to be recognized in the database 500 with the target label. The above method is suitable for scenarios where content management needs to be performed on the server 200, for example, the server 200 is the backend server of the content platform. By labeling the content to be recognized with the target label, the content to be recognized can be pushed to users who are interested in the target label (referring to users of the content platform).

[0084] Alternatively, the server 200 can obtain the content to be identified sent by the terminal device 400, predict the target tag that matches the content to be identified through the trained tag recognition model, and send the target tag to the terminal device 400 so that the terminal device 400 can locally tag the content to be identified. The above method is suitable for scenarios where content management is required on the terminal device 400. For example, the terminal device 400 tags the local content to be identified (such as local videos, images, recordings, etc.) with target tags, so that the content to be identified can be classified by target tags, such as dividing multiple content to be identified with the same target tag into the same directory for users (referring to users of the terminal device 400) to view; or, it can support users to search for content to be identified with the target tag through target tags. It is worth noting that the above method uses the computing power of the server 200 to predict the target tag, which can improve the efficiency of tag recognition and reduce the local resource consumption of the terminal device 400.

[0085] In some embodiments, taking the electronic device as a terminal device as an example, the artificial intelligence-based label recognition model training method and the artificial intelligence-based label recognition method provided in the embodiments of the present application can be implemented by the terminal device. For example, the terminal device 400 can obtain sample content and sample labels of the sample content from a local or other storage location, and train a label recognition model based on the sample content and sample labels. Then, the terminal device 400 can predict the target label that matches the content to be recognized using the trained label recognition model.

[0086] In some embodiments, the artificial intelligence-based tag recognition model training method provided in the embodiments of the present application can be implemented by a server, and the artificial intelligence-based tag recognition method provided in the embodiments of the present application can be implemented by a terminal device. For example, the server 200 can send the trained tag recognition model to the terminal device 400, so that the terminal device 400 can deploy the trained tag recognition model locally. In this way, the computing power of the server 200 can be used to improve the efficiency of model training; after the terminal device 400 locally deploys the trained tag recognition model, it can provide model reasoning capabilities locally.

[0087] In some embodiments, the terminal device 400 or the server 200 can implement the artificial intelligence-based label recognition model training method and the artificial intelligence-based label recognition method provided in the embodiments of the present application by running a computer program. For example, the computer program can be a native program or software module in the operating system; it can be a local (Native) application (APP, Application), that is, a program that needs to be installed in the operating system to run; it can also be a small program, that is, a program that can be run only by downloading it into a browser environment; it can also be a small program that can be embedded in any APP, and the small program can be controlled by the user to run or close. In short, the above-mentioned computer program can be an application, module or plug-in in any form.

[0088] In some embodiments, the server 200 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms and other basic cloud computing services. The terminal device 400 can be a smart phone, a tablet computer, a laptop computer, a desktop computer, an intelligent voice interaction device, a smart home appliance, a vehicle-mounted terminal, an aircraft, etc., but is not limited thereto. The terminal device 400 and the server 200 can be directly or indirectly connected by wired or wireless communication, which is not limited in the embodiments of the present application.

[0089] The embodiments of the present application can be applied to content platforms. By machine-tagging multiple pieces of content to be identified on the content platform, the accuracy of tag recognition can be improved, while reducing the manpower and time costs of the tagging process. This makes the content production-content review-content distribution link more efficient and accurate, and helps optimize the content management ecosystem. There is no limitation on the content platform, and it can be, for example, an online video platform, a news platform, a self-media platform, a short video platform, etc.

[0090] The embodiments of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc.

[0091] In some embodiments, various data involved in the embodiments of the present application (such as various contents, tags, model parameters, etc.) can be stored in the blockchain, and the data credibility can be guaranteed based on the tamper-proof characteristics of the blockchain.

[0092] Take the electronic device provided in the embodiment of the present application as an example, which is a server. Figure 2A , Figure 2Ais a structural diagram of the server 200 provided in an embodiment of the present application, Figure 2A The server 200 shown includes: at least one processor 210, a memory 250 and at least one network interface 220. The various components in the server 200 are coupled together via a bus system 240. It is understood that the bus system 240 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 240 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 240 is not described in detail. Figure 2A Various buses are labeled as bus system 240 .

[0093] The processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0094] The memory 250 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 250 may optionally include one or more storage devices that are physically remote from the processor 210.

[0095] The memory 250 includes volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 250 described in the embodiments of the present application is intended to include any suitable type of memory.

[0096] In some embodiments, the memory 250 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0097] Operating system 251, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, which are used to implement various basic services and process hardware-based tasks;

[0098] A network communication module 252 for reaching other computing devices via one or more (wired or wireless) network interfaces 220 , exemplary network interfaces 220 including Bluetooth, Wireless LAN (WiFi), and Universal Serial Bus (USB);

[0099] In some embodiments, the artificial intelligence-based cultural graph model training device provided in the embodiments of the present application can be implemented in a software manner. Figure 2A The AI-based text graph model training device 255 stored in memory 250 is shown. This device can be software in the form of a program or plug-in, and includes the following software modules: a first acquisition module 2551, a first feature extraction module 2552, a first alignment module 2553, a first generation module 2554, and a first prediction module 2555. These modules are logical and can be arbitrarily combined or further separated according to the functions they implement. The functions of each module will be described below.

[0100] The artificial intelligence-based label recognition model training method provided in the embodiments of the present application will be explained in combination with the exemplary application and implementation of the electronic device provided in the embodiments of the present application.

[0101] See also Figure 3A , Figure 3A This is a flow chart of the artificial intelligence-based label recognition model training method provided in the embodiment of the present application, which will be combined with Figure 3A The steps shown are explained.

[0102] In step 101, sample content and a sample label of the sample content are obtained; wherein the sample label is one of a plurality of candidate labels.

[0103] Here, the sample content and the sample labels obtained by annotating the sample content (such as manual annotation) are obtained. Since the sample labels are obtained through annotation, it can be considered that the sample labels are completely accurate.

[0104] The embodiments of the present application do not limit the modality of the sample content, which can be a single modality, such as text, image, or voice; the sample content can also have multiple modalities at the same time. For example, the sample content can be a video with subtitles or titles, and has the three modalities of text, image, and voice.

[0105] The embodiment of the present application does not limit the number of sample contents. For example, multiple sample contents can be obtained to improve the model training effect of the label recognition model and avoid the label recognition model from falling into overfitting during training. For ease of understanding, the following description uses one sample content as an example.

[0106] It is worth noting that the sample tag is one of multiple pre-set candidate tags, and multiple candidate tags can be included in the tag library for use in annotation. The embodiment of the present application does not limit the number of sample tags for sample content, and can be one or more.

[0107] In step 102, the content feature extraction network in the tag recognition model performs content feature extraction processing on the sample content to obtain content features of the sample content.

[0108] Here, the tag recognition model includes a content feature extraction network (or content feature encoder), which is used to perform content feature extraction processing on the sample content to obtain content features of the sample content, wherein the content features of the sample content include semantic information of the sample content.

[0109] The content feature extraction network corresponds to the modality of the sample content. For example, when the sample content is an image, the content feature extraction network is an image feature extraction network; when the sample content is text, the content feature extraction network is a text feature extraction network; when the sample content is speech, the content feature extraction network is a speech feature extraction network. Of course, when the sample content has multiple modalities, a multimodal feature extraction network is used, that is, it can simultaneously support feature extraction from multiple modalities.

[0110] It is worth noting that the network in the tag recognition model involved in the embodiments of the present application (such as the content feature extraction network) refers to an artificial neural network (ANN), rather than a communication network.

[0111] In some embodiments, the sample content includes content of multiple modalities; the content feature extraction network in the label recognition model includes a sequence encoder; the above-mentioned content feature extraction processing of the sample content by the content feature extraction network in the label recognition model to obtain the content features of the sample content can be achieved in the following manner: the content of each modality in the sample content is converted into multiple content units; all content units of multiple modalities are subjected to sequence encoding processing based on sequence relationships through the sequence encoder to obtain the content features of the sample content.

[0112] When the sample content has multiple modalities, that is, includes content from multiple modalities, the content feature extraction network is a multimodal feature extraction network. The multimodal feature extraction network may include a sequence encoder, which is used to perform sequence encoding processing on sequence data based on sequence relationships to obtain more accurate sequence data. Sequence relationships refer to the relationship between different elements in the sequence data. For example, the sequence encoder may be an encoder in a Transformer model, but this is not limited to this.

[0113] During the content feature extraction process, the content of each modality is first converted (segmented) into multiple meaningful content units (Tokens). The embodiment of the present application does not limit the conversion method, and an artificial neural network may be used or not. For example, for text, the text can be tokenized and vocabulary mapped to obtain multiple text units; for images, the image feature extraction process can be performed on the image through an image feature extraction network (the image feature extraction network can be considered as part of a multimodal feature extraction network). In this way, the obtained image features contain multiple image units. For example, if the dimension of the image feature is n*768, it can be considered as n 1*768 image units; for speech, the speech feature extraction process can be performed on the speech through a speech feature extraction network (the speech feature extraction network can be considered as part of a multimodal feature extraction network). In this way, the obtained speech features contain multiple speech units. Then, all content units of the multiple modalities are input into a sequence encoder in the form of sequence data, so that the sequence encoder performs sequence encoding based on sequence relationships to obtain content features of the sample content, wherein the content features of the sample content are essentially also in the form of sequence data.

[0114] The above method is an early-fusion method, which has at least the following technical effects:

[0115] 1) Simple and direct: Features are directly fused at the input stage of the sequence encoder (here, all content units of multiple modalities are input in the form of sequence data), which simplifies the training and inference process.

[0116] 2) High information utilization: Since all content units of multiple modalities are combined at the input stage of the sequence encoder, the sequence encoder can make full use of the complementary information between different modalities.

[0117] 3) High training efficiency: For content feature extraction, only one model (the sequence encoder) needs to be trained, reducing computing resources and time consumption. Furthermore, training can be performed based on the pre-trained sequence encoder, further improving training efficiency.

[0118] In some embodiments, the sample content includes content of multiple modalities; the content feature extraction network in the label recognition model includes modal feature extraction networks corresponding to the multiple modalities; the above-mentioned content feature extraction processing of the sample content by the content feature extraction network in the label recognition model to obtain the content features of the sample content can be achieved in the following way: the modal feature extraction processing of the content of each modality in the sample content is performed by the modal feature extraction network corresponding to each modality to obtain the modal features of each modality; the modal feature fusion processing is performed on the modal features corresponding to the multiple modalities to obtain the content features of the sample content.

[0119] When the sample content has multiple modalities, that is, includes content of multiple modalities, the content feature extraction network is a multimodal feature extraction network, and the multimodal feature extraction network may include modal feature extraction networks corresponding to the multiple modalities.

[0120] During the content feature extraction process, the modal feature extraction network corresponding to each modality is first used to extract the modal features of each modality in the sample content to obtain the modal features of each modality. For example, if the sample content has two modalities, text and image, the text in the sample content is extracted using the text feature extraction network to obtain text features; and the image in the sample content is extracted using the image feature extraction network to obtain image features. Then, the modal features corresponding to the multiple modalities are subjected to modal feature fusion processing to obtain the content features of the sample content. The method of modal feature fusion processing is not limited, and can be, for example, splicing processing or splicing processing followed by full connection processing.

[0121] The above method is a late-fusion method, which has at least the following technical effects:

[0122] 1) Highly flexible and modular: The modal feature extraction network corresponding to each modality can be trained and inferred independently, which makes the label recognition model more flexible and allows for easy addition or deletion of a modality without retraining the entire label recognition model.

[0123] 2) Introducing a new modality does not affect the original model: When introducing a new modality, it is only necessary to add the modal feature extraction network corresponding to the new modality to the label recognition model and train it, without causing a major impact on the overall network architecture of the label recognition model. This helps to maintain the stability and consistency of the label recognition model.

[0124] 3) Rich semantic information: The content features of the sample content obtained by modal feature fusion processing usually contain rich semantic information, which helps to improve the performance of the model.

[0125] In step 103, a label feature extraction process is performed on each candidate label through the label feature extraction network in the label recognition model to obtain the label feature of each candidate label.

[0126] For each pre-set candidate tag, the tag feature extraction network in the tag recognition model performs tag feature extraction on each candidate tag to obtain the tag feature of each candidate tag. In this way, the tag feature contains the semantic information of the corresponding candidate tag. Considering that the candidate tags are usually described in text form, the tag feature extraction network can be a text feature extraction network, that is, the text feature extraction network performs text feature extraction on the description text of each candidate tag to obtain the tag feature of each candidate tag (essentially text feature). It is worth noting that the text feature extraction network here is different from the text feature extraction network in the content feature extraction network (if any). Prior to this, the description text of each candidate tag can be constructed according to a preset text template. The text template is such as: [Chinese name of the candidate tag] means [description information of the candidate tag].

[0127] In step 104, the feature interaction network in the tag recognition model performs feature interaction processing on the content features of the sample content and the tag features of each candidate tag to obtain the interaction features of each candidate tag.

[0128] The content features of the sample content obtained in step 102 include the semantic information of the sample content, and the label features of the candidate labels obtained in step 103 include the semantic information of the candidate labels. Therefore, in this step, the content features of the sample content and the label features of each candidate label are subjected to feature interaction processing through the feature interaction network in the label recognition model to obtain the interaction features of each candidate label. In this way, the overall semantic information of the sample content can be decoupled to each candidate label, so that the interaction features combine the semantic information of the sample content and the semantic information of the candidate label.

[0129] In some embodiments, the feature interaction network can be an attention network. During the feature interaction processing implemented by the attention network, the association between the content features of the sample content and the tag features of each candidate tag is adaptively captured based on the attention mechanism, thereby improving the feature interaction effect.

[0130] In step 105, the matching probability prediction network in the tag recognition model performs matching probability prediction processing based on the interaction features of each candidate tag to obtain the matching probability between each candidate tag and the sample content.

[0131] Here, the matching probability prediction network in the tag recognition model performs matching probability prediction based on the interaction features of each candidate tag. Specifically, the interaction features of each candidate tag are used as input to the matching probability prediction network, and the matching probability prediction network outputs the matching probability between each candidate tag and the sample content. The greater the matching probability between a candidate tag and the sample content, the more likely the candidate tag is to match the sample content.

[0132] It is worth noting that for the interaction features of each candidate tag, the matching probability prediction process is performed independently. For example, the matching probability prediction process is performed based on the interaction features of candidate tag 1 to obtain the matching probability between candidate tag 1 and the sample content; the matching probability prediction process is performed based on the interaction features of candidate tag 2 to obtain the matching probability between candidate tag 2 and the sample content.

[0133] In some embodiments, the matching probability prediction network may be a multi-layer perceptron (MLP) network with a sigmoid activation function connected to the last layer. Of course, the network structure is not limited thereto.

[0134] In step 106 , the co-occurrence probability of different candidate tags is determined according to the interaction feature of each candidate tag, and the pseudo label of the sample content is determined according to the co-occurrence probability of different candidate tags and the sample label of the sample content.

[0135] In actual application scenarios, due to the diversity of the content itself and the usually large number of candidate labels, it is difficult to completely mark all sample labels of the sample content. The sample labels obtained in step 101 present a high accuracy and low recall state, that is, the sample labels obtained in step 101 are usually accurate, but there may be omissions.

[0136] Since the interaction features of the candidate tags combine the semantic information of the sample content and the semantic information of the candidate tags, in an embodiment of the present application, the co-occurrence probability of any two candidate tags is determined based on the interaction features corresponding to the two candidate tags. For example, the interaction features corresponding to the two candidate tags can be subjected to interaction feature fusion processing (such as splicing processing) to obtain fused interaction features, and then the co-occurrence probability of the two candidate tags is predicted based on the fused interaction features, wherein the co-occurrence probability of the two candidate tags refers to the probability of the two candidate tags appearing together, that is, the probability that the two candidate tags both match the sample content.

[0137] It is understandable that the greater the co-occurrence probability of a candidate label (here, a candidate label other than the sample label) and the sample label, the more likely the candidate label is a sample label that has been omitted. Therefore, the sample label obtained in step 101 and the co-occurrence probability of different candidate labels can be used to infer the sample label that has been omitted. For ease of distinction, the inferred sample label that has been omitted is named a pseudo-label, where the pseudo-label is different from the sample label obtained in step 101. The embodiment of the present application does not limit the method of inferring pseudo-labels. For example, when the co-occurrence probability of a candidate label and a sample label is greater than a co-occurrence probability threshold, the candidate label is determined to be a pseudo-label; alternatively, the co-occurrence probability of each candidate label and the sample label can be calculated, and the candidate labels corresponding to the largest n co-occurrence probabilities can be determined as pseudo-labels, where n is an integer greater than 0.

[0138] In some embodiments, step 106 can be implemented by a network layer in the tag recognition model, for example, by determining the co-occurrence probability of different candidate tags based on the interaction characteristics of each candidate tag through a co-occurrence probability prediction network in the tag recognition model. In the model training phase of the tag recognition model, the co-occurrence probability prediction network can be trained or not (i.e., in a frozen state). After the tag recognition model is trained, the co-occurrence probability prediction network can be deleted, i.e., the co-occurrence probability prediction network does not participate in the model reasoning phase based on the trained tag recognition model.

[0139] In step 107 , a tag recognition model is trained based on the matching probability of each candidate tag, the sample tag of the sample content, the pseudo-tag of the sample content, and the co-occurrence probability of different candidate tags.

[0140] Here, the label recognition ability (or matching probability prediction ability) of the label recognition model can be reflected by the matching probability of each candidate label, the sample label of the sample content, and the pseudo-label of the sample content. The co-occurrence probability of different candidate labels and the sample label of the sample content can reflect the co-occurrence relationship prediction ability of the label recognition model. Therefore, the label recognition model is trained according to the matching probability of each candidate label, the sample label of the sample content, the pseudo-label of the sample content, and the co-occurrence probability of different candidate labels, so as to seek a balance between label recognition ability and co-occurrence relationship prediction ability and improve the model training effect.

[0141] It is worth noting that when training the label recognition model, all network layers or specific part of the network layers in the label recognition model can be trained without limitation.

[0142] It is worth noting that the embodiment of the present application does not limit the stopping conditions for training the label recognition model, such as the number of training times reaching the number threshold, or the model performance reaching the performance threshold, etc.

[0143] In some embodiments, a loss value for reflecting the label recognition ability can be determined based on the matching probability of each candidate label, the sample label of the sample content, and the pseudo label of the sample content. A loss value for reflecting the co-occurrence relationship prediction ability can be determined based on the co-occurrence probability of different candidate labels and the sample label of the sample content. The two loss values ​​are fused to obtain a fused loss value, and the label recognition model is trained based on the fused loss value.

[0144] In the process of determining the loss value used to reflect the label recognition ability and the loss value used to reflect the co-occurrence relationship prediction ability, the loss function used is a classification loss function, such as the cross-entropy loss function. It is worth noting that the loss function is used to calculate the difference between the predicted result and the expected result as the loss value. In the process of determining the loss value used to reflect the label recognition ability, the matching probability of each candidate label is used as the prediction result, and the sample label of the sample content and the pseudo-label of the sample content are both determined as the expected result; in the process of determining the loss value used to reflect the co-occurrence relationship prediction ability, the co-occurrence probability of different candidate labels is used as the prediction result, and the co-occurrence result of different candidate labels (i.e., whether they actually co-occur) is used as the expected result, wherein the co-occurrence result of different candidate labels can be determined based on the sample label of the sample content.

[0145] Then, a loss value fusion process is performed on the loss value used to reflect the label recognition ability and the loss value used to reflect the co-occurrence relationship prediction ability to obtain a fused loss value, and the label recognition model is trained based on the fused loss value. The loss value fusion process may be a direct summation process or a weighted summation process, which is not limited to this. Training the label recognition model based on the fused loss value may mean backpropagating the fused loss value through the label recognition model and updating the model parameters of the label recognition model along the gradient descent direction during the backpropagation process.

[0146] The above method reflects the ability of the label recognition model by calculating the loss value. In this way, when training the label recognition model based on the fusion loss value, it is possible to seek a balance between the label recognition ability and the co-occurrence relationship prediction ability, thereby improving the model training effect.

[0147] like Figure 3AAs shown, in the model training phase of the label recognition model provided in the embodiment of the present application, the sample content and the sample label of the sample content are first obtained, and the sample content is processed by the content feature extraction network in the label recognition model to extract the content features that can represent the overall semantic information of the sample content. At the same time, the label feature extraction network in the label recognition model performs label feature extraction on each candidate label to obtain the label features of each candidate label. Then, the content features of the sample content and the label features of each candidate label are processed for feature interaction through the feature interaction network in the label recognition model to obtain the interaction features of each candidate label. In this way, the overall semantic information of the sample content can be decoupled from each candidate label, so that the interaction features combine the semantic information of the sample content and the semantic information of the candidate label, so that the matching probability between each candidate label and the sample content can be predicted separately according to the interaction features of each candidate label, thereby improving the accuracy of the prediction. In addition, the co-occurrence probability of different candidate labels is determined based on the interactive features of each candidate label, and the pseudo-label of the sample content is determined based on the co-occurrence probability of different candidate labels and the sample label of the sample content. In this way, the sample label (i.e., pseudo-label) of the omitted label is predicted by the co-occurrence probability to achieve the supplementation and correction of the sample label. Finally, the label recognition model is trained based on the matching probability of each candidate label, the sample label of the sample content, the pseudo-label of the sample content, and the co-occurrence probability of different candidate labels. Since the label recognition ability of the label recognition model can be reflected by the matching probability of each candidate label, the sample label of the sample content, and the pseudo-label of the sample content, and the co-occurrence probability of different candidate labels and the sample label of the sample content can reflect the co-occurrence relationship prediction ability of the label recognition model, the above training method can seek a balance between the label recognition ability and the co-occurrence relationship prediction ability, thereby improving the model training effect. At the same time, the embodiment of the present application does not have high requirements for sample labels (i.e., complete labeling is not required), and there is no need for multiple rounds of manual iterative labeling, saving time and labor costs.

[0148] In some embodiments, see Figure 3B , Figure 3B This is a flow chart of a method for training a label recognition model based on artificial intelligence provided by an embodiment of the present application. Figure 3A Step 107 shown can be implemented through steps 201 to 205 , which will be described in conjunction with each step.

[0149] In step 201, a first loss value is determined according to the matching probability of each candidate label and the sample label of the sample content.

[0150] For example, the matching probability of each candidate label is used as the prediction result, the sample label of the sample content is used as the expected result, and the difference between the predicted result and the expected result is calculated as the first loss value.

[0151] In step 202, a second loss value is determined based on the matching probability of each candidate label and the pseudo label of the sample content.

[0152] For example, the matching probability of each candidate label is used as the prediction result, the pseudo label of the sample content is used as the expected result, and the difference between the predicted result and the expected result is calculated as the second loss value.

[0153] It is worth noting that the loss function used in step 201 is the same as the loss function used in step 202, that is, step 202 essentially regards the pseudo label as the sample label and then calculates the loss value.

[0154] In step 203, a third loss value is determined based on the sample label of the sample content and the co-occurrence probability of different candidate labels.

[0155] Here, the co-occurrence results of different candidate tags (i.e., whether they actually co-occur) can be determined based on the sample tags of the sample content. For example, the tag library includes candidate tags 1, candidate tags 2, candidate tags 3, and candidate tags 4. The sample content is annotated with two sample tags, candidate tags 1 and candidate tags 2. Then, among the tag pairs formed by different candidate tags, only the tag pair (candidate tag 1, candidate tag 2) has a co-occurrence result, and the co-occurrence results of the other tag pairs are all non-co-occurrence.

[0156] Then, the co-occurrence probability of different candidate tags is used as the prediction result, the co-occurrence result of different candidate tags is used as the expected result, and the difference between the predicted result and the expected result is calculated as the third loss value.

[0157] In some embodiments, the loss functions used in step 201, step 202, and step 203 are all classification loss functions, such as cross entropy loss function, negative log-likelihood loss function, exponential loss function, square loss function, etc.

[0158] In some embodiments, the above-mentioned determination of the co-occurrence probability of different candidate tags based on the interaction features of each candidate tag can be achieved in the following manner: multiple candidate tags are combined to obtain multiple tag pairs; wherein, each tag pair includes two candidate tags; based on the interaction features corresponding to the two candidate tags in each tag pair, the co-occurrence probability of the two candidate tags in each tag pair is determined; the above-mentioned determination of the third loss value based on the sample label of the sample content and the co-occurrence probability of different candidate tags can be achieved in the following manner: based on the result of whether each tag pair includes the sample label, the co-occurrence result of the two candidate tags in each tag pair is determined; based on the co-occurrence result of the two candidate tags in each tag pair and the co-occurrence probability of the two candidate tags in each tag pair, the loss value of each tag pair is determined; loss value fusion processing is performed on the loss values ​​corresponding to the multiple tag pairs to obtain a third loss value.

[0159] After obtaining the interaction features for each candidate tag, to determine the co-occurrence probabilities of all possible different candidate tags, multiple candidate tags can be combined in pairs to obtain multiple tag pairs, where each tag pair includes two candidate tags. This combination process can be exhaustive. For each obtained tag pair, the co-occurrence probability of the two candidate tags in the tag pair is determined based on the interaction features corresponding to the two candidate tags in the tag pair.

[0160] Then, based on whether the label pair includes the sample label, the co-occurrence result of the two candidate labels in the label pair is determined. The co-occurrence result of the two candidate labels in the label pair is used as the expected result, the co-occurrence probability of the two candidate labels in the label pair is used as the predicted result, and the difference between the predicted result and the expected result is calculated as the loss value of the label pair.

[0161] Finally, the loss values ​​corresponding to multiple label pairs are fused to obtain a third loss value. This allows all label pairs to be comprehensively considered, making the calculated third loss value more accurately reflect the co-occurrence relationship prediction ability of the label recognition model.

[0162] In some embodiments, the above-mentioned determination of the loss value of each label pair based on the co-occurrence results of the two candidate labels in each label pair and the co-occurrence probability of the two candidate labels in each label pair can be achieved in the following manner: for any label pair, the following processing is performed: the co-occurrence results of the two candidate labels in any label pair and the co-occurrence probability of the two candidate labels in any label pair are calculated and processed; the calculated difference is weighted according to at least one of the balance weight and the adjustment weight to obtain the loss value of any label pair; wherein the balance weight is negatively correlated with the label pair ratio, the label pair ratio represents the quantitative ratio between the first label pair and the second label pair, the first label pair and any label pair correspond to the same co-occurrence result, and the second label pair and any label pair correspond to different co-occurrence results; wherein the adjustment weight is positively correlated with the difference.

[0163] Since all the sample labels annotated for the sample content usually only account for a small part of the label library, for the multiple label pairs generated by the combined processing, the co-occurrence results of most of the label pairs (referring to the co-occurrence results of the two candidate labels in the label pair, the same applies below) are non-co-occurrence, and there is a problem of imbalance between positive and negative samples. During the model training stage, it may be biased towards these label pairs whose co-occurrence results are non-co-occurrence, resulting in overfitting. In the face of the problem of imbalance between positive and negative samples, in an embodiment of the present application, for each label pair such as label pair 1, the difference between the co-occurrence result and the co-occurrence probability of label pair 1 is first calculated (for example, calculated by a cross entropy loss function or other classification loss function), and then the calculated difference is weighted according to the balance weight to obtain the loss value of label pair 1. Among them, the balance weight is negatively correlated with the label pair ratio, and the label pair ratio represents the quantitative ratio between the first label pair and the second label pair. The first label pair and label pair 1 correspond to the same co-occurrence result, and the second label pair and label pair 1 correspond to different co-occurrence results. In this way, the ratio of positive and negative samples is balanced by balancing the weights, so that more attention is paid to samples with a smaller proportion (i.e., label pairs with co-occurrence results) during the model training stage, thereby improving the model training effect.

[0164] At the same time, the difference between the co-occurrence probability of some label pairs (referring to the co-occurrence probability of the two candidate labels in the label pair, and the same applies below) and the co-occurrence result is small, that is, the samples are easy to classify, and they may be biased towards samples that are easy to classify during the model training stage, resulting in poor model training effect. In response to this problem, in an embodiment of the present application, for each label pair such as label pair 1, the difference between the co-occurrence result and the co-occurrence probability of label pair 1 is first calculated (for example, by using a cross entropy loss function or other classification loss function), and then the calculated difference is weighted according to the adjustment weight to obtain the loss value of label pair 1. Among them, the adjustment weight is positively correlated with the difference. In this way, the loss value of samples that are easier to classify is reduced, and the loss value of samples that are more difficult to classify is increased, which can make the more difficult to classify samples more concerned during the model training stage and improve the model training effect.

[0165] The above-mentioned balancing weights and adjustment weights can also be used in combination. For example, after calculating the difference between the co-occurrence result and the co-occurrence probability of label pair 1, the calculated difference is weighted according to the balancing weight and the adjustment weight to obtain the loss value of label pair 1.

[0166] In some embodiments, the above-mentioned determination of the co-occurrence result of the two candidate tags in each tag pair based on whether each tag pair includes a sample tag can be achieved in the following way: for any tag pair, the following processing is performed: when the two candidate tags in any tag pair are both sample tags, the co-occurrence result of the two candidate tags in any tag pair is determined to be co-occurrence; when at least one candidate tag in any tag pair is not a sample tag, the co-occurrence result of the two candidate tags in any tag pair is determined to be non-co-occurrence.

[0167] Here, for each label pair, when both candidate labels in the label pair are sample labels (provided that the number of sample labels is multiple), the co-occurrence result of the two candidate labels in the label pair is determined to be co-occurrence; when any candidate label in the label pair is not a sample label, or both candidate labels in the label pair are not sample labels, the co-occurrence result of the two candidate labels in the label pair is determined to be non-co-occurrence.

[0168] For example, the label library contains candidate label 1, candidate label 2, candidate label 3, and candidate label 4. The sample labels annotated for the sample content include two, namely candidate label 1 and candidate label 2. The following example can be given:

[0169] 1) A label pair is (candidate label 1, candidate label 2). Since candidate label 1 and candidate label 2 are both sample labels, the co-occurrence result of the two candidate labels in the label pair is determined to be co-occurrence.

[0170] 2) A label pair is (candidate label 1, candidate label 3). Since candidate label 3 is not a sample label, the co-occurrence result of the two candidate labels in the label pair is determined to be non-co-occurrence.

[0171] 3) A label pair is (candidate label 3, candidate label 4). Since candidate label 3 and candidate label 4 are not sample labels, the co-occurrence result of the two candidate labels in the label pair is determined to be non-co-occurrence.

[0172] By combining the sample labels obtained in step 101 to determine the co-occurrence results of the label pairs, the accuracy of the co-occurrence results of the determined label pairs can be guaranteed, thereby ensuring the accuracy of the loss values ​​of the determined label pairs.

[0173] In some embodiments, the above-mentioned determination of the co-occurrence probability of the two candidate tags in each tag pair based on the interaction features corresponding to the two candidate tags in each tag pair can be achieved in the following way: for any tag pair, the following processing is performed: the interaction features corresponding to the two candidate tags in any tag pair are fused to obtain fused interaction features; the co-occurrence probability prediction processing is performed based on the fused interaction features to obtain the co-occurrence probability of the two candidate tags in any tag pair.

[0174] Here, for each label pair, the interaction features corresponding to the two candidate labels in the label pair can be subjected to interaction feature fusion processing to obtain fused interaction features, wherein the interaction feature fusion processing is such as splicing processing. Then, co-occurrence probability prediction processing is performed based on the fused interaction features to obtain the co-occurrence probability of the two candidate labels in the label pair, wherein the co-occurrence probability prediction processing can be achieved through a co-occurrence probability prediction network, and the co-occurrence probability prediction network can be a multi-layer perceptron (MLP) network with a sigmoid activation function connected to the last layer. Of course, the network structure is not limited to this. In the above method, since the fused interaction features combine the semantic information corresponding to the two candidate labels, the co-occurrence probability prediction processing based on the fused interaction features can accurately predict the co-occurrence relationship of the two candidate labels, that is, it can improve the accuracy of the obtained co-occurrence probability.

[0175] In step 204, loss value fusion processing is performed on the first loss value, the second loss value, and the third loss value to obtain a fused loss value.

[0176] Here, the first loss value and the second loss value are used to reflect the label recognition ability of the label recognition model, and the second loss value is completely dependent on the pseudo-label determined by the label recognition model. At the same time, the third loss value is used to reflect the co-occurrence relationship prediction ability of the label recognition model, and the third loss value is completely dependent on the marked sample label. Therefore, the first loss value, the second loss value and the third loss value can be subjected to loss value fusion processing to obtain a fused loss value.

[0177] The embodiments of the present application do not limit the method for fusion processing of loss values, and can be, for example, direct summation or weighted summation. When the method for fusion processing of loss values ​​is weighted summation, considering that sample labels are more accurate than pseudo labels, the weight of the first loss value can be set to be greater than the weight of the second loss value, so that the first loss value has a larger proportion when calculating the fusion loss value.

[0178] In step 205, the label recognition model is trained according to the fusion loss value.

[0179] Here, the label recognition model is trained based on the fusion loss value to seek a balance between label recognition ability and co-occurrence relationship prediction ability during the model training process, that is, neither the labeled sample labels nor the determined pseudo labels are fully trusted.

[0180] like Figure 3B As shown, the embodiment of the present application determines a first loss value based on the matching probability of each candidate label and the sample label of the sample content, and determines a second loss value based on the matching probability of each candidate label and the pseudo label of the sample content, wherein the first loss value and the second loss value are used to reflect the label recognition ability of the label recognition model, and the second loss value is completely dependent on the pseudo label determined by the label recognition model; a third loss value is determined based on the sample label of the sample content and the co-occurrence probability of different candidate labels, and the third loss value is used to reflect the co-occurrence relationship prediction ability of the label recognition model, and the third loss value is completely dependent on the labeled sample label. Then, the label recognition model is trained by combining the first loss value, the second loss value, and the third loss value, so as to seek a balance between the label recognition ability and the co-occurrence relationship prediction ability, and effectively improve the model training effect.

[0181] In some embodiments, see Figure 3C , Figure 3C This is a flow chart of a method for training a label recognition model based on artificial intelligence provided by an embodiment of the present application. Figure 3A Step 106 shown can be implemented through steps 301 to 303 , which will be described in conjunction with each step.

[0182] In step 301 , the co-occurrence probability of different candidate tags is determined based on the interaction feature of each candidate tag.

[0183] In step 302, a co-occurrence probability fusion process is performed on the co-occurrence probability of any candidate tag and all sample tags of the sample content to obtain a fused co-occurrence probability.

[0184] Here, based on the co-occurrence probabilities of different candidate tags, for any candidate tag that is different from the sample tag, the co-occurrence probabilities of the candidate tag and all sample tags of the sample content can be fused to obtain a fused co-occurrence probability.

[0185] For example, the label library contains candidate label 1, candidate label 2, candidate label 3 and candidate label 4. The sample labels annotated for the sample content include two, namely candidate label 1 and candidate label 2. For candidate label 3, the co-occurrence probability of candidate label 3 and candidate label 1, and the co-occurrence probability of candidate label 3 and candidate label 2 can be fused to obtain the fused co-occurrence probability of candidate label 3.

[0186] It is worth noting that when the sample content has only one sample label, the co-occurrence probability of any candidate label and the sample label can be directly determined as the fused co-occurrence probability of the any candidate label.

[0187] In step 303, when the fused co-occurrence probability is greater than the co-occurrence probability threshold, any candidate label is determined as a pseudo label.

[0188] Taking the above example again, when the fused co-occurrence probability of candidate tag 3 is greater than the co-occurrence probability threshold, candidate tag 3 is determined to be a pseudo-tag. The co-occurrence probability threshold can be set according to the actual application scenario. For different candidate tags, the same co-occurrence probability threshold is used in step 303.

[0189] In some embodiments, the above-mentioned determination of any candidate label as a pseudo label when the fused co-occurrence probability is greater than the co-occurrence probability threshold can be achieved in the following way: when the fused co-occurrence probability is greater than the co-occurrence probability threshold corresponding to the number of times the label recognition model has been trained, any candidate label is determined as a pseudo label; wherein the co-occurrence probability threshold is negatively correlated with the number of times the label recognition model has been trained.

[0190] Here, the co-occurrence probability threshold is negatively correlated with the number of times the label recognition model has been trained. For example, when the label recognition model has been trained a small number of times (i.e., in the early stages of the model training phase), the label recognition ability of the label recognition model is relatively weak. At this time, a larger co-occurrence probability threshold can be set to introduce fewer or even no pseudo-labels, and to train the label recognition ability of the label recognition model more. When the label recognition model has been trained a large number of times (i.e., in the middle and late stages of the model training phase), the label recognition model already has a certain label recognition ability. At this time, a smaller co-occurrence probability threshold can be set to introduce more pseudo-labels, thereby seeking a balance between label recognition ability and co-occurrence relationship prediction ability. The above method realizes the adaptive adjustment of the co-occurrence probability threshold, so that the co-occurrence probability threshold adapts to the current training focus of the label recognition model, and can achieve good model training results at different stages of the model training phase.

[0191] like Figure 3C As shown, the embodiment of the present application performs co-occurrence probability fusion processing on the co-occurrence probability of any candidate label and all sample labels of the sample content to obtain a fused co-occurrence probability. When the fused co-occurrence probability is greater than the co-occurrence probability threshold, the candidate label is determined as a pseudo-label. In this way, all sample labels of the sample content are referenced rather than individual sample labels during the pseudo-label determination process, making the process more comprehensive and interpretable, and improving the accuracy of the determined pseudo-labels.

[0192] In some embodiments, see Figure 3D , Figure 3D This is a flow chart of a method for training a label recognition model based on artificial intelligence provided by an embodiment of the present application. Figure 3A Step 104 shown can be implemented through steps 401 to 404 , which will be described in conjunction with each step.

[0193] In step 401, feature transformation processing is performed on the label features of any candidate label using a feature transformation matrix.

[0194] For ease of understanding, we take any candidate label, such as candidate label 1, as an example to illustrate the process of feature interaction processing based on the attention mechanism.

[0195] First, the label features of candidate label 1 are transformed by the feature transformation matrix in the feature interaction network, wherein the purpose of the feature transformation is to transform the dimension of the label features of candidate label 1 so that it can be operated with the content features of the sample content.

[0196] In step 402, correlation calculation is performed between the tag features after feature transformation and the content features of the sample content to obtain correlation features.

[0197] Here, correlation calculation is performed between the label features after feature transformation and the content features of the sample content to obtain correlation features, wherein the correlation features are used to reflect the correlation between the candidate label 1 and the sample content.

[0198] It is worth noting that the embodiment of the present application does not limit the method of calculating the correlation, which can be a product, a cosine similarity, or an MLP operation.

[0199] In step 403, the correlation features are normalized to obtain correlation weight features.

[0200] Here, the correlation feature is normalized to obtain a correlation weight feature, wherein the correlation weight feature includes multiple values, each value is in the interval of (0, 1), and the sum of all values ​​is 1.

[0201] The embodiment of the present application does not limit the method of normalization processing, for example, it can be implemented through a softmax activation function.

[0202] In step 404, the content features of the sample content are weighted according to the correlation weight feature to obtain the interaction feature of any candidate tag.

[0203] Here, the content features of the sample content are weighted (multiplied) according to the correlation weight features, so as to focus on the part of the content features of the sample content that is related to candidate label 1, and eliminate the part of the content features of the sample content that is not related to candidate label 1, and obtain the interactive features of candidate label 1.

[0204] like Figure 3D As shown, the embodiment of the present application calculates the correlation weight feature and weights the content features of the sample content according to the correlation weight feature, thereby focusing attention on the part of the content features of the sample content that is related to a specific candidate tag, so that the final interactive feature effectively combines the semantic information of the sample content and the semantic information of the candidate tag.

[0205] The artificial intelligence-based tag recognition method provided in the embodiments of the present application will be explained in combination with the exemplary application and implementation of the electronic device provided in the embodiments of the present application.

[0206] See also Figure 4 , Figure 4 This is a flow chart of the tag recognition method based on artificial intelligence provided by the embodiment of the present application, which will be combined with Figure 4 The steps shown are explained.

[0207] In step 501, content to be identified and a plurality of candidate tags are obtained.

[0208] After the label recognition model is trained, the model inference phase can be entered based on the trained label recognition model. The following example illustrates the model inference phase.

[0209] First, the content to be identified and a plurality of candidate tags in a tag library are obtained, wherein the content to be identified and the sample content have the same modality.

[0210] In step 502, a content feature extraction network in the tag recognition model is used to extract content features from the content to be recognized, thereby obtaining content features of the content to be recognized.

[0211] Here, the content feature extraction network in the trained tag recognition model performs content feature extraction processing on the content to be identified to obtain content features of the content to be identified. In this way, the content features of the content to be identified contain the overall semantic information of the content to be identified.

[0212] In step 503, a label feature extraction process is performed on each candidate label through the label feature extraction network in the label recognition model to obtain the label feature of each candidate label.

[0213] Here, the label feature extraction network in the trained label recognition model performs label feature extraction processing on each candidate label to obtain the label feature of each candidate label. In this way, the label feature of each candidate label contains the semantic information of the candidate label.

[0214] In step 504, the feature interaction network in the tag recognition model performs feature interaction processing on the content features of the content to be recognized and the tag features of each candidate tag to obtain the interaction features of each candidate tag.

[0215] Here, the feature interaction network in the trained label recognition model is used to perform feature interaction processing on the content features of the content to be identified and the label features of each candidate label to obtain the interaction features of each candidate label. In this way, the overall semantic information of the content to be identified is decoupled to each candidate label, so that the interaction features of each candidate label contain both the semantic information of the content to be identified and the semantic information of the candidate label.

[0216] In step 505, a matching probability prediction network in the tag recognition model performs matching probability prediction processing based on the interaction features of each candidate tag to obtain the matching probability between each candidate tag and the content to be recognized.

[0217] Here, the matching probability prediction network in the trained tag recognition model performs matching probability prediction based on the interaction characteristics of each candidate tag, and obtains the matching probability between each candidate tag and the content to be recognized. The greater the matching probability between a candidate tag and the content to be recognized, the more likely the candidate tag is to match the content to be recognized.

[0218] In step 506 , the candidate tags with matching probabilities greater than the matching probability threshold are determined as target tags that match the content to be identified.

[0219] Here, the matching probability threshold can be set based on the actual application scenario. Candidate tags with matching probabilities greater than the matching probability threshold are determined as target tags that match the content to be identified, completing the entire model inference phase. It is worth noting that during the model inference phase, there is no need to determine pseudo-labels for the content to be identified.

[0220] The embodiments of the present application do not limit the use of target tags. For example, content can be distributed based on target tags, that is, content can be recommended to users who are interested in the target tags of the content; or, multiple contents can be sorted according to the target tags of each content in the multiple contents, for example, multiple contents matching the same target tag can be divided into the same directory or channel, so that users can directly obtain the content of interest.

[0221] like Figure 4 As shown, the embodiment of the present application can implement model reasoning based on the trained label recognition model. Since the trained label recognition model has a strong label recognition capability, it can improve the accuracy of the determined target label and facilitate further application based on the target label, such as content distribution, content organization, etc.

[0222] The following describes an exemplary application of the embodiment of the present application in a practical application scenario. For ease of understanding, the example is given using a video as the content.

[0223] In the solutions provided by related technologies, a label recognition model is usually trained based on a certain scale of labeled data sets (including several sample videos and sample labels manually labeled for each sample video), and machine labeling is achieved through the trained label recognition model. However, in actual application scenarios, manual reviewers can generally label sample videos with accurate sample labels after being trained on the detailed implementation rules of the labeling rules. However, due to the diversity of the sample videos themselves and the huge number of candidate labels, it is difficult for manual reviewers to completely label all sample labels that match the sample videos. Therefore, the sample labels in the labeled data set show a high accuracy and low recall state. Simply put: the sample labels marked on the sample videos are usually accurate, but there may be omissions. If the model training directly relies on manually labeled sample labels, the manually labeled sample labels will be regarded as positive samples, and the missed sample labels will be mistakenly suppressed as negative samples, which is equivalent to introducing data noise and is not conducive to model training. At the same time, it is very difficult to completely label all sample labels, and it is difficult to improve the quality of the labeled data set from a manual perspective.

[0224] The embodiments of the present application take into account that there is generally a strong correlation between different candidate labels, and this correlation includes statistical co-occurrence relationships and mutually exclusive relationships. For example, for a certain picture, sample labels such as "Christmas market", "pedestrians", and "lights" are typed. Then, even without looking at the picture, based on the sample labels that have been typed, it can be inferred that the picture is likely to match candidate labels such as "Christmas tree" and "shop", and does not match candidate labels such as "grassland". Based on this, the embodiments of the present application target the situation where labels are missing in the labeled data set by mining and migrating the co-occurrence relationship between candidate labels. On the premise of only giving some sample labels, a training target for co-occurrence relationship prediction is added, and the sample labels are corrected and supplemented by the determined pseudo-labels, so as to better adapt to and solve the label recognition tasks in actual business scenarios.

[0225] The embodiments of this application can be applied to various business scenarios, such as online video platforms, news platforms, self-media platforms, short video platforms, etc. Figure 5 As shown, the embodiments of the present application can be applied to the content review link in the chain of content production-content review-content distribution. Machine labeling through the trained label recognition model can improve the accuracy of machine labeling, thereby reducing or even omitting the work of manual labeling, and improving the efficiency and accuracy of the content review link and the content distribution link.

[0226] The present application provides an embodiment of Figure 6 The schematic diagram of the model training phase of the label recognition model shown in the figure will be combined with Figure 6 Provide explanation.

[0227] 1) Multimodal feature extraction and processing of sample videos.

[0228] Videos naturally have multiple modalities such as images, text, and voice. Among them, the content of image modality is such as video frames, the content of text modality is such as titles, text obtained by optical character recognition (OCR) of video frames, and text obtained by automatic speech recognition (ASR) of audio in videos, and the content of voice modality is such as audio in videos. Figure 7 As shown, a multimodal feature extraction network can be used to extract multimodal features from the sample video to obtain the multimodal features of the sample video (corresponding to the content features mentioned above).

[0229] In the multimodal feature extraction network, a late-fusion form can be adopted. For example, the multimodal feature extraction network includes modal feature extraction networks corresponding to multiple modalities, and the modal feature extraction network corresponding to each modality is used to extract the modal features of each modality in the sample video to obtain the modal features of each modality. Then, the modal features corresponding to the multiple modalities are fused to obtain the multimodal features of the sample video. Alternatively, an early-fusion form can be adopted. For example, the multimodal feature extraction network includes an image feature extraction network corresponding to the image modality, a speech feature extraction network corresponding to the speech modality, and a transformer model (corresponding to the sequence encoder above). First, the image features of the sample video are extracted through the image feature extraction network (equivalent to multiple image tokens), and the speech features of the sample video are extracted through the speech feature extraction network as speech tokens (equivalent to multiple speech tokens). The text in the sample video is segmented (Tokenization) and vocabulary mapped (Vocabulary Mapping) is performed. Mapping) to obtain multiple text tokens, and then, through the transformer model, all image tokens, voice tokens and text tokens are subjected to sequence encoding processing based on sequence relations (equivalent to fusion processing) to obtain the multimodal features of the sample video.

[0230] Figure 6 In the example of early-fusion, for the sake of convenience, the multimodal feature of the sample video is named X, and the dimension of the multimodal feature X is N*d, where N is the sum of the number of tokens of all modalities of the sample video and d is the feature dimension.

[0231] 2) Label feature extraction and processing.

[0232] For each candidate tag in the tag library, the specific information of the candidate tag is usually expressed in plain text, as shown in Table 1.

[0233] Table 1 Label information table

[0234]

[0235] Therefore, the description text of each candidate tag can be constructed through the following text template:

[0236] [Label Chinese Name]: Belongs to the [Label Secondary Category] type under the [Label First Category] category, meaning: [Label Description Information]

[0237] For example, the description text of the candidate tag "Interesting Experiment" is:

[0238] Interesting experiments: belong to the "Highlights - Fun" type under the "Life" category, which means: interesting experiments and challenging content without scientific experimental content, as long as they are interesting (this label does not apply to normal scientific experiments). If the video experiment project is interesting and has a big imagination and a strong curiosity, it can be included in the "Bizarre Experiments".

[0239] For the description text of each candidate tag, after the text is tokenized, it is input into the trained text feature extraction network to obtain the label feature of each candidate tag. The text feature extraction network here is different from the text feature extraction network in the multimodal feature extraction network. Taking the text feature extraction network here as the BERT model as an example, the Embedding corresponding to the CLSToken output by the BERT model can be taken as the label feature. For the sake of convenience, the label feature is named q, and the dimension of the label feature q is 1*d'. If there are C candidate tags, each candidate tag can obtain the corresponding label feature, which is expressed as {q1,q2,q3,…,q C}, where q1 represents the label feature of the first candidate label, and so on.

[0240] 3) Feature interaction processing (semantic decoupling).

[0241] In this step, the label feature q is interacted with the multimodal feature X of the sample video to obtain the interaction feature between the sample video and the candidate label.

[0242] For example, feature interaction processing can be implemented based on the attention mechanism, which can be expressed as:

[0243] f=softmax(q*W*X T )*X

[0244] Where W represents the feature transformation matrix with a dimension of d'*d.

[0245] In the above formula, after scaling the label feature q to d dimensions, the correlation is calculated with the multimodal feature X, and the obtained q*W*X T This is the correlation feature between the label feature q and the multimodal feature X. After performing softmax normalization on the correlation feature, we obtain the correlation weight feature. The multimodal feature X is weighted by the correlation weight feature to obtain the interaction feature between the sample video and the candidate label. The interaction feature can be regarded as the feature of the sample video on the candidate label. In this way, the overall semantic information concentrated in the multimodal feature X is decoupled to different candidate labels through the attention mechanism.

[0246] For each candidate label, the corresponding interaction feature f can be obtained through feature interaction processing. The interaction features of C candidate labels can be recorded as {f1,f2,f3,…,f C}.

[0247] 4) Co-occurrence relationship migration.

[0248] Here, for C candidate labels, we can combine them in pairs to get multiple label pairs, and define a training goal: predict the probability of two candidate labels in a label pair appearing together (i.e., co-occurrence probability). This training goal can be expressed as:

[0249] p i,j =MLP(f i ,f j )

[0250] Among them, the interaction features corresponding to candidate label i (referring to the i-th candidate label, the same applies below) and candidate label j are first spliced ​​together to obtain a 1*2d dimensional fused interaction feature. The fused interaction feature is passed through an MLP (the last layer of the MLP is connected to the sigmoid activation function to make the output value between 0 and 1) to predict the co-occurrence probability of candidate label i and candidate label j, that is, p i,j .

[0251] For any i, j (1≤i, j≤C), the co-occurrence probability can be calculated.

[0252] Since manually annotated sample labels can be obtained for sample videos, other possible sample labels (i.e., pseudo labels) can be inferred using the known sample labels and the above predicted co-occurrence probabilities, as shown below:

[0253]

[0254] Where I(·) represents the sign function; y j =1 indicates that the candidate label j is a manually annotated sample label; θ represents the co-occurrence probability threshold.

[0255] The meaning of this formula is: for an unlabeled candidate tag i, the co-occurrence probability p corresponding to all labeled candidate tags j is i,j Add together to get the probability of candidate label i, which is If the occurrence probability of candidate label i is greater than θ, the candidate label i is determined as a pseudo label.

[0256] 5) Train the label recognition model.

[0257] Here, two tasks are trained together to realize the label recognition model.

[0258] ① Label recognition task.

[0259] For the interaction features {f1,f2,f3,…,f C}, after an MLP (the last layer of MLP is connected to the sigmoid activation function), the matching probability between each candidate label and the sample video is obtained.

[0260] In an embodiment of the present application, the loss value of the label recognition task can be calculated by the binary cross entropy loss function (Binary Cross EntropyLoss, BCE Loss). BCE Loss is calculated based on the matching probability of each candidate label and the true label, wherein when a candidate label is a sample label or a pseudo label, the candidate label is considered to be a true label. If a candidate label is a true label and the matching probability of the candidate label is close to 1, the loss value of the candidate label is close to 0; if a candidate label is a true label and the matching probability of the candidate label is close to 0, the loss value of the candidate label is close to infinity. BCE Loss obtains the loss value of the label recognition task by summing the loss values ​​of all candidate labels.

[0261] For example, for a multi-label classification problem with N sample videos and C candidate labels, the calculation formula of BCE Loss is:

[0262]

[0263] Among them, y n，c Indicates whether the cth candidate label is the true label for the nth sample video. If the cth candidate label is the true label, then y n，c The value of is 1; if the cth candidate label is not the true label, then y n，c The value of p is 0. n,c represents the matching probability between the cth candidate label and the nth sample video.

[0264] For the sake of convenience, BCELoss is abbreviated as L BCE(x, y), where x represents the matching probability and y represents whether it is the true label.

[0265] Considering that there are two versions of the true label, namely the manually annotated sample label and the determined pseudo label, the loss value of the label recognition task can be expressed as:

[0266] L1=L BCE (x,y1)+L BCE (x,y2)

[0267] Among them, L1 represents the loss value of the label recognition task; y1 represents the result of whether it is a sample label, L BCE (x, y1) corresponds to the first loss value above; y2 indicates whether it is a pseudo-label result, which is the above L BCE (x, y2) corresponds to the second loss value above.

[0268] In some embodiments, considering that manually annotated sample labels can be guaranteed to be accurate, while pseudo labels cannot be guaranteed to be accurate, the above formula can be updated as follows:

[0269] L1=W1*L BCE (x,y1)+W2*L BCE (x,y2)

[0270] Where W1 represents L BCE The weight of (x, y1), W2 represents L BCE The weight of (x, y2), and W1>W2, so that L BCE (x, y1) has a larger proportion in L1.

[0271] ② Co-occurrence relationship prediction task.

[0272] The co-occurrence relationship prediction task is a binary classification task. The loss value of each label pair can be calculated based on the co-occurrence result and co-occurrence probability of each label pair, and the loss value of all label pairs is summed to obtain the loss value of the co-occurrence relationship prediction task (corresponding to the third loss value above). Considering that the co-occurrence results of most label pairs are non-co-occurrence, that is, most label pairs are negative samples, the positive and negative samples are seriously unbalanced. At the same time, when calculating the loss value of the co-occurrence relationship prediction task, it will be biased towards samples that are easier to classify. Therefore, the balance weight and adjustment weight can be combined to calculate the loss value of the co-occurrence relationship prediction task, which can be expressed as:

[0273] L2=-α t (1-p t ) γ log(p t )

[0274] Among them, L2 represents the loss value of the co-occurrence relationship prediction task; -log(p t ) represents the cross entropy loss value calculated according to the cross entropy loss function (i.e., the difference between the co-occurrence result and the co-occurrence probability above); α t Represents the balance weight, which is used to balance the ratio of positive and negative samples; (1-pt) γ Represents the adjustment weight, γ represents the adjustment factor, and γ can be set according to the actual application scenario. When γ>0, more attention will be paid to samples that are difficult to classify during training.

[0275] It is worth noting that L2 fully trusts the manually labeled sample labels, L BCE (x, y2) fully trusts the determined pseudo-label, and there is an adversarial relationship between the two, so it is necessary to seek a balance between the two. The solution is to add the loss value of the label recognition task to the loss value of the co-occurrence relationship prediction task to obtain the fusion loss value L, and train the label recognition model based on the fusion loss value L, that is, L = L1 + L2.

[0276] In some embodiments, in the early stage of the model training phase of the label recognition model (such as the first 5 epochs), θ can be set to 1 to avoid introducing any pseudo labels and focus on training the label recognition ability of the label recognition model. At this time, L BCE (x, y2) = 0; in the middle and late stages of the label recognition model training phase, the value of θ is gradually reduced, and more pseudo labels are introduced to guide the label recognition model to be better trained. Due to the introduction of pseudo labels, L BCE (x,y2)>0.

[0277] It is worth noting that after completing the training of the label recognition model, the model inference stage can be entered based on the trained label recognition model. In the model inference stage, the interaction features {f1,f2,f3,…,f C}Calculate the matching probability between each candidate tag and the video to be identified, and filter out the candidate tags with matching probability greater than the matching probability threshold as the target tags that match the video to be identified. Figure 6 The dotted part corresponds to the steps.

[0278] The embodiments of the present application can at least achieve the following technical effects:

[0279] 1) A module for semantic decoupling and co-occurrence relationship transfer is designed to predict the labels of missing sample labels, thereby guiding the label recognition model to better train.

[0280] 2) The method is simple and efficient, has low requirements for sample labels (i.e., complete labeling is not required), and does not require multiple rounds of manual iterative labeling, saving time and labor costs.

[0281] 3) It has a wide range of applications and can be flexibly combined with mainstream multimodal classification frameworks. In addition to videos, it can also be applied to other forms of media such as plain text and graphics.

[0282] The following continues to describe the exemplary structure of the artificial intelligence-based label recognition model training device 2551 provided in the embodiment of the present application implemented as a software module. In some embodiments, such as Figure 2A As shown, the software modules in the artificial intelligence-based label recognition model training device 2551 stored in the memory 250 may include: a first acquisition module 25511, for acquiring sample content and a sample label of the sample content; wherein the sample label is one of a plurality of candidate labels; a first content feature extraction module 25512, for performing content feature extraction processing on the sample content through the content feature extraction network in the label recognition model to obtain content features of the sample content; a first label feature extraction module 25513, for performing label feature extraction processing on each candidate label through the label feature extraction network in the label recognition model to obtain label features of each candidate label; a first feature interaction module 25514, for extracting the sample content through the feature interaction network in the label recognition model The content features of the candidate tags are interactively processed with the tag features of each candidate tag to obtain the interactive features of each candidate tag; the first matching probability prediction module 25515 is used to perform matching probability prediction processing based on the interactive features of each candidate tag through the matching probability prediction network in the tag recognition model to obtain the matching probability between each candidate tag and the sample content; the pseudo-label screening module 25516 is used to determine the co-occurrence probability of different candidate tags based on the interactive features of each candidate tag, and determine the pseudo-label of the sample content based on the co-occurrence probability of different candidate tags and the sample label of the sample content; the training module 25517 is used to train the tag recognition model based on the matching probability of each candidate tag, the sample label of the sample content, the pseudo-label of the sample content, and the co-occurrence probability of different candidate tags.

[0283] In some embodiments, the training module 25517 is further used to: determine a first loss value based on the matching probability of each candidate label and the sample label of the sample content; determine a second loss value based on the matching probability of each candidate label and the pseudo label of the sample content; determine a third loss value based on the sample label of the sample content and the co-occurrence probability of different candidate labels; perform loss value fusion processing on the first loss value, the second loss value and the third loss value to obtain a fused loss value; and train a label recognition model based on the fused loss value.

[0284] In some embodiments, the pseudo-label screening module 25516 is further used to: combine and process multiple candidate labels to obtain multiple label pairs; wherein each label pair includes two candidate labels; and determine the co-occurrence probability of the two candidate labels in each label pair based on the interaction features corresponding to the two candidate labels in each label pair; the training module 25517 is further used to: determine the co-occurrence result of the two candidate labels in each label pair based on whether each label pair includes a sample label; determine the loss value of each label pair based on the co-occurrence result of the two candidate labels in each label pair and the co-occurrence probability of the two candidate labels in each label pair; and perform loss value fusion processing on the loss values ​​corresponding to the multiple label pairs to obtain a third loss value.

[0285] In some embodiments, the training module 25517 is also used to: perform the following processing for any label pair: perform difference calculation processing on the co-occurrence results of the two candidate labels in any label pair, and the co-occurrence probability of the two candidate labels in any label pair; perform weighted processing on the calculated differences according to at least one of the balance weight and the adjustment weight to obtain the loss value of any label pair; wherein the balance weight is negatively correlated with the label pair ratio, the label pair ratio represents the quantitative ratio between the first label pair and the second label pair, the first label pair and any label pair correspond to the same co-occurrence result, and the second label pair and any label pair correspond to different co-occurrence results; wherein the adjustment weight is positively correlated with the difference.

[0286] In some embodiments, the training module 25517 is also used to: perform the following processing for any label pair: when the two candidate labels in any label pair are both sample labels, determine the co-occurrence result of the two candidate labels in any label pair as co-occurrence; when at least one candidate label in any label pair is not a sample label, determine the co-occurrence result of the two candidate labels in any label pair as non-co-occurrence.

[0287] In some embodiments, the pseudo-label screening module 25516 is also used to: perform the following processing for any label pair: perform interaction feature fusion processing on the interaction features corresponding to the two candidate labels in any label pair to obtain fused interaction features; perform co-occurrence probability prediction processing based on the fused interaction features to obtain the co-occurrence probability of the two candidate labels in any label pair.

[0288] In some embodiments, the pseudo-label screening module 25516 is further used to: for any candidate label that is different from the sample label, perform the following processing: perform co-occurrence probability fusion processing on the co-occurrence probability of any candidate label and all sample labels of the sample content to obtain a fused co-occurrence probability; when the fused co-occurrence probability is greater than the co-occurrence probability threshold, determine any candidate label as a pseudo-label.

[0289] In some embodiments, the pseudo-label screening module 25516 is further used to: determine any candidate label as a pseudo-label when the fusion co-occurrence probability is greater than the co-occurrence probability threshold corresponding to the number of times the label recognition model has been trained; wherein the co-occurrence probability threshold is negatively correlated with the number of times the label recognition model has been trained.

[0290] In some embodiments, the first feature interaction module 25514 is further used to: perform the following processing on any candidate tag through the feature interaction network: perform feature transformation processing on the label features of any candidate tag through the feature transformation matrix; calculate the correlation between the label features after feature transformation processing and the content features of the sample content to obtain correlation features; normalize the correlation features to obtain correlation weight features; and weight the content features of the sample content according to the correlation weight features to obtain interaction features of any candidate tag.

[0291] In some embodiments, the sample content includes content of multiple modalities; the content feature extraction network in the tag recognition model includes a sequence encoder; the first content feature extraction module 25512 is also used to: convert the content of each modality in the sample content into multiple content units; and perform sequence encoding processing based on sequence relationships on all content units of multiple modalities through the sequence encoder to obtain content features of the sample content.

[0292] In some embodiments, the sample content includes content of multiple modalities; the content feature extraction network in the label recognition model includes modal feature extraction networks corresponding to the multiple modalities; the first content feature extraction module 25512 is also used to: perform modal feature extraction processing on the content of each modality in the sample content through the modal feature extraction network corresponding to each modality to obtain the modal features of each modality; perform modal feature fusion processing on the modal features corresponding to the multiple modalities to obtain the content features of the sample content.

[0293] The following continues to describe the exemplary structure of the artificial intelligence-based tag recognition device 2552 provided in the embodiment of the present application implemented as a software module. In some embodiments, such as Figure 2BAs shown, the software modules in the artificial intelligence-based tag recognition device 2552 stored in the memory 250 may include: a second acquisition module 25521 for acquiring the content to be recognized and a plurality of candidate tags; a second content feature extraction module 25522 for performing content feature extraction processing on the content to be recognized through the content feature extraction network in the tag recognition model to obtain the content features of the content to be recognized; a second tag feature extraction module 25523 for performing tag feature extraction processing on each candidate tag through the tag feature extraction network in the tag recognition model to obtain the tag features of each candidate tag; a second feature interaction module 25524 for performing tag feature extraction processing on each candidate tag through the tag recognition model to obtain the tag features of each candidate tag; The feature interaction network in the identification model performs feature interaction processing on the content features of the content to be identified and the label features of each candidate label to obtain the interaction features of each candidate label; the second matching probability prediction module 25525 is used to perform matching probability prediction processing based on the interaction features of each candidate label through the matching probability prediction network in the label identification model to obtain the matching probability between each candidate label and the content to be identified; the target label screening module 25526 is used to determine the candidate label whose matching probability is greater than the matching probability threshold as the target label that matches the content to be identified; wherein, the label identification model is trained according to the label recognition model training method based on artificial intelligence.

[0294] The embodiments of the present application provide a computer program product or computer program, which includes executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, causing the electronic device to perform the artificial intelligence-based label recognition model training method or artificial intelligence-based label recognition method described in the embodiments of the present application.

[0295] An embodiment of the present application provides a computer-readable storage medium storing executable instructions, wherein executable instructions are stored. When the executable instructions are executed by a processor, the processor will execute the artificial intelligence-based label recognition model training method or the artificial intelligence-based label recognition method provided in the embodiment of the present application.

[0296] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or various devices including one or any combination of the above memories.

[0297] In some embodiments, executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0298] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0299] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0300] The above are merely examples of the present application and are not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. A label recognition model training method based on artificial intelligence, characterized in that: include: Obtaining sample content and a sample label of the sample content; wherein the sample label is one of a plurality of candidate labels; Performing content feature extraction processing on the sample content through a content feature extraction network in a tag recognition model to obtain content features of the sample content; Perform label feature extraction on each candidate label through the label feature extraction network in the label recognition model to obtain the label features of each candidate label; Performing feature interaction processing on the content features of the sample content and the tag features of each candidate tag through a feature interaction network in the tag recognition model to obtain an interaction feature of each candidate tag; The matching probability prediction network in the tag recognition model performs matching probability prediction processing based on the interaction characteristics of each candidate tag to obtain the matching probability between each candidate tag and the sample content; Determining the co-occurrence probability of different candidate tags according to the interaction feature of each candidate tag, and determining the pseudo label of the sample content according to the co-occurrence probability of different candidate tags and the sample label of the sample content; The tag recognition model is trained according to the matching probability of each candidate tag, the sample tag of the sample content, the pseudo tag of the sample content, and the co-occurrence probability of different candidate tags.

2. The method according to claim 1, characterized in that The tag recognition model is trained according to the matching probability of each candidate tag, the sample tag of the sample content, the pseudo tag of the sample content, and the co-occurrence probability of different candidate tags, including: Determining a first loss value according to the matching probability of each candidate label and the sample label of the sample content; Determining a second loss value based on the matching probability of each candidate label and the pseudo label of the sample content; Determining a third loss value according to the sample label of the sample content and the co-occurrence probability of different candidate labels; Performing loss value fusion processing on the first loss value, the second loss value, and the third loss value to obtain a fused loss value; A label recognition model is trained according to the fusion loss value.

3. The method according to claim 2, characterized in that The determining of the co-occurrence probability of different candidate tags according to the interaction feature of each candidate tag includes: Combining multiple candidate labels to obtain multiple label pairs; each label pair includes two candidate labels; Determine the co-occurrence probability of the two candidate tags in each tag pair based on the interaction features corresponding to the two candidate tags in each tag pair; The determining of the third loss value according to the sample label of the sample content and the co-occurrence probability of different candidate labels includes: According to the result of whether each label pair includes the sample label, the co-occurrence result of the two candidate labels in each label pair is determined; Determine the loss value of each label pair based on the co-occurrence results of the two candidate labels in each label pair and the co-occurrence probability of the two candidate labels in each label pair; The loss values ​​corresponding to multiple label pairs are fused to obtain a third loss value.

4. The method according to claim 3, characterized in that The determining of the loss value of each label pair according to the co-occurrence result of the two candidate labels in each label pair and the co-occurrence probability of the two candidate labels in each label pair includes: For any tag pair, perform the following processing: Performing difference calculation processing on the co-occurrence results of the two candidate tags in any one tag pair and the co-occurrence probability of the two candidate tags in any one tag pair; Performing weighted processing on the calculated difference according to at least one of the balancing weight and the adjustment weight to obtain a loss value of any one label pair; The balance weight is negatively correlated with the label pair ratio, and the label pair ratio represents the ratio of the number of the first label pair to the number of the second label pair. The first label pair corresponds to the same co-occurrence result as any one of the label pairs, and the second label pair corresponds to a different co-occurrence result from any one of the label pairs. Therein, the adjustment weight is positively correlated with the difference.

5. The method according to claim 3, characterized in that The step of determining the co-occurrence result of two candidate tags in each tag pair according to the result of whether each tag pair includes the sample tag includes: For any tag pair, perform the following processing: When both candidate tags in any one tag pair are sample tags, determining a co-occurrence result of the two candidate tags in any one tag pair as co-occurrence; When at least one candidate tag in any one tag pair is not a sample tag, the co-occurrence result of the two candidate tags in the any one tag pair is determined to be non-co-occurrence.

6. The method according to claim 3, characterized in that The determining the co-occurrence probability of the two candidate tags in each tag pair according to the interaction features respectively corresponding to the two candidate tags in each tag pair includes: For any tag pair, perform the following processing: Performing interaction feature fusion processing on the interaction features corresponding to the two candidate tags in any tag pair to obtain a fused interaction feature; A co-occurrence probability prediction process is performed based on the fused interaction features to obtain the co-occurrence probability of the two candidate tags in any tag pair.

7. The method according to claim 1, characterized in that The determining of the pseudo label of the sample content according to the co-occurrence probabilities of different candidate labels and the sample label of the sample content includes: For any candidate label that is different from the sample label, perform the following processing: Performing co-occurrence probability fusion processing on the co-occurrence probability of any candidate tag and all sample tags of the sample content to obtain a fused co-occurrence probability; When the fused co-occurrence probability is greater than the co-occurrence probability threshold, any one of the candidate labels is determined as a pseudo label.

8. The method according to claim 7, characterized in that When the fusion co-occurrence probability is greater than a co-occurrence probability threshold, determining any one of the candidate labels as a pseudo label includes: When the fusion co-occurrence probability is greater than the co-occurrence probability threshold corresponding to the number of times the label recognition model has been trained, any one of the candidate labels is determined as a pseudo label; Among them, the co-occurrence probability threshold is negatively correlated with the number of times the label recognition model has been trained.

9. The method according to claim 1, characterized in that The feature interaction network in the tag recognition model performs feature interaction processing on the content features of the sample content and the tag features of each candidate tag to obtain the interaction features of each candidate tag, including: The following processing is performed for any candidate label through the feature interaction network: Performing feature transformation processing on the label features of any one of the candidate labels through a feature transformation matrix; Calculate the correlation between the label features after feature transformation and the content features of the sample content to obtain a correlation feature; Normalizing the correlation feature to obtain a correlation weight feature; The content features of the sample content are weighted according to the correlation weight features to obtain the interaction features of any one of the candidate tags.

10. The method according to any one of claims 1 to 9, characterized in that The sample content includes content of multiple modalities; the content feature extraction network in the tag recognition model includes a sequence encoder; the content feature extraction processing of the sample content by the content feature extraction network in the tag recognition model to obtain the content features of the sample content includes: Converting the content of each modality in the sample content into multiple content units; The content units of the multiple modalities are subjected to sequence encoding processing based on sequence relationships by using a sequence encoder to obtain content features of the sample content.

11. The method according to any one of claims 1 to 9, characterized in that The sample content includes content of multiple modalities; the content feature extraction network in the tag recognition model includes modality feature extraction networks corresponding to the multiple modalities respectively; The performing content feature extraction processing on the sample content by the content feature extraction network in the tag recognition model to obtain the content features of the sample content includes: Performing modal feature extraction processing on the content of each modality in the sample content through a modal feature extraction network corresponding to each modality to obtain modal features of each modality; Modal feature fusion processing is performed on the modal features corresponding to the multiple modalities to obtain content features of the sample content.

12. A label recognition method based on artificial intelligence, characterized in that: include: Obtain the content to be identified and multiple candidate labels; Performing content feature extraction processing on the content to be identified by a content feature extraction network in a tag recognition model to obtain content features of the content to be identified; Perform label feature extraction on each candidate label through the label feature extraction network in the label recognition model to obtain the label features of each candidate label; Performing feature interaction processing on the content features of the content to be identified and the tag features of each candidate tag through a feature interaction network in the tag recognition model to obtain an interaction feature of each candidate tag; The matching probability prediction network in the tag recognition model performs matching probability prediction processing based on the interaction characteristics of each candidate tag to obtain the matching probability between each candidate tag and the content to be recognized; Determine the candidate tag with a matching probability greater than a matching probability threshold as the target tag that matches the content to be identified; The label recognition model is trained according to the method according to any one of claims 1 to 11.

13. A label recognition model training device based on artificial intelligence, characterized in that: include: A first acquisition module is configured to acquire sample content and a sample label of the sample content; wherein the sample label is one of a plurality of candidate labels; a first content feature extraction module, configured to perform content feature extraction processing on the sample content through a content feature extraction network in a tag recognition model to obtain content features of the sample content; A first label feature extraction module is used to perform label feature extraction processing on each candidate label through the label feature extraction network in the label recognition model to obtain the label feature of each candidate label; A first feature interaction module is configured to perform feature interaction processing on the content features of the sample content and the tag features of each candidate tag through a feature interaction network in the tag recognition model to obtain an interaction feature of each candidate tag; A first matching probability prediction module is configured to perform matching probability prediction processing based on the interaction features of each candidate tag through a matching probability prediction network in a tag recognition model to obtain a matching probability between each candidate tag and the sample content; a pseudo-label screening module, configured to determine the co-occurrence probability of different candidate labels based on the interaction characteristics of each candidate label, and determine the pseudo-label of the sample content based on the co-occurrence probability of different candidate labels and the sample label of the sample content; The training module is used to train a tag recognition model according to the matching probability of each candidate tag, the sample tag of the sample content, the pseudo-tag of the sample content, and the co-occurrence probability of different candidate tags.

14. A label recognition device based on artificial intelligence, characterized in that: include: The second acquisition module is used to obtain the content to be identified and multiple candidate tags; A second content feature extraction module is configured to perform content feature extraction processing on the content to be identified through a content feature extraction network in a tag recognition model to obtain content features of the content to be identified; The second label feature extraction module is used to perform label feature extraction processing on each candidate label through the label feature extraction network in the label recognition model to obtain the label feature of each candidate label; A second feature interaction module is configured to perform feature interaction processing on the content features of the content to be identified and the tag features of each candidate tag through a feature interaction network in the tag recognition model to obtain an interaction feature of each candidate tag; A second matching probability prediction module is configured to perform matching probability prediction processing based on the interaction features of each candidate tag through a matching probability prediction network in the tag recognition model to obtain a matching probability between each candidate tag and the content to be recognized; A target tag screening module is used to determine a candidate tag with a matching probability greater than a matching probability threshold as a target tag matching the content to be identified; The label recognition model is trained according to the method according to any one of claims 1 to 11.

15. An electronic device, characterized in that: include: a memory for storing executable instructions; A processor, configured to implement the method according to any one of claims 1 to 11, or the method according to claim 12, when executing the executable instructions stored in the memory.

16. A computer-readable storage medium, characterized in that Executable instructions are stored for implementing the method according to any one of claims 1 to 11 or the method according to claim 12 when executed by a processor.

17. A computer program product, characterized in that The method comprises executable instructions for implementing the method according to any one of claims 1 to 11 or the method according to claim 12 when executed by a processor.

Images