A method and apparatus for generating a label
By combining a multimodal approach with a contrastive language-image pre-trained model and a generative pre-trained converter model to generate tags for UGC text and image content, the accuracy problem of the single-modal method is solved, and more efficient text and image content tag generation is achieved.
Patent Information
- Application Number
- CN202310707805.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-14
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2043-06-14
AI Technical Summary
Existing technologies have poor accuracy in tagging and classifying UGC text and image content using single-modal methods, and the training process for machine learning models is cumbersome and time-consuming.
A multimodal approach is adopted, using a contrastive language-image pre-trained model and a generative pre-trained converter model to identify images and text content in the image-text content, obtain a set of labels, and generate the final set of labels through confidence fusion and filtering.
It improves the accuracy of label classification, simplifies the training process, and enables more accurate label generation for text and image content.
Smart Images

Figure CN116824235B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer technology, and in particular to a method and apparatus for generating tags. Background Technology
[0002] User-generated content (UGC) refers to various forms of text and image content created and published by ordinary users through the internet, social media, and other channels. Examples include text and image content on Weibo and Xiaohongshu, as well as comments on others, products, shops, restaurants, and other content. This content is typically characterized by diversity, real-time nature, and timeliness. In the fields of machine learning and artificial intelligence, it is often necessary to add one or more tags or identifiers to data—a process known as tagging—to facilitate data classification, searching, and analysis.
[0003] For the task of tagging UGC (User Generated Content) images and text, a common approach is to first use a machine learning classifier to automatically categorize the images or text in the user-generated content and assign initial tags, followed by manual verification and correction. The cost of the second step, manual verification and correction, depends primarily on the accuracy and robustness of the machine learning model; therefore, automating the tagging process in the first step becomes crucial to the entire task.
[0004] The above-mentioned tagging scheme has the following drawbacks: classifying images or text in user-generated content means considering one modality, which leads to poor tag accuracy for some business scenarios that heavily rely on two modalities; the training process of machine learning classifiers is cumbersome and time-consuming. Summary of the Invention
[0005] To overcome the problems existing in related technologies, embodiments of the present invention provide a method and apparatus for generating labels. The technical solution is as follows:
[0006] According to a first aspect of an embodiment of the present invention, a method for generating a tag is provided, comprising:
[0007] For the image content in the text and image content to be processed, the first model based on the contrastive language-image pre-trained model is used to identify the image content and obtain the first label set;
[0008] For the text content in the image and text content to be processed, a second model based on a generative pre-trained converter model is used to identify the text content and obtain a second tag set;
[0009] The first tag set and the second tag set are merged to generate the final tag set of the text and image content to be processed.
[0010] In one embodiment of this application, the first tag set and the second tag set are merged to generate the final tag set of the text and image content to be processed, including:
[0011] A third tag set is generated by merging the first tag set and the second tag set, wherein the first tag set and the second tag set include tags and their corresponding confidence scores; for tags that exist in both the first tag set and the second tag set, the tags are added to the third tag set, and the tag with the larger confidence score is used as the confidence score of the tag; for other tags in the first tag set and the second tag set, the tags and their corresponding confidence scores are directly added to the third tag set.
[0012] The third tag set is filtered according to a preset confidence threshold to generate the final tag set.
[0013] In one embodiment of this application, a first model based on a contrastive language-image pre-trained model is used to recognize image content and obtain a first label set, including:
[0014] Based on the image-text pairing dataset, construct the first training sample set;
[0015] The first model is obtained by training the contrastive language-image pre-trained model using the first training sample set;
[0016] The image content of the text and image content to be processed is input into the first model to obtain the first tag set.
[0017] In one embodiment of this application, a second model based on a generative pre-trained converter model is used to recognize text content and obtain a second tag set, including:
[0018] A second training sample set is constructed based on the text content and the corresponding tags and reasons;
[0019] The second model is obtained by training the generative pre-trained converter model using the second training sample set;
[0020] The text content of the image and text content to be processed is input into the second model to obtain the second tag set.
[0021] According to a second aspect of an embodiment of the present invention, a label generation apparatus is provided, comprising:
[0022] The first processing module is used to identify the image content in the text and image content to be processed using a first model based on a contrastive language-image pre-trained model and obtain a first tag set.
[0023] The second processing module uses a second model based on a generative pre-trained converter model to identify the text content in the text and image content to be processed, and obtains a second tag set.
[0024] The third processing module is used to merge the first tag set and the second tag set to generate the final tag set of the text and image content to be processed.
[0025] In one embodiment of this application, the third processing module includes:
[0026] A fusion unit is used to fuse a first tag set and a second tag set to generate a third tag set. The first tag set and the second tag set include tags and their corresponding confidence scores. For tags that exist in both the first tag set and the second tag set, the tags are added to the third tag set, and the tag with the larger confidence score is used as its confidence score. For other tags in the first tag set and the second tag set, the tags and their corresponding confidence scores are directly added to the third tag set.
[0027] The generation unit is used to filter the third tag set according to a preset confidence threshold and generate the final tag set.
[0028] In one embodiment of this application, the first processing module includes:
[0029] The first building unit is used to construct the first training sample set based on the image-text pairing dataset;
[0030] The first training unit is used to train the contrastive language-image pre-trained model using the first training sample set to obtain the first model.
[0031] The first processing unit is used to input the image content of the text and image content to be processed into the first model to obtain the first tag set.
[0032] In one embodiment of this application, the second processing module includes:
[0033] The second building unit is used to construct a second training sample set based on the text content and the corresponding tags and reasons;
[0034] The second training unit is used to train the generative pre-trained converter model using the second training sample set to obtain the second model;
[0035] The second processing unit is used to input the text content of the image and text content to be processed into the second model to obtain the second tag set.
[0036] According to a third aspect of the present invention, a label generation apparatus is provided, comprising:
[0037] processor;
[0038] Memory used to store processor-executable instructions;
[0039] The processor is configured as follows:
[0040] For the image content in the text and image content to be processed, the first model based on the contrastive language-image pre-trained model is used to identify the image content and obtain the first label set;
[0041] For the text content in the image and text content to be processed, a second model based on a generative pre-trained converter model is used to identify the text content and obtain a second tag set;
[0042] The first tag set and the second tag set are merged to generate the final tag set of the text and image content to be processed.
[0043] According to a fourth aspect of an embodiment of the present invention, a computer-readable storage medium is provided having computer instructions stored thereon that, when executed by a processor, implement the steps of any of the methods in the first aspect of the embodiments of the present invention.
[0044] The technical solution provided by the embodiments of the present invention adopts a multimodal mode when generating tags for graphic content. It is not limited to text content or image content in graphic content, but uses two modes to tag graphic content at the same time, and merges the tags obtained from the two modes. The accuracy is much higher than the method of tagging using only one mode.
[0045] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description
[0046] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.
[0047] Figure 1 This is a flowchart illustrating a method for generating a label according to an exemplary embodiment.
[0048] Figure 2 This is a flowchart illustrating a method for generating a label according to an exemplary embodiment.
[0049] Figure 3 This is a block diagram of a tag generation apparatus according to an exemplary embodiment.
[0050] Figure 4 This is a block diagram of a tag generation apparatus according to an exemplary embodiment.
[0051] Figure 5 This is a block diagram of a tag generation apparatus according to an exemplary embodiment. Detailed Implementation
[0052] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.
[0053] Currently, users independently create and publish various forms of UGC (User Generated Content) through the internet, social media, and other channels, such as articles, images, videos, and comments. This content typically features diversity, real-time updates, and timeliness. In the fields of machine learning and artificial intelligence, it is often necessary to add one or more tags or identifiers to data—a process known as tagging—to facilitate data classification, searching, and analysis.
[0054] Embodiments of the present invention provide a method for generating tags, which can be applied to terminals such as servers and computers used for tag generation. Figure 1 As shown, the method includes the following steps S101 to S103:
[0055] In step S101, for the image content in the text and image content to be processed, the first model based on the contrastive language-image pre-trained model is used to identify the image content and obtain the first tag set.
[0056] The content to be processed refers to the content uploaded by users to the server for publication. This content must include at least two types of information: images and text. Examples include user-posted Weibo posts and Xiaohongshu posts, which contain both image and text components.
[0057] In step S102, for the text content in the image and text content to be processed, a second model based on a generative pre-trained converter model is used to identify the text content and obtain a second tag set.
[0058] In step S103, the first tag set and the second tag set are merged to generate the final tag set of the text and image content to be processed.
[0059] The technical solution of this application adopts a multimodal mode when generating tags for text and image content. It is not limited to text content or image content in the text and image content, but uses two modes to tag the text and image content at the same time, and merges the tags obtained from the two modes. The accuracy is much higher than the method of tagging using only one mode.
[0060] In one embodiment of this application, step 103 involves merging the first tag set and the second tag set to generate the final tag set for the text and image content to be processed, including the following steps A1-A2:
[0061] In step A1, a third tag set is generated by merging the first tag set and the second tag set, wherein the first tag set and the second tag set include tags and their corresponding confidence scores; for tags that exist in both the first tag set and the second tag set, the tags are added to the third tag set, and the tag with the larger confidence score is used as the confidence score of the tag; for other tags in the first tag set and the second tag set, the tags and their corresponding confidence scores are directly added to the third tag set.
[0062] In step A2, the third tag set is filtered according to a preset confidence threshold to generate the final tag set.
[0063] This embodiment provides a specific method for fusing the tagging results output by the models of different modalities to obtain the tags for the text and image content after simultaneously using two modalities to tag the text and image content.
[0064] In one embodiment of this application, step 101 includes the following steps: Step B1-Step B3:
[0065] In step B1, a first training sample set is constructed based on the image-text pairing dataset.
[0066] In step B2, the first model is obtained by training the contrastive language-image pre-trained model using the first training sample set.
[0067] In step B3, the image content of the text and image content to be processed is input into the first model to obtain the first tag set.
[0068] In one embodiment of this application, step 102 includes the following steps: Steps C1-C3:
[0069] In step C1, a second training sample set is constructed based on the text content and the corresponding tags and reasons.
[0070] In step C2, the second model is obtained by training the generative pre-trained converter model using the second training sample set.
[0071] In step C3, the text content of the image and text content to be processed is input into the second model to obtain the second tag set.
[0072] The implementation process will be described in detail below through examples.
[0073] Figure 2 This is a schematic flowchart illustrating a method for generating a label according to an exemplary embodiment. This method can be executed by a server. Figure 2 As shown, it includes the following steps:
[0074] Step S201: Construct the first training sample set based on the image-text pairing dataset.
[0075] In this step, a large number of image-text pairs are collected, where the text in the pair is a description of the corresponding image and can serve as a label for that image.
[0076] For example, the format of the first training sample set is as follows:
[0077] / path / to / image tags (each line represents a training sample)
[0078] Here, / path / to / image represents the storage path of the image, and tags represent the text paired with the image.
[0079] An example of the first training sample set is as follows:
[0080] / data / images / 16958461638.jpg Plain wool, cotton doll
[0081] / data / images / 16958449895.jpg Long-legged body, cotton doll
[0082] / data / images / 16958498513.jpg (Image showing a fan of celebrities)
[0083] / data / images / 16958461567.jpg Movies, Idol Chasing, Celebrities 2
[0084] / data / images / 16958466534.jpg Beauty
[0085] …(slightly)
[0086] Step S202: Use the first training sample set to train the contrastive language-image pre-trained model to obtain the first model.
[0087] CLIP (Contrastive Language-Image Pre-Training) is used as the pre-trained model, and it is trained using actual business data (i.e., the first training sample set mentioned above). The trained model is used as the first model, and the first model is used to label the image content in the text and image content to be processed.
[0088] CLIP's training data consists of text-image pairs: an image and its corresponding text description (i.e., label). The CLIP pre-trained model learns the matching relationship between text-image pairs through contrastive learning. The CLIP pre-trained model is multimodal, and its main structure includes two encoders: a Text Encoder and an Image Encoder. The Text Encoder encodes the text, while the Image Encoder encodes the image. CLIP inputs paired images and text into the Image Encoder and Text Encoder respectively, obtaining image vectors and text vectors. These image and text vectors are then mapped to the same multimodal space, resulting in new, directly comparable image and text vector representations. The cosine similarity between the image and text vectors is then calculated. Finally, the objective function of contrastive learning is to maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs.
[0089] Step S203: Based on the text content and the corresponding tags and reasons, construct the second training sample set.
[0090] For example, the second training sample set is formatted as follows (each line represents a training sample):
[0091] {"prompt":"Celebrity 3's flying kiss"} So exciting! ->","completion":"Celebrity Stanning: 0.99 Reason: Celebrity 3 was mentioned in the article; she is a celebrity."}
[0092] {"prompt":"I'm getting ready to buy the Celebrity 2 doll! Any recommendations, everyone?->","completion":"Celebrity Suggestion: 0.89 Reason: The article mentions Celebrity 2, who is a celebrity\nCotton Doll Suggestion: 0.88 Reason: The article mentions dolls, so this text is likely related to cotton dolls."}
[0093] ...(slightly)
[0094] Here, prompt represents the text input to the model, and completion represents the output of the model. Each training sample includes text content, corresponding labels, and reasons.
[0095] Step S204: Use the second training sample set to train the generative pre-trained converter model to obtain the second model.
[0096] For example, the GPT-3 (Generative Pre-Trained Transformer 3) model is used as the pre-trained model, and it is trained using actual business data (i.e., the second training sample set mentioned above) to obtain the second model. The second model is then used to label the text content in the image and text content to be processed.
[0097] For example, fine-tuning models can be created using the OpenAI API (an application programming interface provided by OpenAI to developers):
[0098] openai api fine_tunes.create-t<TRAIN_FILE_ID_OR_PATH> -m<BASE_MODEL>
[0099] This interface allows you to initiate a fine-tuning task: using<TRAIN_FILE_ID_OR_PATH> Data for fine-tuning<BASE_MODEL> Model.
[0100] Based on the above steps, the first and second models required in this application can be obtained, which can respectively label the image content and text content in the text and image content to be processed. The execution order of the steps to generate the first and second models is not important. The following describes the processing flow for labeling the text and image content to be processed.
[0101] Step S205: Input the image content of the text and image content to be processed into the first model to obtain the first tag set.
[0102] For example, the image from the user-uploaded text-image content to be processed is input into an image encoder to obtain an image embedding vector v_img. Predefined category labels are then input into a text encoder to obtain a text embedding vector set [v_label1, v_label2, ...]. The cosine similarity between v_img and [v_label1, v_label2, ...] is then calculated to obtain a matching score; for example, label1: 0.98; label2: 0.87. This yields the first label set corresponding to the image content in the text-image content to be processed. The first label set includes each label and its corresponding confidence score.
[0103] Step S206: Input the text content of the image and text content to be processed into the second model to obtain the second tag set.
[0104] For example, this can be done via the OpenAI API: openai api completions.create-m<FINE_TUNED_MODEL> -p<YOUR_PROMPT> This interface allows you to use a trained GPT-3 model to predict inputs.<YOUR_PROMPT> The category labels corresponding to the text are used to obtain the second set of labels corresponding to the text content of the image and text content to be processed. The second set of labels includes each label and the confidence level of each label.
[0105] Step S207: Merge the first tag set and the second tag set to generate a third tag set: For tags that exist in both the first tag set and the second tag set, add the tags to the third tag set and obtain the confidence score with the larger value as the confidence score of the tag; for other tags in the first tag set and the second tag set, directly add the tags and their corresponding confidence scores to the third tag set.
[0106] The following example illustrates how to merge the first and second tag sets to generate a third tag set. For the text and image content to be processed:
[0107] Suppose the first set of labels output by the first model is as follows, in the format (label: confidence score;):
[0108] Cotton doll: 0.87; Anime / Manga: 0.77; Idol worship: 0.43
[0109] Suppose the second set of labels output by the second model is as follows, in the format (label: confidence score;):
[0110] Cotton doll: 0.9, Stanning: 0.51
[0111] The fusion rules are as follows: if a label exists in two label sets at the same time, the one with the higher confidence score is added to the third label set; otherwise, it is directly added to the third label set.
[0112] According to the above fusion rules, the final fusion result is:
[0113] Cotton doll: 0.9; Anime / Manga: 0.77; Idol worship: 0.51
[0114] Step S208: Filter the third tag set according to the confidence level preset threshold to generate the final tag set.
[0115] For example, the results are filtered based on a predetermined confidence threshold (e.g., 0.6) to obtain the final tag set of the text and image content to be processed: cotton doll, anime.
[0116] This application employs a multimodal model, simultaneously processing images and text in UGC content and fusing the results to obtain the final label, achieving a significantly higher accuracy than methods using only one modality. The output categories of both the image and text data classification models can be flexibly specified. Both image and text data can be classified into different domain labels simultaneously, such as "doll, cute pet, coffee" (general domain) and "happy, angry, sad" (emotional domain). For text modal content, the generative model can provide reasons explaining why the current content has a certain probability (confidence) of being classified into this category, along with the classification result and confidence score. The training process is simple: for text modal content, there is no need to train the encoder and classifier in series; instead, training is performed end-to-end.
[0117] The following are embodiments of the apparatus of the present invention, which can be used to execute embodiments of the method of the present invention.
[0118] Figure 3 This is a block diagram illustrating a tag generation apparatus according to an exemplary embodiment. The tag generation apparatus can be a server or part of a server, or a terminal or part of a terminal. The tag generation apparatus can be implemented as part or all of an electronic device through software, hardware, or a combination of both. Figure 3 As shown, the label generating apparatus includes:
[0119] The first processing module 301 is used to identify the image content in the text and image content to be processed using a first model based on a contrastive language-image pre-trained model and obtain a first tag set.
[0120] The second processing module 302 uses a second model based on a generative pre-trained converter model to identify the text content in the text content to be processed and obtains a second tag set.
[0121] The third processing module 303 is used to merge the first tag set and the second tag set to generate the final tag set of the text and image content to be processed.
[0122] In one embodiment, the third processing module includes:
[0123] A fusion unit is used to fuse a first tag set and a second tag set to generate a third tag set. The first tag set and the second tag set include tags and their corresponding confidence scores. For tags that exist in both the first tag set and the second tag set, the tags are added to the third tag set, and the tag with the larger confidence score is used as its confidence score. For other tags in the first tag set and the second tag set, the tags and their corresponding confidence scores are directly added to the third tag set.
[0124] The generation unit is used to filter the third tag set according to a preset confidence threshold and generate the final tag set.
[0125] In one embodiment, the first processing module includes:
[0126] The first building unit is used to construct the first training sample set based on the image-text pairing dataset;
[0127] The first training unit is used to train the contrastive language-image pre-trained model using the first training sample set to obtain the first model.
[0128] The first processing unit is used to input the image content of the text and image content to be processed into the first model to obtain the first tag set.
[0129] In one embodiment, the second processing module includes:
[0130] The second building unit is used to construct a second training sample set based on the text content and the corresponding tags and reasons;
[0131] The second training unit is used to train the generative pre-trained converter model using the second training sample set to obtain the second model;
[0132] The second processing unit is used to input the text content of the image and text content to be processed into the second model to obtain the second tag set.
[0133] Figure 4 This is a block diagram illustrating a tag generation apparatus 40 according to an exemplary embodiment. The tag generation apparatus may be a server or part of a server, or a terminal or part of a terminal. The tag generation apparatus includes:
[0134] Processor 4001;
[0135] Memory 4002 is used to store executable instructions of processor 4001;
[0136] The processor 4001 is configured as follows:
[0137] For the image content in the text and image content to be processed, the first model based on the contrastive language-image pre-trained model is used to identify the image content and obtain the first label set;
[0138] For the text content in the image and text content to be processed, a second model based on a generative pre-trained converter model is used to identify the text content and obtain a second tag set;
[0139] The first tag set and the second tag set are merged to generate the final tag set of the text and image content to be processed.
[0140] Figure 5 This is a block diagram illustrating a tag generation apparatus 800 according to an exemplary embodiment. The apparatus may be a computer, server, etc.
[0141] The device may include one or more of the following components: processing component 802, memory 804, power supply component 806, multimedia component 808, audio component 810, input / output (I / O) interface 812, sensor component 814, and communication component 816.
[0142] Processing component 802 typically controls the overall operation of device 800, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 802 may include one or more processors 820 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.
[0143] Memory 804 is configured to store various types of data to support the operation of device 800. Examples of such data include instructions for any application or method operating on device 800, contact data, phonebook data, messages, pictures, videos, etc. Memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.
[0144] Power supply component 806 provides power to various components of device 800. Power supply component 806 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to device 800.
[0145] Multimedia component 808 includes a screen that provides an output interface between device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When device 800 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.
[0146] Audio component 810 is configured to output and / or input audio signals. For example, audio component 810 includes a microphone (MIC) configured to receive external audio signals when device 800 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 804 or transmitted via communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.
[0147] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.
[0148] Sensor assembly 814 includes one or more sensors for providing status assessments of various aspects of device 800. For example, sensor assembly 814 may detect the on / off state of device 800, the relative positioning of components such as the display and keypad of device 800, changes in the position of device 800 or a component of device 800, the presence or absence of user contact with device 800, the orientation or acceleration / deceleration of device 800, and temperature changes of device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 814 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.
[0149] Communication component 816 is configured to facilitate wired or wireless communication between device 800 and other devices. Device 800 can access wireless networks based on communication standards, such as walkie-talkie networks, WiFi, 2G, 3G, 4G, or 5G, or combinations thereof. In one exemplary embodiment, communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 816 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.
[0150] In an exemplary embodiment, the apparatus 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.
[0151] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, which can be executed by a processor 820 of the device 800 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.
[0152] A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of a device 800, enables the device 800 to perform the aforementioned tag generation method, the method comprising:
[0153] For the image content in the text and image content to be processed, the first model based on the contrastive language-image pre-trained model is used to identify the image content and obtain the first label set;
[0154] For the text content in the image and text content to be processed, a second model based on a generative pre-trained converter model is used to identify the text content and obtain a second tag set;
[0155] The first tag set and the second tag set are merged to generate the final tag set of the text and image content to be processed.
[0156] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention described herein. This application is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not invented herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the following claims.
[0157] It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.
Claims
1. A method for generating a label, characterized in that, include: For the image content in the text and image content to be processed, the first model based on the contrastive language-image pre-trained model is used to identify the image content and obtain the first label set; For the text content in the image and text content to be processed, a second model based on a generative pre-trained converter model is used to identify the text content and obtain a second tag set; The first tag set and the second tag set are merged to generate the final tag set for the text and image content to be processed, including: A third tag set is generated by merging the first tag set and the second tag set, wherein the first tag set and the second tag set include tags and their corresponding confidence scores; for tags that exist in both the first tag set and the second tag set, the tags are added to the third tag set, and the tag with the larger confidence score is used as the confidence score of the tag; for other tags in the first tag set and the second tag set, the tags and their corresponding confidence scores are directly added to the third tag set. The third tag set is filtered according to a preset confidence threshold to generate the final tag set; Using a first model based on a contrastive language-image pre-trained model, image content is identified to obtain a first set of labels, including: Based on the image-text pairing dataset, construct the first training sample set; The first model is obtained by training the contrastive language-image pre-training model using the first training sample set; the image content of the text and image content to be processed is input into the first model to obtain the first tag set; A second model based on a generative pre-trained converter model is used to identify the text content and obtain a second set of labels, including: A second training sample set is constructed based on the text content and the corresponding tags and reasons; The second model is obtained by training the generative pre-trained converter model using the second training sample set; The text content of the image and text content to be processed is input into the second model to obtain the second tag set.
2. A label generating apparatus, characterized in that, include: The first processing module is used to identify the image content in the text and image content to be processed using a first model based on a contrastive language-image pre-trained model and obtain a first tag set. The second processing module uses a second model based on a generative pre-trained converter model to identify the text content in the text and image content to be processed, and obtains a second tag set. The third processing module is used to merge the first tag set and the second tag set to generate the final tag set of the text and image content to be processed; The third processing module includes: A fusion unit is used to fuse a first tag set and a second tag set to generate a third tag set. The first tag set and the second tag set include tags and their corresponding confidence scores. For tags that exist in both the first tag set and the second tag set, the tags are added to the third tag set, and the tag with the larger confidence score is used as its confidence score. For other tags in the first tag set and the second tag set, the tags and their corresponding confidence scores are directly added to the third tag set. The generation unit is used to filter the third tag set according to a preset confidence threshold and generate the final tag set. The first processing module includes: The first building unit is used to construct the first training sample set based on the image-text pairing dataset; The first training unit is used to train the contrastive language-image pre-trained model using the first training sample set to obtain the first model. The first processing unit is used to input the image content of the text and image content to be processed into the first model to obtain the first tag set; The second processing module includes: The second building unit is used to construct a second training sample set based on the text content and the corresponding tags and reasons; The second training unit is used to train the generative pre-trained converter model using the second training sample set to obtain the second model; The second processing unit is used to input the text content of the image and text content to be processed into the second model to obtain the second tag set.
3. A label generating apparatus, characterized in that, The apparatus implements the method as described in claim 1, the apparatus comprising: processor; Memory used to store processor-executable instructions; The processor is configured as follows: For the image content in the text and image content to be processed, the first model based on the contrastive language-image pre-trained model is used to identify the image content and obtain the first label set; For the text content in the image and text content to be processed, a second model based on a generative pre-trained converter model is used to identify the text content and obtain a second tag set; The first tag set and the second tag set are merged to generate the final tag set of the text and image content to be processed.
4. A computer-readable storage medium storing computer instructions thereon, characterized in that, When executed by the processor, this instruction implements the steps of the method of claim 1.
Citation Information
Patent Citations
Method and device for generating text for media, computing equipment and storage medium
CN113722527A
Social event abstract generation method and device, terminal equipment and medium
CN115905598A
