Method, device and medium for generating tags for user-generated content

Abstract

Embodiments of the present disclosure relate to a method, device and medium for generating labels for user-generated content, including: acquiring content data corresponding to the user-generated content; classifying the acquired content data into first content data and second content data; extracting a first feature of the first content data based on a first content analysis tool and extracting a second feature of the second content data based on a second content analysis tool; interacting the first feature and the second feature to obtain an interactive third feature; and generating labels for the user-generated content based on the first feature, the second feature and the third feature according to the time correlation of the content data.

Description

Technical Field

[0001] Embodiments of the present disclosure generally relate to the field of computer technology, and more particularly to a method, a computing device, and a computer-readable storage medium for generating tags for user-generated content (UGC). Background Art

[0002] Tags, also known as hashtags, are often used in social media. Content using the same topic will be automatically aggregated based on the topic, allowing users to discuss related content around the same topic.

[0003] With the advent of the Web 2.0 era, various social media have emerged. Various multimedia data (images, videos, audio) on social media platforms have exploded, and we have entered the era of social multimedia big data. Social media (Weibo, Youku, etc.) allow users to generate content (User Generated Content, UGC) and publish, share, communicate and disseminate it through social networks. Multimedia as an information carrier can usually express richer and more vivid content, so more and more multimedia content appears on social media.

[0004] The tags of user-generated content that can be used in the prior art are usually generated based on the following methods: First, when users edit content, they manually enter topics and associate content. In this case, even if different users publish content for the same topic center, they may add different topic tags due to the expression habits of different users. The content that can be associated with the topic is limited and has great limitations. Second, new topics can be generated based on the topic tags manually entered by users through custom rules or machine learning. The templates required for this type of method rely on a large amount of prior knowledge, which is costly and difficult to migrate quickly and effectively.

[0005] In summary, the conventional solution for generating tags for user-generated content has the following disadvantages: it relies on user editing or a large amount of prior knowledge, is costly, and is difficult to migrate quickly and effectively. Summary of the invention

[0006] In view of the above problems, the present disclosure provides a method, device and medium for generating tags for user-generated content. Based on the above scheme, the topic tags expected by users can be accurately generated based on the different modal features (such as text, image or video) in UGC, which can maintain the differences of different modal features and understand one modality in the context of another modality, thereby improving the accuracy of topic tags.

[0007] According to a first aspect of the present disclosure, there is provided a method for generating labels for user-generated content, comprising: acquiring content data corresponding to the user-generated content; classifying the acquired content data into first content data and second content data; extracting a first feature of the first content data based on a first content analysis tool and extracting a second feature of the second content data based on a second content analysis tool; interacting the first feature and the second feature to obtain an interactive third feature; and generating labels for the user-generated content based on the first feature, the second feature, and the third feature according to the time correlation of the content data.

[0008] According to a second aspect of the present disclosure, a computing device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method of the first aspect of the present disclosure.

[0009] In a third aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method of the first aspect of the present disclosure.

[0010] In some embodiments, classifying the acquired content data into the first content data and the second content data includes: determining the classified second content data as one of image data, an image data set including a plurality of images, and video data.

[0011] In some embodiments, extracting a second feature of the second content data based on a second content analysis tool includes: in response to the second content data being image data, inputting the image data into a convolutional neural network; extracting image features based on neural network learning; and performing dimensionality reduction processing on the extracted image features to obtain the second feature.

[0012] In some embodiments, extracting a second feature of the second content data based on a second content analysis tool includes: in response to the second content data being an image dataset including multiple images, inputting the image dataset into a convolutional neural network in batches; extracting multiple image features based on neural network learning; obtaining image weights corresponding to the multiple images; and performing dimensionality reduction processing on the extracted multiple image features and performing fusion based on the obtained image weights, thereby obtaining a second feature.

[0013] In some embodiments, extracting a second feature of the second content data based on a second content analysis tool includes: in response to the second content data being video data, performing frame extraction processing on the video data according to the time correlation of the content data, thereby obtaining multiple frames of image data; inputting the obtained multiple frames of image data into a convolutional neural network; extracting multiple image features based on neural network learning; and performing dimensionality reduction processing and fusion on the extracted multiple image features, thereby obtaining a second feature.

[0014] In some embodiments, interacting the first feature and the second feature to obtain a third feature of the interaction includes: connecting the first feature and the second feature to obtain a fused feature; inputting the fused feature into a graph neural network; and obtaining a feature vector output by the graph neural network as the third feature of the interaction.

[0015] In some embodiments, generating a label for user-generated content based on the first feature, the second feature, and the third feature according to the time correlation of the content data includes: setting a first weight, a second weight, and a third weight, respectively, for the first feature, the second feature, and the interacted third feature; adjusting the relationship between the first weight, the second weight, and the third weight based on the time correlation of the content data; and generating a label for the user-generated content using a multimodal fusion label generator based on the adjusted first weight, second weight, and third weight.

[0016] In some embodiments, adjusting the relationship between the first weight, the second weight, and the third weight based on the time correlation of the content data includes: determining, based on the time correlation of the content data, whether the search content data is data that is strongly dependent on time information and data that is weakly dependent on time information; in response to the content data being data that is strongly dependent on time information, increasing the second weight; and in response to the content data being data that is weakly dependent on time information, increasing the first weight.

[0017] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements.

[0019] Figure 1 A schematic diagram of a system for implementing a method for retrieving user information according to an embodiment of the present invention is shown.

[0020] Figure 2 A flowchart of a method 200 for generating user-generated content according to an embodiment of the present disclosure is shown.

[0021] Figure 3 A flowchart of a method 300 for extracting a second feature of second content data based on a second content analysis tool according to an embodiment of the present disclosure is shown.

[0022] Figure 4 A flowchart of a method 400 for extracting a second feature of second content data based on a second content analysis tool according to an embodiment of the present disclosure is shown.

[0023] Figure 5 A schematic block diagram of an example electronic device that can be used to implement embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0024] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0025] As used herein, the term "including" and its variations mean open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "based at least in part on". The terms "an example embodiment" and "an embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0026] As described above, when editing content, users can manually enter topics and associate content. In this case, even if different users publish content for the same topic center, they may add different topic tags due to different users' expression habits. The content that can be associated with the topic is limited, which is very restrictive. Alternatively, new topics can be generated based on the topic tags manually entered by users through custom rules or machine learning. The templates required for this type of method rely on a large amount of prior knowledge, which is costly and difficult to migrate quickly and effectively.

[0027] In summary, the traditional method for generating tags for user-generated content has the following disadvantages: it relies on user editing or a large amount of prior knowledge, is costly, and is difficult to migrate quickly and effectively. Tags generated by machine learning cannot integrate multimodal features, resulting in inaccurate generated tags.

[0028] In order to at least partially solve one or more of the above problems and other potential problems, the example embodiments of the present disclosure propose a solution for generating tags for user-generated content. The method for generating tags for user-generated content proposed in the present invention can be based on multimodal understanding of UGC content (posts) and annotate relevant tags. The generation of these tags is not limited to keywords in the post text, but combines multimodal information such as text, images, and videos, thereby achieving a multi-angle understanding of UGC content, so that the accuracy and relevance of annotations are improved.

[0029] Figure 1 A schematic diagram of a system for implementing a method for generating tags for user-generated content according to an embodiment of the present invention is shown. Figure 1 As shown in FIG, the system includes a computing device 110, a user-generated content data management device 130, and a network 140. The computing device 110 and the user-generated content data management device 130 can perform data exchange through the network 140 (eg, the Internet).

[0030] The user-generated content data management device 130 may, for example, store one or more user-generated content data file data. The user-generated content data management device 130 may also send the stored user-generated content data to the computing device 110. The user-generated content data management device may, for example, be, but not limited to, an electronic computer, a network server, a storage calculator, etc. The user-generated content data management device 130 may, for example, collect user-generated content data file information from multiple sources, such as user-written texts (posts), images, videos, etc.

[0031] Regarding the computing device 110, it is used, for example, to obtain user-generated content data from the user-generated content data management device 130; and receive the user-generated content data to be pushed and push the data. By processing the user-generated content data to be processed, targeted tag generation for the user-generated content data can be achieved. The computing device 110 can receive data to be pushed from a user or the user-generated content data management device 130. By applying the method disclosed in the present disclosure, the computing device 110 can perform tag generation processing on the user-generated content data.

[0032] The computing device 110 may have one or more processing units, including dedicated processing units such as GPU, FPGA and ASIC, and general processing units such as CPU. In addition, one or more virtual machines may also be running on each computing device 110. In some embodiments, the computing device 110 and the user-generated content data management device 130 may be integrated together or may be separately arranged. In some embodiments, the computing device 110 includes, for example, the following modules.

[0033] The acquisition module 112 is configured to acquire content data corresponding to the user-generated content.

[0034] The classification module 114 is configured to classify the acquired content data into first content data and second content data.

[0035] The extraction module 116 is configured to extract a first feature of the first content data based on a first content analysis tool and to extract a second feature of the second content data based on a second content analysis tool.

[0036] The interaction module 118 is configured to interact the first feature and the second feature to obtain an interactive third feature.

[0037] The generating module 120 is configured to generate a tag of the user-generated content based on the first feature, the second feature, and the third feature according to the time correlation of the content data.

[0038] Figure 2 1 is a flow chart of a method 200 for generating user-generated content according to an embodiment of the present disclosure. The method 200 may be performed as follows: Figure 1 The computing device 110 shown may also be executed in Figure 5 The method 200 is executed at the electronic device 500 shown. It should be understood that the method 200 may further include additional blocks not shown and / or may omit the blocks shown, and the scope of the present disclosure is not limited in this respect.

[0039] In step 202 , the computing device 110 is configured to obtain content data corresponding to user-generated content.

[0040] In one embodiment, as described above, user-generated content may be various types of multimedia content that appear on social media, such as text, sound, image, image set (e.g., multiple images under the same post), and video. The computing device 110 may obtain the above multimedia content and extract content data therein for analysis below. In the method provided in the present disclosure, content data such as text, sound, image, image set (e.g., multiple images under the same post), and video may be processed to generate tags.

[0041] In step 204 , the computing device 110 is configured to classify the acquired content data into first content data and second content data.

[0042] In one embodiment, the computing device 110 may classify the content data acquired in step 202 into first content data and second content data. Specifically, the first content data may refer to text or sound recognized as text by audio recognition or image recognized as text by image recognition. That is, the first content data may correspond to text information. The second content data may refer to a pure image, a set of images, a video, etc. That is, the second content data may correspond to visual information.

[0043] The computing device may also determine the classified second content data as one of image data, an image data set including a plurality of images, and video data.

[0044] The above contents will be introduced in detail below.

[0045] In step 206 , the computing device 110 is configured to extract a first feature of the first content data based on a first content analysis tool and to extract a second feature of the second content data based on a second content analysis tool.

[0046] In one embodiment, the computing device 110 can use a first content analysis tool such as a neural network framework of Ernie to extract independent text features for the first content data, thereby serving as the first feature. At the same time, visually, the computing device 110 can use a second content analysis tool such as a neural network framework of Transformer to extract independent video features for the second content data, or use a second content analysis tool of a neural network framework of an attention mechanism to extract the quality and content of an image or an image set to filter image features to form independent image features, thereby using video features or image features as second features.

[0047] The above contents will be introduced in detail below.

[0048] In step 208 , the computing device 110 is configured to interact the first feature and the second feature to obtain an interacted third feature.

[0049] In one embodiment, the computing device 110 may connect the first feature for text and the second feature for vision obtained in step 206 to obtain a fused feature. The fusion may be a feature fusion method commonly used in the art. The present disclosure uses a convolutional neural network to learn the visual semantic features of an image, and uses a graph neural network to learn the semantic feature expression.

[0050] The computing device 110 may then input the fused features into a graph neural network (GNN), such as a recursive graph neural network, a convolutional graph neural network, or a graph autoencoder. Taking the Transformer graph neural network framework as an example, the fused features may be input into the encoder of the graph neural network to obtain a recursive vector or feature vector output by the encoder.

[0051] Optionally, the third feature of the interaction can be obtained by aggregating or averaging the obtained recursive vector, or the feature vector can be directly used as the third feature. Specifically, the first feature extracted by the graph neural network fusion and the second feature extracted by the convolutional neural network can be used. Specifically, the computing device 110 can represent the text content as a first feature (first feature vector), and then stack the first feature table with the visual second feature of the text content on the social network as the input of the encoder. The fusion vector is obtained through multiple layers of self-attention, such as a six-layer self-attention or co-attention network. The obtained fusion vector can be used as the third feature of the interaction. In this step, the obtained fusion vector can also be averaged or aggregated. Averaging can supplement the input of the decoder in the subsequent step, and can also better represent the time-weakly dependent data in the time correlation.

[0052] In step 210 , the computing device 110 is configured to generate a tag for the user-generated content based on the first feature, the second feature, and the third feature according to the temporal correlation of the content data.

[0053] In one embodiment, the computing device 110 can set a first weight, a second weight, and a third weight for the first feature, the second feature, and the interacted third feature, respectively. For example, the first weight for the first feature is set to 0.4, the second weight for the second feature is set to 0.3, and the third weight for the third feature is set to 0.3. In combination with the characteristics of UGC posts in social scenarios that contain strong dependence on time information and weak dependence on time information, the above features are organically combined as the output of the label generator. Generate labels for user-generated content using a multimodal fusion label generator such as a decoder. Note that the labels generated by the label generation method provided in the present disclosure can be one or more, that is, the method of the present disclosure can be used to generate a label set including multiple labels for a group of UGC content. The generated label set can use an algorithm such as beam search to select the most relevant or user-expected labels.

[0054] In one embodiment, the computing device 110 may determine whether the search content data is time-dependent data or time-dependent data based on the time correlation of the content data. If the video content changes rapidly over time, it is a video with strong time dependence; conversely, if the scene or content in the video does not change much, such as a guitar singing video or some slides in a social post, it is a video with weak time dependence.

[0055] In response to the content data being time information strongly dependent data, the second weight for vision is increased, for example, the second weight is increased from 0.3 to 0.5, and the first weight and the third weight are adjusted accordingly. In response to the content data being time information weakly dependent data, the first weight for text is increased, for example, the first weight is increased from 0.4 to 0.6, and the first weight and the third weight are adjusted accordingly.

[0056] By using the above technical means, we can integrate multimodal features for vision and text, and combine the original visual features and text features to generate the tags expected by users. This enables multimodal information to understand UGC from multiple angles and improves the accuracy of tags.

[0057] Figure 3 FIG. 3 is a flowchart of a method 300 for extracting a second feature of second content data based on a second content analysis tool according to an embodiment of the present disclosure. The method 300 may be performed as follows: Figure 1 The computing device 110 shown may also be executed in Figure 5 The method 400 is executed at the electronic device 500 shown. It should be understood that the method 400 may further include additional blocks not shown and / or may omit the blocks shown, and the scope of the present disclosure is not limited in this respect.

[0058] In step 302, the computing device 110 is configured to input the image data into a convolutional neural network in response to the second content data being image data. For example, the image data may be input into a convolutional neural network such as Resnet-inception to obtain image features.

[0059] In step 304, the computing device 110 is configured to extract image features based on neural network learning. The image features are extracted using a general pre-trained image convolutional neural network. The image features extracted in this step are often of high dimension (1536 dimensions).

[0060] In step 306, the computing device 110 is configured to perform dimensionality reduction processing on the extracted image features to obtain a second feature. The computing device 110 can reduce it to a lower dimension (512 dimensions) through two fully connected layers as the second visual feature to facilitate the feature fusion described above.

[0061] Figure 4 FIG. 4 is a flowchart of a method 400 for extracting a second feature of second content data based on a second content analysis tool according to an embodiment of the present disclosure. The method 400 may be performed as follows: Figure 1 The computing device 110 shown may also be executed in Figure 5 The method 400 is executed at the electronic device 500 shown. It should be understood that the method 400 may further include additional blocks not shown and / or may omit the blocks shown, and the scope of the present disclosure is not limited in this respect.

[0062] In step 402, the computing device 110 is configured to, in response to the second content data being an image data set including a plurality of images, input the image data set into a convolutional neural network in batches. For example, the image data may be input into a convolutional neural network such as Resnet-inception to obtain image features. When the second content data is a picture set of a plurality of pictures (e.g., 4 pictures), the image data set may be input into the convolutional neural network in batches.

[0063] In step 404, the computing device 110 is configured to extract multiple image features based on neural network learning. The image features are extracted using a general pre-trained image convolutional neural network. The image features extracted in this step are often of high dimension (1536 dimensions).

[0064] In step 406, the computing device 110 is configured to obtain image weights corresponding to the plurality of images. For an image dataset including a plurality of images, two attention blocks may be used to adjust the weights of the plurality of images in the image dataset. The attention blocks may adaptively aggregate feature vectors so as to form a single feature in the convex hull spanned by the two blocks. Different weights may be set according to the importance of each image, thereby increasing the weight of the image feature of the image.

[0065] In step 408, the computing device 110 is configured to perform dimensionality reduction processing on the extracted multiple image features and perform fusion based on the obtained image weights to obtain a second feature. The computing device 110 can reduce it to a lower dimension (512 dimensions) through two fully connected layers as the second feature for vision, so as to facilitate the feature fusion described above.

[0066] In one embodiment, in response to the second content data being video data, the computing device 110 may perform frame extraction processing on the video data according to the spatiotemporal correlation of the content data, thereby obtaining multiple frames of image data. For example, multiple frames (e.g., 5 frames) in a 10s video are extracted. The multiple frames of images are processed according to the method described above. For example, the obtained multiple frames of image data are input into a graph neural network such as Transformer; multiple image features are extracted based on neural network learning; and dimensionality reduction processing is performed on the extracted multiple image features and fusion is performed based on the spatiotemporal correlation of the content data, thereby obtaining the second feature.

[0067] Using the above technical means, we can extract video features based on Transformer when processing video modality, and extract picture set features in combination with the attention mechanism when processing picture set modality. Then, we can fuse the features of different modalities to obtain the output of the encoder, thereby generating multi-module labels for multi-modal UGC content in social scenarios.

[0068] Figure 5 Schematic block diagram of an example electronic device 500 that can be used to implement an embodiment of the present disclosure is shown. Figure 1The computing device 110 shown can be implemented by an electronic device 500. As shown, the electronic device 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 502 or computer program instructions loaded from a storage unit 508 to a random access memory (RAM) 503. In the random access memory 503, various programs and data required for the operation of the electronic device 500 can also be stored. The central processing unit 501, the read-only memory 502, and the random access memory 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0069] Multiple components in the electronic device 500 are connected to the input / output interface 505, including: an input unit 506, such as a keyboard, a mouse, a microphone, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0070] The various processes and processing described above, such as methods 200, 300, and 400, may be performed by the central processing unit 501. For example, in some embodiments, the methods 200, 300, and 400 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 500 via the read-only memory 502 and / or the communication unit 509. When the computer program is loaded into the random access memory 503 and executed by the central processing unit 501, one or more actions of the methods 200, 300, and 400 described above may be performed.

[0071] The present disclosure relates to methods, apparatuses, systems, electronic devices, computer-readable storage media and / or computer program products. The computer program products may include computer-readable program instructions for executing various aspects of the present disclosure.

[0072] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0073] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge computing devices. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing / processing device.

[0074] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0075] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0076] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0077] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0078] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0079] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for generating tags for user-generated content, comprising: Obtaining content data corresponding to user-generated content; classifying the acquired content data into first content data and second content data; extracting a first feature of the first content data based on a first content analysis tool and extracting a second feature of the second content data based on a second content analysis tool; Interacting the first feature with the second feature, thereby obtaining an interactive third feature; as well as According to the time correlation of the content data, a label of the user-generated content is generated based on the first feature, the second feature and the third feature, including setting a first weight, a second weight and a third weight respectively for the first feature, the second feature and the interacted third feature, adjusting the relationship between the first weight, the second weight and the third weight based on the time correlation of the content data, and generating a label of the user-generated content based on the adjusted first weight, second weight and third weight using a multimodal fusion label generator.

2. The method according to claim 1, wherein classifying the acquired content data into first content data and second content data comprises: The classified second content data is determined to be one of image data, an image data set including a plurality of images, and video data.

3. The method according to claim 2, wherein extracting the second feature of the second content data based on a second content analysis tool comprises: In response to the second content data being image data, inputting the image data into a convolutional neural network; Extract image features based on neural network learning; as well as A dimensionality reduction process is performed on the extracted image features to obtain a second feature.

4. The method according to claim 2, wherein extracting the second feature of the second content data based on a second content analysis tool comprises: In response to the second content data being an image dataset including a plurality of images, inputting the image dataset into a convolutional neural network in batches; Based on neural network learning, multiple image features are extracted; obtaining image weights corresponding to a plurality of images; as well as Dimensionality reduction processing is performed on the extracted multiple image features and fusion is performed based on the acquired image weights, thereby acquiring a second feature.

5. The method according to claim 2, wherein extracting the second feature of the second content data based on a second content analysis tool comprises: In response to the second content data being video data, performing frame extraction processing on the video data according to the time correlation of the content data, thereby acquiring multiple frames of image data; Input the acquired multiple frames of image data into a convolutional neural network; Based on neural network learning, multiple image features are extracted; as well as Dimensionality reduction processing is performed on the extracted multiple image features and fusion is performed to obtain the second feature.

6. The method according to claim 1, wherein interacting the first feature and the second feature to obtain an interactive third feature comprises: Connect the first feature and the second feature to obtain a fusion feature; Inputting the fused features into a graph neural network; as well as Get the feature vector output by the graph neural network as the third feature of the interaction.

7. The method according to claim 1, wherein adjusting the relationship between the first weight, the second weight, and the third weight based on the time correlation of the content data comprises: Based on the time correlation of the content data, determining whether the search content data is time information strongly dependent data or time information weakly dependent data; In response to the content data being time information-strongly dependent data, increasing the second weight; as well as In response to the content data being data weakly dependent on time information, the first weight is increased.

8. A computing device comprising: at least one processor; as well as a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

9. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the method of any one of claims 1 to 7.

Images