Label determination method, device, equipment, and storage medium

Abstract

The present application discloses a label determination method, apparatus, device and storage medium, which can be applied to cloud technology, artificial intelligence and vehicle-mounted scenarios. Through the technical solution provided in the embodiment of the present application, in the process of determining the video label, the first semantic feature of the target video and the second semantic feature of the description text are respectively obtained. The first semantic feature and the second semantic feature are fused based on the first fusion parameter and the second fusion parameter to obtain a fused semantic feature, and the video label can be determined subsequently based on the fused semantic feature. In the process of fusing the first semantic feature and the second semantic feature, the first fusion parameter and the second fusion parameter obtained by decomposing the target fusion parameter are used, which can reduce the number of parameters in the feature fusion process, and also reduce the amount of calculation for fusing the first semantic feature and the second semantic feature, thereby improving the efficiency of determining the video label.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a tag determination method, apparatus, device, and storage medium. Background Art

[0002] With the development of computer technology, more and more users are watching videos for relaxation and entertainment. To attract users to watch videos, video recommendation platforms will push videos that may be of interest to users. During the video push process, it is often based on the matching between video tags and user tags.

[0003] In related technologies, manual screening is often used to determine video tags. However, manual screening of video tags is slow, resulting in low efficiency in determining video tags. Summary of the Invention

[0004] The embodiments of the present application provide a tag determination method, apparatus, device, and storage medium, which can improve the efficiency of determining video tags. The technical solution is as follows:

[0005] In one aspect, a label determination method is provided, the method comprising:

[0006] Obtain a target video and a description text of the target video;

[0007] Performing feature extraction on the target video and the description text to obtain a first semantic feature of the target video and a second semantic feature of the description text;

[0008] Using a first fusion parameter and a second fusion parameter, the first semantic feature and the second semantic feature are fused to obtain a fused semantic feature of the target video, wherein the first fusion parameter and the second fusion parameter are parameters obtained by decomposing a target fusion parameter, and the target fusion parameter is a fusion parameter obtained by training based on a sample video, a sample description text of the sample video, and a label of the sample video;

[0009] Based on the fused semantic features, a label of the target video is determined.

[0010] In one aspect, a label determination device is provided, the device comprising:

[0011] An acquisition module, configured to acquire a target video and a description text of the target video;

[0012] A feature extraction module is used to extract features from the target video and the description text to obtain a first semantic feature of the target video and a second semantic feature of the description text;

[0013] a feature fusion module, configured to fuse the first semantic feature and the second semantic feature using a first fusion parameter and a second fusion parameter to obtain a fused semantic feature of the target video, wherein the first fusion parameter and the second fusion parameter are parameters obtained by decomposing a target fusion parameter, and the target fusion parameter is a fusion parameter obtained by training based on a sample video, a sample description text of the sample video, and a label of the sample video;

[0014] The label determination module is used to determine the label of the target video based on the fused semantic features.

[0015] In one possible embodiment, the feature extraction module is used to input the target video into an audio-visual feature extraction model, perform feature extraction on the target video through the audio-visual feature extraction model, and obtain the first semantic feature of the target video; input the description text into a text feature extraction model, perform feature extraction on the description text through the text feature extraction model, and obtain the second semantic feature of the description text.

[0016] In a possible implementation, the feature extraction module is configured to perform any of the following:

[0017] Performing convolution, residual connection, and full connection on at least one of the image and audio of the target video using the audio-visual feature extraction model to obtain the first semantic feature of the target video;

[0018] Through the audio-visual feature extraction model, at least one of the image and audio of the target video is encoded based on the attention mechanism to obtain the first semantic feature of the target video.

[0019] In a possible implementation, the feature extraction module is configured to perform convolution, residual connection, and full connection on the image of the target video using the audio-visual feature extraction model to obtain image semantic features of the target video;

[0020] Performing convolution, residual connection, and full connection on the audio of the target video through the audio-visual feature extraction model to obtain audio semantic features of the target video;

[0021] The image semantic feature and the audio semantic feature are fused to obtain the first semantic feature of the target video.

[0022] In a possible implementation, the feature extraction module is configured to encode the image of the target video based on an attention mechanism using the audio-visual feature extraction model to obtain an image attention feature of the target video;

[0023] Encoding the audio of the target video based on the attention mechanism through the audio-visual feature extraction model to obtain the audio attention feature of the target video;

[0024] The image attention feature and the audio attention feature are fused to obtain the first semantic feature of the target video.

[0025] In a possible implementation, the feature extraction module is configured to perform any of the following:

[0026] Performing sequence encoding on the description text by using the text feature extraction model to obtain the second semantic feature of the target video;

[0027] Encoding the description text based on the attention mechanism through the text feature extraction model to obtain the second semantic feature of the target video;

[0028] The description text is embedded and encoded using the text feature extraction model to obtain the second semantic feature of the target video.

[0029] In one possible embodiment, the feature fusion module is used to multiply the first fusion parameter by the first semantic feature to obtain a first fusion feature; multiply the second fusion parameter by the second semantic feature to obtain a second fusion feature; and multiply the first fusion feature by the second fusion feature to obtain a fused semantic feature of the target video.

[0030] In a possible implementation, the label determination module is used to perform full connection on the fused semantic features to obtain multiple prediction values ​​corresponding to multiple candidate labels; and determine the candidate label whose prediction value meets the target condition among the multiple candidate labels as the label of the target video.

[0031] In one possible embodiment, the label determination module is configured to sort the plurality of candidate labels in descending order of predicted value, and determine the first target number of candidate labels among the plurality of candidate labels after sorting as the labels of the target video. In one possible embodiment, the apparatus further includes:

[0032] A decomposition module is used to perform tensor decomposition on the target fusion parameter to obtain the first fusion parameter and the second fusion parameter, and the product of the transposition of the first fusion parameter and the second fusion parameter is the target fusion parameter.

[0033] In a possible implementation, the device further includes:

[0034] A target fusion parameter acquisition module is used to obtain a sample video and a sample description text of the sample video; perform feature extraction on the sample video and the sample description text to obtain a first sample semantic feature of the sample video and a second sample semantic feature of the sample description text; use the target fusion parameter to fuse the first sample semantic feature and the second sample semantic feature to obtain a sample fusion semantic feature of the sample video; determine a predicted label of the sample video based on the sample fusion semantic feature; and update the target fusion parameter based on the difference information between the predicted label of the sample video and the labeled label of the sample video.

[0035] On the one hand, a computer device is provided, which includes one or more processors and one or more memories, wherein at least one computer program is stored in the one or more memories, and the computer program is loaded and executed by the one or more processors to implement the label determination method.

[0036] In one aspect, a computer-readable storage medium is provided, wherein at least one computer program is stored in the computer-readable storage medium, and the computer program is loaded and executed by a processor to implement the label determination method.

[0037] On the one hand, a computer program product or computer program is provided, which includes a program code, which is stored in a computer-readable storage medium. A processor of a computer device reads the program code from the computer-readable storage medium, and the processor executes the program code, so that the computer device performs the above-mentioned label determination method.

[0038] Through the technical solution provided by the embodiment of the present application, in the process of determining the video label, the first semantic feature of the target video and the second semantic feature of the descriptive text are respectively obtained. The first semantic feature and the second semantic feature are fused based on the first fusion parameter and the second fusion parameter to obtain a fused semantic feature, and the video label can be determined subsequently based on the fused semantic feature. In the process of fusing the first semantic feature and the second semantic feature, the first fusion parameter and the second fusion parameter obtained by decomposing the target fusion parameter are used, which can reduce the number of parameters in the feature fusion process, and also reduce the amount of calculation for fusing the first semantic feature and the second semantic feature, thereby improving the efficiency of determining the video label. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0040] Figure 1 is a schematic diagram of an implementation environment of a tag determination method provided in an embodiment of the present application;

[0041] Figure 2 This is a flowchart of a label determination method provided by an embodiment of the present application;

[0042] Figure 3 This is a flowchart of a label determination method provided by an embodiment of the present application;

[0043] Figure 4 This is a flowchart of a label determination method provided by an embodiment of the present application;

[0044] Figure 5 This is a structural diagram of a label determination device provided in an embodiment of the present application;

[0045] Figure 6 This is a schematic diagram of the structure of a terminal provided in an embodiment of the present application;

[0046] Figure 7 This is a structural diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0048] In this application, the terms "first", "second", etc. are used to distinguish identical or similar items with substantially the same effects and functions. It should be understood that there is no logical or temporal dependency between "first", "second", and "nth", nor is there any limitation on the quantity and execution order.

[0049] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also involves studying the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0050] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0051] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge sub-models to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning through demonstration.

[0052] Semantic features: Features used to represent the semantic meaning of a text. Different texts can correspond to the same semantic features. For example, the text "How's the weather today?" and the text "How is the weather today?" can both correspond to the same semantic feature. Computers can map characters in a text into character vectors and, based on the relationships between characters, combine and operate on these character vectors to obtain the semantic features of the text. For example, computers can use bidirectional encoder representations from transformers (BERT).

[0053] Normalization: Mapping sequences of numbers with different value ranges to the interval (0, 1) facilitates data processing. In some cases, the normalized values ​​can be directly implemented as probabilities.

[0054] Embedded Coding: Embedded coding mathematically represents a correspondence, that is, mapping data in X space to Y space through a function F, where the function F is an injective function. The mapping result is structural preservation. The injective function indicates that the data after mapping is uniquely corresponding to the data before mapping. Structural preservation means that the size relationship of the data before mapping is the same as the size relationship of the data after mapping. For example, before mapping, there are data X1 and X2, and after mapping, the data corresponding to X1 is Y1 and the data corresponding to X2 is Y2. If the data before mapping X1 is greater than X2, then the data after mapping Y1 is greater than Y2. For words, this means mapping the words to another space to facilitate subsequent machine learning and processing.

[0055] Attention weight: This indicates the importance of a piece of data during training or prediction. Importance indicates the impact of the input data on the output data. Highly important data has a higher attention weight, while lowly important data has a lower attention weight. Data importance varies in different scenarios, and the process of training a model's attention weights is also the process of determining data importance.

[0056] Figure 1 This is a schematic diagram of the implementation environment of a tag determination method provided in an embodiment of the present application, see Figure 1 , the implementation environment may include a terminal 110 and a server 140.

[0057] The terminal 110 is connected to the server 140 via a wireless network or a wired network. Optionally, the terminal 110 includes but is not limited to a mobile phone, a computer, an intelligent voice interaction device, a smart home appliance, a vehicle terminal, an aircraft, etc. The terminal 110 installs and runs an application that supports tag determination.

[0058] Server 140 is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Server 140 provides background services for applications running on terminal 110.

[0059] Those skilled in the art will appreciate that the number of terminals may be greater or less. For example, there may be only one terminal, or there may be dozens, hundreds, or even more terminals, in which case the implementation environment may also include other terminals. The embodiments of this application do not limit the number or device types of terminals.

[0060] After introducing the implementation environment of the embodiment of the present application, the application scenario of the embodiment of the present application will be introduced in combination with the relevant description of the above implementation environment. In the following description, the terminal is the terminal 110 in the above implementation environment, and the server is the server 140 in the above implementation environment.

[0061] The tag determination method provided in the embodiment of the present application can be applied in the recommendation scenario of multimedia content, the retrieval scenario of multimedia content, and the classification scenario of multimedia content, wherein the multimedia content includes videos, music, and picture collections.

[0062] Taking the tag determination method provided in the embodiment of the present application as an example and applied to the scenario of video recommendation, in the video recommendation process, it is often necessary to add tags to the video and implement the video recommendation based on the video tag and user tag. After adopting the tag determination method provided in the embodiment of the present application, the server can perform feature extraction on the target video to obtain the first semantic feature of the target video, and the first semantic feature can reflect the characteristics of the target video from the audio-visual perspective. The server can perform feature extraction on the description text of the target video to obtain the second semantic feature of the description text, wherein the description text is the title of the target video, the introduction of the target video, or the subtitles in the target video, etc. The server adopts the first fusion parameter and the second fusion parameter to fuse the first semantic feature and the second semantic feature to obtain the fused semantic feature of the target video, wherein the first fusion parameter and the second fusion parameter are obtained by decomposing the target fusion parameter, and the parameter amount of the first fusion parameter and the second fusion parameter are both smaller than the target fusion parameter. Compared with adopting the target fusion parameter to fuse the first semantic feature and the second semantic feature, the computational complexity of adopting the first fusion parameter and the second fusion parameter to fuse the first semantic feature and the second semantic feature is smaller, and the fusion efficiency is higher. The server can determine the target video's tag based on the fused semantic features of the target video. The server can then recommend the target video based on the tag. This process of determining the target video's tag requires no manual effort; the server can automatically add tags to the target video, resulting in a highly efficient tagging process.

[0063] Taking the label determination method provided in the embodiment of the present application as an example and applied to the scene of video classification, in the process of video classification, it is often necessary to add labels to the video and implement the classification of the video based on the video labels. The label determination method provided in the embodiment of the present application can determine the label of the target video. The subsequent server can classify the target video based on the label of the target video. For example, the server can cluster the determined labels and classify the video according to the clustering results. In the process of determining the label of the target video, no manual operation is required. The server can automatically add labels to the target video, and the efficiency of label addition is high.

[0064] Taking the application of the tag determination method provided in the embodiment of the present application in the scenario of video retrieval as an example, before video retrieval, it is often necessary to add tags to the video to be retrieved, and then the video can be retrieved based on the video tags.

[0065] It should be noted that the above description is based on the application of the technical solutions provided in the embodiments of this application in the scenarios of video recommendation, classification, and retrieval. For the application scenarios based on other multimedia content in the above description, the implementation process belongs to the same inventive concept as the above description and is not repeated here. In the following description, the label of the target video is used as an example.

[0066] In addition, in the above description of the application scenario of the embodiment of the present application, the server is used as the execution subject as an example. In other possible implementations, the technical solution provided by the present application can also be executed by the terminal as the execution subject, and the embodiment of the present application does not limit the type of execution subject.

[0067] After introducing the implementation environment and application scenarios of the embodiment of the present application, the tag determination method provided by the embodiment of the present application will be described below in combination with the above implementation environment and application scenarios, taking the execution subject as a server as an example, see Figure 2 , methods include:

[0068] 201. The server obtains a target video and a description text of the target video.

[0069] The target video is the video to be tagged, and the description text of the target video is the title, introduction or subtitles of the target video.

[0070] 202. The server performs feature extraction on the target video and the description text to obtain a first semantic feature of the target video and a second semantic feature of the description text.

[0071] The first semantic feature is a feature extracted from the target video, which can reflect the audiovisual characteristics of the target video. The second semantic feature is a feature extracted from the description text of the target video, which can reflect the characteristics of the target video from the perspective of text.

[0072] 203. The server adopts a first fusion parameter and a second fusion parameter to fuse the first semantic feature and the second semantic feature to obtain a fused semantic feature of the target video. The first fusion parameter and the second fusion parameter are parameters obtained by decomposing a target fusion parameter. The target fusion parameter is a fusion parameter obtained by training based on a sample video, a sample description text of the sample video, and a label of the sample video.

[0073] The process of decomposing the target fusion parameter is also the process of reducing the dimension of the target fusion parameter. The dimensions of the obtained first and second fusion parameters are both lower than the target fusion parameter. Using the first and second fusion parameters for feature fusion is less computationally intensive and faster. The target fusion parameter is a fusion parameter obtained through training and can fuse semantic features. However, the target fusion parameter has a higher dimension, so the computational complexity of using the target fusion parameter for semantic feature fusion is higher.

[0074] 204. The server determines a label of the target video based on the fused semantic features.

[0075] Among them, since the fused semantic feature integrates the audio-visual features and text features of the target video, the accuracy of determining the label of the target video based on the fused semantic feature is relatively high.

[0076] Through the technical solution provided by the embodiment of the present application, in the process of determining the video label, the first semantic feature of the target video and the second semantic feature of the descriptive text are respectively obtained. The first semantic feature and the second semantic feature are fused based on the first fusion parameter and the second fusion parameter to obtain a fused semantic feature, and the video label can be determined subsequently based on the fused semantic feature. In the process of fusing the first semantic feature and the second semantic feature, the first fusion parameter and the second fusion parameter obtained by decomposing the target fusion parameter are used, which can reduce the number of parameters in the feature fusion process, and also reduce the amount of calculation for fusing the first semantic feature and the second semantic feature, thereby improving the efficiency of determining the video label.

[0077] It should be noted that the above steps 201-204 are a brief introduction to the technical solution provided in the embodiment of the present application. The tag determination method provided in the embodiment of the present application will be described in more detail below with reference to some examples. Figure 3 , taking the execution subject as a server as an example, the method includes:

[0078] 301. The server obtains a target video and a description text of the target video.

[0079] The target video is the video whose tag is to be determined. In some embodiments, the target video is a short video, such as a food show short video, a gaming short video, a knowledge short video, a travel short video, or a lifestyle short video. In some embodiments, the target video is a long video, such as a movie or TV series. The description text of the target video can be the title, introduction, or subtitles of the target video, and the description text can reflect the content of the target video from a textual perspective.

[0080] In one possible implementation, the server obtains a target video from a video collection to be tagged, where the video collection is to be tagged, such as a video collection just uploaded by a video creator or a video collection to be tagged. The server obtains at least one of the target video's title, introduction, and subtitles as the target video's description text.

[0081] In some embodiments, when there are multiple target videos, the server can retrieve multiple target videos from the video collection. The server obtains multiple description texts for the multiple target videos. The number of target videos retrieved by the server from the video collection is proportional to the server's computing power. That is, the greater the server's computing power, the more target videos the server can retrieve at the same time. After obtaining the multiple target videos and the description texts for the multiple target videos, the server can simultaneously process the description texts for the multiple target videos.

[0082] In one possible embodiment, a terminal displays a video upload page, which includes a video selection area for selecting a video to be uploaded to a server and a description text input area for inputting a description text of the video to be uploaded. In response to an operation on the video upload page, the terminal transmits the target video selected in the video selection area and the description text in the description text input area to the server, and the server obtains the target video and the description text of the target video.

[0083] 302. The server inputs the target video into an audio-visual feature extraction model, performs feature extraction on the target video through the audio-visual feature extraction model, and obtains the first semantic feature of the target video.

[0084] Among them, the audio-visual feature extraction model is used to extract features of the visual information and auditory information of the target video to obtain the first semantic feature of the target video, wherein the visual information of the target video is also the relevant information of the video frame of the target video, and the auditory information of the target video is also the relevant information of the audio frame of the target video.

[0085] In one possible implementation, the server inputs the target video into an audio-visual feature extraction model. The audio-visual feature extraction model performs convolution, residual connection, and full connection on at least one of the image and audio of the target video to obtain the first semantic feature of the target video. In this case, the audio-visual feature extraction model is also called a residual neural network (ResNet).

[0086] For example, the server inputs the image of the target video into the audio-visual feature extraction model, and performs convolution, residual connection, and full connection on the image of the target video through the audio-visual feature extraction model to obtain the image semantic features of the target video. The server inputs the audio of the target video into the audio-visual feature extraction model, and performs convolution, residual connection, and full connection on the audio of the target video through the audio-visual feature extraction model to obtain the audio semantic features of the target video. The server fuses the image semantic features and the audio semantic features to obtain the first semantic features of the target video. The image of the target video is the video frame of the target video, and the audio is the audio frame of the target video.

[0087] In some embodiments, the audiovisual feature extraction model includes a visual feature extraction sub-model and an auditory feature extraction sub-model. In the above example, the server inputs the target video image into the visual feature extraction sub-model and convolves the target video image through the convolutional layer of the visual feature extraction sub-model, i.e., sliding the convolution kernel over the image, to obtain a first feature map of the image. The server superimposes the image with the first feature map through the visual feature extraction sub-model, i.e., performing a residual connection on the first feature map, to obtain a second feature map of the image. The server fully connects the second feature map through the fully connected layer of the visual feature extraction sub-model, i.e., using the weight matrix corresponding to the fully connected layer, to obtain the image semantic features of the target video. The server performs a time-frequency transformation on the audio of the target video to obtain the frequency domain information of the target video. The server inputs the frequency domain information of the target video into the auditory feature extraction sub-model and convolves the frequency domain information of the target video through the convolutional layer of the auditory feature extraction sub-model, i.e., sliding the convolution kernel over the frequency domain information, to obtain a third feature map of the frequency domain information. The server uses the auditory feature extraction sub-model to superimpose the frequency domain information with the third feature map, performing a residual connection on the third feature map, to obtain a fourth feature map of the frequency domain information. The server uses the fully connected layer of the auditory feature extraction sub-model, using the weight matrix corresponding to the fully connected layer, to fully connect the fourth feature map to obtain the audio semantic features of the target video. The server then performs a weighted summation of the image semantic features and audio semantic features of the target video to obtain the first semantic features of the target video.

[0088] For example, the server inputs the image matrix corresponding to the image of the target video into the visual feature extraction sub-model. The image matrix includes three sub-matrices, and the three sub-matrices correspond to the three color channels (RGB) of the image respectively. The server performs convolution processing on the three sub-matrices through the visual feature extraction sub-model, that is, three convolution kernels are respectively used to slide on the three sub-matrices to obtain the first feature maps corresponding to the three sub-matrices. In the process of sliding the convolution kernel on the sub-matrix, the covered area will be convolved and mapped to eigenvalues. Multiple eigenvalues ​​constitute the first feature map. The server performs residual connection on the first feature maps corresponding to the three sub-matrices through the residual layer of the visual feature extraction sub-model to obtain the second feature maps corresponding to the three sub-matrices. The residual connection is to superimpose the first feature map with the corresponding sub-matrix to ensure that the information in the sub-matrix is ​​not lost during the feature extraction process. The server fully connects the second feature maps corresponding to the three sub-matrices through the fully connected layer of the visual feature extraction sub-model. That is, the server uses the corresponding weight matrix of the fully connected layer to multiply the corresponding second feature map, and maps the second feature maps corresponding to the three sub-matrices to the semantic features corresponding to the three sub-matrices. The server uses the visual feature extraction sub-model to perform weighted summation of the semantic features corresponding to the three sub-matrices to obtain the semantic features of the image, that is, the image semantic features of the target video. The server frames and windows the audio of the target video to obtain multiple audio frames of the target video. The server performs Fourier transform on the multiple audio frames to obtain a spectrogram of the target video. The server inputs the spectrogram of the target video into the auditory feature extraction sub-model and convolves the spectrogram of the target video through the convolution layer of the auditory feature extraction sub-model. That is, the server slides the convolution kernel on the spectrogram to obtain the third feature map of the spectrogram. The server uses the auditory feature extraction sub-model to superimpose the spectrogram and the third feature map, performing a residual connection on the third feature map, to obtain a fourth feature map of the spectrogram. The server uses the fully connected layer of the auditory feature extraction sub-model, using the weight matrix corresponding to the fully connected layer, to fully connect the fourth feature map to obtain the audio semantic features of the target video. The server then performs a weighted summation of the image semantic features and audio semantic features of the target video to obtain the first semantic features of the target video.

[0089] It should be noted that, in the above description, the example in which the server processes an image of the target video through the visual feature extraction sub-model is used. In this case, the image is the cover image of the target video. Since the cover image can often represent the most important content of the target video, the image semantic features obtained by extracting features from the cover image can also reflect the visual features of the target video. In other embodiments, the server can also input multiple video frames of the target video into the visual feature extraction sub-model, and perform feature extraction on the multiple video frames respectively through the visual feature extraction sub-model to obtain video frame features of each video frame. The server fuses the video frame features of the multiple video frames to obtain the image semantic features of the target video. In this way, the obtained image semantic features fuse the information in the multiple video frames of the target video, and the image semantic features can more completely represent the visual features of the target video.

[0090] In addition, in the above description, the example in which the server extracts features from both the image and audio of the target video through an audio-visual feature extraction model is used for explanation. In other possible implementations, the server can also perform feature extraction on one of the image and audio of the target video through an audio-visual feature extraction model to obtain the first semantic feature of the target video. The implementation process belongs to the same inventive concept as the above description and will not be repeated here.

[0091] In one possible implementation, the server inputs the target video into an audio-visual feature extraction model, and through the audio-visual feature extraction model, encodes at least one of the image and audio of the target video based on an attention mechanism to obtain the first semantic feature of the target video.

[0092] For example, the server uses the audio-visual feature extraction model to encode the image of the target video based on the attention mechanism to obtain the image attention feature of the target video. The server uses the audio-visual feature extraction model to encode the audio of the target video based on the attention mechanism to obtain the audio attention feature of the target video. The server fuses the image attention feature and the audio attention feature to obtain the first semantic feature of the target video. In this way, the audio-visual feature extraction model is also an attention encoding model.

[0093] In some embodiments, the audiovisual feature extraction model includes a visual feature extraction sub-model and an auditory feature extraction sub-model. In the above example, the server inputs multiple images of a target video into the visual feature extraction sub-model, and performs embedding encoding on the multiple images through the visual feature extraction sub-model to obtain image codes and position codes for the multiple images, wherein the multiple images are multiple video frames of the target video, and the position codes are used to indicate the positions of the corresponding images in the target video. The server, through the visual feature extraction sub-model, obtains attention weights between the multiple images based on the image codes and position codes. The server, through the visual feature extraction sub-model, fuses the image codes of the multiple images based on the attention weights between the multiple images to obtain image attention features for the target video. The server performs time-frequency transformation on the audio of the target video to obtain multiple frequency domain information of the target video, wherein the multiple frequency domain information corresponds to multiple audio frames of the audio of the target video. The server inputs the multiple frequency domain information of the target video into the auditory feature extraction sub-model, and performs embedding encoding on the multiple frequency domain information through the auditory feature extraction sub-model to obtain frequency domain information codes and position codes for the multiple frequency domain information. The server uses the auditory feature extraction sub-model to obtain attention weights between the multiple frequency domain information based on the frequency domain information encodings and position encodings of the multiple frequency domain information. The server uses the auditory feature extraction sub-model to fuse the frequency domain information encodings of the multiple frequency domain information based on the attention weights between the multiple frequency domain information to obtain an audio attention feature for the target video. The server performs a weighted summation of the image attention feature and the audio attention feature of the target video to obtain a first semantic feature of the target video.

[0094] For example, the server inputs multiple images of a target video into a visual feature extraction sub-model. The visual feature extraction sub-model then performs image embedding coding and position embedding coding on the multiple images, obtaining image and position codes for the multiple images. The server then fuses the image and position codes of each image through the visual feature extraction sub-model to obtain an attention code for each image. This attention code also incorporates the image information and position information of the image. The server then obtains the query matrix, key matrix, and value matrix for each image through the visual feature extraction sub-model. Specifically, the server multiplies the attention code of each image using three linear transformation matrices, obtained during training of the visual feature extraction sub-model, to obtain the query matrix, key matrix, and value matrix for each image. These three linear transformation matrices are the same as those obtained during training of the visual feature extraction sub-model. The server then determines the attention weights between the multiple images through the visual feature extraction sub-model based on the query and key matrices of each image. Based on the attention weights between the multiple images, the server fuses the value matrices of the multiple images through the visual feature extraction sub-model to obtain the image attention features of the target video. The server frames and windows the audio of the target video to obtain multiple audio frames of the target video. The server performs Fourier transform on the multiple audio frames to obtain multiple spectrograms corresponding to the multiple audio frames. The server inputs the multiple spectrograms of the target video into the auditory feature extraction sub-model. The auditory feature extraction sub-model performs spectrogram embedding coding and position embedding coding on the multiple spectrograms to obtain spectrogram codes and position codes for the multiple spectrograms. The server, through the auditory feature extraction sub-model, fuses the spectrogram codes and position codes of each spectrogram to obtain an attention code for each spectrogram. The attention code also fuses the spectrogram information and position information of the spectrogram. The server, through the auditory feature extraction sub-model, obtains the query matrix, key matrix, and value matrix for each spectrogram. Specifically, the server multiplies the attention code of each spectrogram with three linear transformation matrices respectively to obtain the query matrix, key matrix, and value matrix for each spectrogram. The three linear transformation matrices are the linear transformation matrices obtained when training the auditory feature extraction sub-model. The server uses the auditory feature extraction sub-model to determine the attention weights between the multiple spectrograms based on the query matrix and key matrix of each spectrogram. The server uses the auditory feature extraction sub-model to fuse the value matrices of the multiple spectrograms based on the attention weights between the multiple spectrograms to obtain the audio attention feature of the target video.

[0095] It should be noted that, in the above description, the example in which the server extracts features from both the image and audio of the target video through an audio-visual feature extraction model is used for explanation. In other possible implementations, the server can also perform feature extraction on one of the image and audio of the target video through an audio-visual feature extraction model to obtain the first semantic feature of the target video. The implementation process belongs to the same inventive concept as the above description and will not be repeated here.

[0096] In one possible implementation, the server inputs the target video into an audio-visual feature extraction model. The audio-visual feature extraction model then performs sequence encoding on at least one of the image and audio of the target video to obtain the first semantic feature of the target video. In this case, the audio-visual feature extraction model is a sequence encoding model, for example, a long short-term memory (LSTM) network.

[0097] For example, the server inputs multiple images of the target video into the audio-visual feature extraction model, and uses the audio-visual feature extraction model to perform sequence encoding on the multiple images to obtain the image feature sequence of the target video. The server uses the audio-visual feature extraction model to perform full connection on the image feature sequence to obtain the image sequence feature of the target video. The server inputs the audio of the target video into the audio-visual feature extraction model, and uses the audio-visual feature extraction model to perform sequence encoding on the audio to obtain the audio feature sequence of the target video. The server uses the audio-visual feature extraction model to perform full connection on the audio feature sequence to obtain the audio sequence feature of the target video. The server fuses the image sequence feature and the audio sequence feature to obtain the first semantic feature of the target video.

[0098] In some embodiments, the audiovisual feature extraction model includes a visual feature extraction sub-model and an auditory feature extraction sub-model. In the above example, the server performs embedded encoding on multiple images of the target video to obtain multiple image codes corresponding to the multiple images. The server inputs the multiple image codes into the visual feature extraction sub-model, which then performs sequence encoding in the order of the multiple image codes to obtain image feature sequences corresponding to the multiple image codes. The server fully connects the image feature sequence through the fully connected layer of the visual feature extraction sub-model to obtain image sequence features of the target video. The server performs time-frequency transformation on the audio of the target video to obtain multiple frequency domain information of the target video. The server inputs the multiple frequency domain information of the target video into the auditory feature extraction sub-model, which then performs sequence encoding in the order of the multiple frequency domain information to obtain audio feature sequences corresponding to the multiple frequency domain information. The server fully connects the audio feature sequence through the fully connected layer of the auditory feature extraction sub-model to obtain audio sequence features of the target video. The server performs weighted summation on the image sequence feature and the audio sequence feature to obtain the first semantic feature of the target video.

[0099] 303. The server inputs the description text into a text feature extraction model, performs feature extraction on the description text through the text feature extraction model, and obtains the second semantic feature of the description text.

[0100] In one possible implementation, the server inputs the description text into a text feature extraction model, and the text feature extraction model sequentially encodes the description text to obtain the second semantic feature of the target video. In this case, the text feature extraction model is a long short-term memory network.

[0101] For example, the server inputs the multiple characters of the description text into a text feature extraction model, and uses the text feature extraction model to perform sequence encoding on the multiple characters of the description text to obtain the second semantic feature of the target video.

[0102] For example, the server performs embedded encoding on multiple characters of the description text to obtain multiple character codes corresponding to the multiple characters. The server inputs the multiple character codes into the text feature extraction model, and the text feature extraction model performs sequence encoding according to the order of the multiple character codes to obtain a character feature sequence corresponding to the multiple character codes. The server performs a full connection on the character feature sequence through the fully connected layer of the text feature extraction model to obtain the second semantic feature of the description text.

[0103] In a possible implementation, the server encodes the description text based on the attention mechanism through the text feature extraction model to obtain the second semantic feature of the target video.

[0104] For example, the server inputs multiple characters of the description text into a text feature extraction model, and embeds and encodes the multiple characters through the text feature extraction model to obtain character codes and position codes of the multiple characters, wherein the position code is used to indicate the position of the corresponding character in the description text. The server obtains the attention weights between the multiple characters based on the character codes and position codes of the multiple characters through the text feature extraction model. The server fuses the character codes of the multiple characters based on the attention weights between the multiple characters through the text feature extraction model to obtain the second semantic feature of the description text.

[0105] For example, the server inputs multiple characters describing a text into a text feature extraction model. The text feature extraction model then performs character embedding coding and position embedding coding on the multiple characters to obtain character codes and position codes for the multiple characters. The server then fuses the character codes and position codes of each character through the text feature extraction model to obtain an attention code for each character. This attention code also fuses the character information and position information of the character. The server then obtains a query matrix, a key matrix, and a value matrix for each character through the text feature extraction model. Specifically, the server multiplies the attention code of each character using three linear transformation matrices, respectively, to obtain a query matrix, a key matrix, and a value matrix for each character. These three linear transformation matrices are the linear transformation matrices obtained when training the text feature extraction model. The server then determines attention weights between the multiple characters through the text feature extraction model based on the query matrix and key matrix of each character. Based on the attention weights between the multiple characters, the server then fuses the value matrices of the multiple characters through the text feature extraction model to obtain a second semantic feature describing the text.

[0106] In a possible implementation, the server performs embedding encoding on the description text through the text feature extraction model to obtain the second semantic feature of the target video.

[0107] 304. The server uses a first fusion parameter and a second fusion parameter to fuse the first semantic feature and the second semantic feature to obtain a fused semantic feature of the target video. The first fusion parameter and the second fusion parameter are parameters obtained by decomposing a target fusion parameter. The target fusion parameter is a fusion parameter obtained by training based on a sample video, a sample description text of the sample video, and a label of the sample video.

[0108] In one possible embodiment, the server multiplies the first fusion parameter by the first semantic feature to obtain a first fusion feature. The server multiplies the second fusion parameter by the second semantic feature to obtain a second fusion feature. The server multiplies the first fusion feature and the second fusion feature to obtain a fusion semantic feature of the target video. In some embodiments, this embodiment is also referred to as a second-order interaction process based on bilinear multiplication of the first semantic feature and the second semantic feature. In some embodiments, step 304 is performed by a feature fusion model.

[0109] For example, the server uses a matrix to represent the first fusion parameter and the second fusion parameter, which are respectively recorded as the first fusion matrix and the second fusion matrix; and uses a vector to represent the first semantic feature and the second semantic feature, which are respectively recorded as the first semantic feature vector and the second semantic feature vector. The server multiplies the first fusion matrix by the first semantic feature vector to obtain a first fusion feature matrix. The server multiplies the second fusion matrix by the second semantic feature vector to obtain a second fusion feature matrix. The server multiplies the first fusion feature matrix by the transpose of the second fusion feature matrix to obtain a fusion semantic feature matrix of the target video. For example, the server obtains the fusion semantic feature matrix of the target video through the following formula (1).

[0110] logit i =(U i *f img )*(V i *f text ) T +b(1)

[0111] Among them, logit i is the fusion semantic feature matrix, U i is the first fusion parameter, f img is the first semantic feature vector, V i is the second fusion parameter, f text is the second semantic feature vector, i is the parameter number, and b is the bias parameter.

[0112] The following describes a method for the server to decompose the target fusion parameter to obtain the first fusion parameter and the second fusion parameter.

[0113] In a possible implementation, the server performs tensor decomposition on the target fusion parameter to obtain the first fusion parameter and the second fusion parameter, and the product of the transpose of the first fusion parameter and the second fusion parameter is the target fusion parameter.

[0114] The tensor decomposition of the target fusion parameter decomposes the high-dimensional target fusion parameter into two low-dimensional first and second fusion parameters. In other words, the sum of the first and second fusion parameters is less than the target fusion parameter. Compared to using the target fusion parameter to fuse the first and second semantic features, using the lower-dimensional first and second fusion parameters to fuse the first and second semantic features requires less computation and is faster.

[0115] For example, the server performs tensor decomposition on the target fusion parameter using the following formula (2) to obtain the first fusion parameter and the second fusion parameter.

[0116] W i =U i *V i T (2)

[0117] Among them, W i is the target fusion parameter numbered i, W∈R n*m , U i ∈R n*k , V i ∈R m*k , k<<min(m,n). That is, in the embodiment of the present application, two low-rank matrices U and V are used to approximate the high-dimensional matrix W.

[0118] After introducing the process of the server performing tensor decomposition on the target fusion parameters to obtain the first fusion parameters and the second fusion parameters, the method for the server to obtain the target fusion parameters is described below.

[0119] In one possible embodiment, the server obtains a sample video and a sample description text of the sample video. The server performs feature extraction on the sample video and the sample description text to obtain a first sample semantic feature of the sample video and a second sample semantic feature of the sample description text. The server fuses the first sample semantic feature with the second sample semantic feature using the target fusion parameter to obtain a sample fused semantic feature of the sample video. The server determines a predicted label of the sample video based on the sample fused semantic feature. The server updates the target fusion parameter based on the difference information between the predicted label of the sample video and the annotated label of the sample video.

[0120] For example, the server obtains a sample video and a sample description text of the sample video from a sample database. The server inputs the sample video into a visual feature extraction model, performs feature extraction on the sample video through the visual feature extraction model, and obtains a first sample semantic feature of the sample video. The server inputs the sample description text into a text feature extraction model, performs feature extraction on the sample description text through the text feature extraction model, and obtains a second sample semantic feature of the sample description text. The server uses the target fusion parameter to fuse the first sample semantic feature and the second sample semantic feature to obtain a sample fusion semantic feature of the sample video. Based on the sample fusion semantic feature, the server determines the predicted value corresponding to the predicted label of the sample video. The server brings the predicted value corresponding to the predicted label of the sample video and the labeled value of the corresponding labeled label into the target loss function, and updates the target fusion parameter based on the difference between the predicted value and the labeled value through the target loss function.

[0121] For example, the server constructs a target loss function based on the difference information between the predicted label of the sample video and the annotated label of the sample video using the following formula (3).

[0122]

[0123] Among them, loss is the target loss function, x k is the predicted value of the label numbered k, y k is the label value of the label numbered k, n is the number of labels, and ||||2 is the two-norm.

[0124] It should be noted that when determining the label, it can be based on the position of the label, that is, the label value corresponding to the label closer to the front is higher. For example, if the label value of the first label of the sample video is 1, the label value of the second label of the sample video can be 0.8. The label value decreases by 0.2 for each digit lower, and so on.

[0125] 305. The server performs full connection on the fused semantic features to obtain multiple prediction values ​​corresponding to multiple candidate labels.

[0126] Among them, multiple candidate tags are tags set by technicians according to actual conditions, or are tags collected by the server from the Internet, which is not limited in the embodiments of the present application. The multiple candidate tags are stored on the server, and the technicians can expand and delete the multiple candidate tags to make the multiple candidate tags more suitable for the video.

[0127] In a possible implementation, the server multiplies the fused semantic feature by a fully connected matrix to map the fused semantic feature into a reference prediction value sequence, which includes multiple reference prediction values, and the multiple reference prediction values ​​correspond one-to-one to the multiple candidate labels. The server normalizes the reference prediction value sequence to obtain a prediction value sequence, which includes multiple prediction values, and the multiple prediction values ​​correspond one-to-one to the multiple candidate labels. Among them, the process of normalizing the reference prediction sequence by the server is that the server uses a normalization function to normalize the multiple reference prediction values ​​in the reference prediction sequence to obtain multiple prediction values. In some embodiments, the normalization function is a Softmax (soft maximization) function or a Sigmoid (S-type growth) function, which is not limited in the embodiments of the present application. In some embodiments, the prediction value obtained by the above implementation can also be regarded as the probability of the corresponding label.

[0128] 306. The server determines, among the multiple candidate tags, a candidate tag whose predicted value meets the target condition as the tag of the target video.

[0129] In a possible implementation, the server sorts the multiple candidate tags in descending order of predicted values ​​and determines the first target number of candidate tags among the sorted multiple candidate tags as tags for the target video.

[0130] The number of targets is set by the technician according to the actual situation and is not limited in the present embodiment. The greater the number of targets, the greater the number of tags for the target videos determined; the fewer the number of targets, the fewer the number of tags for the target videos determined.

[0131] The following will be combined Figure 4 As well as the various optional implementations in the above steps 301-306, the tag determination method provided in the embodiment of the present application is described, see Figure 4 , the label determination method provided in the embodiment of the present application is implemented by three units, which include an audio-visual feature extraction unit 401, a text feature extraction unit 402 and a feature fusion unit 403. Among them, the audio-visual feature extraction unit 401 is used to extract the first semantic feature of the target video, the text feature extraction unit 402 is used to extract the second semantic feature of the target video, and the feature fusion unit 403 is used to fuse the first semantic feature and the second semantic feature to obtain a fused semantic feature of the target video. The server can determine the label of the target video and the predicted value corresponding to the label based on the fused semantic feature. In some embodiments, the higher the predicted value, the more important the corresponding label is to the target video.

[0132] All of the above optional technical solutions can be combined in any way to form optional embodiments of the present application, and will not be described in detail here.

[0133] Through the technical solution provided by the embodiment of the present application, in the process of determining the video label, the first semantic feature of the target video and the second semantic feature of the descriptive text are respectively obtained. The first semantic feature and the second semantic feature are fused based on the first fusion parameter and the second fusion parameter to obtain a fused semantic feature, and the video label can be determined subsequently based on the fused semantic feature. In the process of fusing the first semantic feature and the second semantic feature, the first fusion parameter and the second fusion parameter obtained by decomposing the target fusion parameter are used, which can reduce the number of parameters in the feature fusion process, and also reduce the amount of calculation for fusing the first semantic feature and the second semantic feature, thereby improving the efficiency of determining the video label.

[0134] Figure 5 This is a schematic diagram of the structure of a label determination device provided in an embodiment of the present application, see Figure 5 The device includes: an acquisition module 501, a feature extraction module 502, a feature fusion module 503 and a label determination module 504.

[0135] The acquisition module 501 is used to acquire a target video and a description text of the target video.

[0136] The feature extraction module 502 is configured to perform feature extraction on the target video and the description text to obtain a first semantic feature of the target video and a second semantic feature of the description text.

[0137] The feature fusion module 503 is used to use the first fusion parameter and the second fusion parameter to fuse the first semantic feature and the second semantic feature to obtain the fused semantic feature of the target video. The first fusion parameter and the second fusion parameter are parameters obtained by decomposing the target fusion parameter. The target fusion parameter is a fusion parameter obtained based on the sample video, the sample description text of the sample video and the label training of the sample video.

[0138] The label determination module 504 is configured to determine a label of the target video based on the fused semantic features.

[0139] In one possible implementation, the feature extraction module 502 is configured to input the target video into an audio-visual feature extraction model, perform feature extraction on the target video using the audio-visual feature extraction model, and obtain the first semantic feature of the target video. Input the description text into a text feature extraction model, perform feature extraction on the description text using the text feature extraction model, and obtain the second semantic feature of the description text.

[0140] In a possible implementation, the feature extraction module 502 is configured to perform any of the following:

[0141] Through the audio-visual feature extraction model, convolution, residual connection and full connection are performed on at least one of the image and audio of the target video to obtain the first semantic feature of the target video.

[0142] Through the audio-visual feature extraction model, at least one of the image and audio of the target video is encoded based on the attention mechanism to obtain the first semantic feature of the target video.

[0143] In a possible implementation, the feature extraction module 502 is configured to perform convolution, residual connection, and full connection on the image of the target video through the audio-visual feature extraction model to obtain the image semantic features of the target video.

[0144] Through the audio-visual feature extraction model, the audio of the target video is convolved, residually connected, and fully connected to obtain the audio semantic features of the target video.

[0145] The image semantic feature and the audio semantic feature are fused to obtain the first semantic feature of the target video.

[0146] In a possible implementation, the feature extraction module 502 is configured to encode the image of the target video based on the attention mechanism through the audio-visual feature extraction model to obtain the image attention feature of the target video.

[0147] Through the audio-visual feature extraction model, the audio of the target video is encoded based on the attention mechanism to obtain the audio attention feature of the target video.

[0148] The image attention feature and the audio attention feature are fused to obtain the first semantic feature of the target video.

[0149] In a possible implementation, the feature extraction module 502 is configured to perform any of the following:

[0150] The description text is sequentially encoded using the text feature extraction model to obtain the second semantic feature of the target video.

[0151] Through the text feature extraction model, the description text is encoded based on the attention mechanism to obtain the second semantic feature of the target video.

[0152] The description text is embedded and encoded using the text feature extraction model to obtain the second semantic feature of the target video.

[0153] In one possible implementation, the feature fusion module 503 is configured to multiply the first fusion parameter by the first semantic feature to obtain a first fused feature, multiply the second fusion parameter by the second semantic feature to obtain a second fused feature, and multiply the first fused feature by the second fused feature to obtain a fused semantic feature of the target video.

[0154] In one possible implementation, the tag determination module 504 is configured to perform a full connection on the fused semantic features to obtain multiple prediction values ​​corresponding to multiple candidate tags, and determine the candidate tags whose prediction values ​​meet the target conditions as the tags of the target video.

[0155] In one possible embodiment, the label determination module 504 is configured to sort the plurality of candidate labels in descending order of predicted value. The first target number of candidate labels among the sorted plurality of candidate labels are determined as labels for the target video. In one possible embodiment, the apparatus further includes:

[0156] A decomposition module is used to perform tensor decomposition on the target fusion parameter to obtain the first fusion parameter and the second fusion parameter, and the product of the transposition of the first fusion parameter and the second fusion parameter is the target fusion parameter.

[0157] In one possible implementation, the device further includes:

[0158] The target fusion parameter acquisition module 501 is used to obtain a sample video and a sample description text of the sample video. Feature extraction is performed on the sample video and the sample description text to obtain a first sample semantic feature of the sample video and a second sample semantic feature of the sample description text. The first sample semantic feature and the second sample semantic feature are fused using the target fusion parameter to obtain a sample fused semantic feature of the sample video. Based on the sample fused semantic feature, a predicted label of the sample video is determined. Based on the difference information between the predicted label of the sample video and the annotated label of the sample video, the target fusion parameter is updated.

[0159] It should be noted that the label determination device provided in the above embodiment only uses the division of the above functional modules as an example to illustrate when determining a label. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device can be divided into different functional modules to complete all or part of the functions described above. In addition, the label determination device provided in the above embodiment and the label determination method embodiment are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0160] Through the technical solution provided by the embodiment of the present application, in the process of determining the video label, the first semantic feature of the target video and the second semantic feature of the descriptive text are respectively obtained. The first semantic feature and the second semantic feature are fused based on the first fusion parameter and the second fusion parameter to obtain a fused semantic feature, and the video label can be determined subsequently based on the fused semantic feature. In the process of fusing the first semantic feature and the second semantic feature, the first fusion parameter and the second fusion parameter obtained by decomposing the target fusion parameter are used, which can reduce the number of parameters in the feature fusion process, and also reduce the amount of calculation for fusing the first semantic feature and the second semantic feature, thereby improving the efficiency of determining the video label.

[0161] The present application embodiment provides a computer device for executing the above method. The computer device can be implemented as a terminal or a server. The structure of the terminal is first introduced below:

[0162] Figure 6 This is a schematic diagram of the structure of a terminal provided in an embodiment of the present application. The terminal 600 includes but is not limited to a mobile phone, a computer, an intelligent voice interaction device, a smart home appliance, a vehicle-mounted terminal, an aircraft, etc.

[0163] Typically, the terminal 600 includes one or more processors 601 and one or more memories 602 .

[0164] The processor 601 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 601 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). The processor 601 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 601 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 601 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0165] Memory 602 may include one or more computer-readable storage media, which may be non-transitory. Memory 602 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in memory 602 is used to store at least one computer program, which is executed by processor 601 to implement the tag determination method provided in the method embodiment of the present application.

[0166] The above-mentioned computer device can also be implemented as a server. The structure of the server is introduced below:

[0167] Figure 7 This is a structural diagram of a server provided in an embodiment of the present application. The server 700 may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 701 and one or more memories 702, wherein the one or more memories 702 store at least one computer program, and the at least one computer program is loaded and executed by the one or more processors 701 to implement the methods provided in the above-mentioned various method embodiments. Of course, the server 700 may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The server 700 may also include other components for implementing device functions, which will not be described in detail here.

[0168] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including a computer program. The computer program can be executed by a processor to perform the label determination method in the above embodiment. For example, the computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0169] In an exemplary embodiment, a computer program product or computer program is also provided, which includes a program code, which is stored in a computer-readable storage medium. A processor of a computer device reads the program code from the computer-readable storage medium, and the processor executes the program code, so that the computer device performs the above-mentioned label determination method.

[0170] In some embodiments, the computer program involved in the embodiments of the present application may be deployed and executed on a computer device, or on multiple computer devices located at one location, or on multiple computer devices distributed at multiple locations and interconnected through a communication network. Multiple computer devices distributed at multiple locations and interconnected through a communication network may constitute a blockchain system.

[0171] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or may be accomplished by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0172] The above are only optional embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A label determination method, characterized in that: The method comprises: Obtain a target video and a description text of the target video; Performing feature extraction on the target video and the description text to obtain a first semantic feature of the target video and a second semantic feature of the description text; Multiplying the first fusion parameter by the first semantic feature to obtain a first fusion feature; multiplying the second fusion parameter by the second semantic feature to obtain a second fusion feature; multiplying the first fusion feature and the second fusion feature to obtain a fusion semantic feature of the target video; the first fusion parameter and the second fusion parameter are parameters obtained by performing tensor decomposition on the target fusion parameter, the sum of the parameter amounts of the first fusion parameter and the second fusion parameter is less than the parameter amount of the target fusion parameter, and the target fusion parameter is a fusion parameter obtained by training based on a sample video, a sample description text of the sample video, and a label of the sample video; Based on the fused semantic features, a label of the target video is determined.

2. The method according to claim 1, characterized in that The performing feature extraction on the target video and the description text to obtain a first semantic feature of the target video and a second semantic feature of the description text includes: Inputting the target video into an audio-visual feature extraction model, performing feature extraction on the target video through the audio-visual feature extraction model to obtain the first semantic feature of the target video; The description text is input into a text feature extraction model, and feature extraction is performed on the description text by the text feature extraction model to obtain the second semantic feature of the description text.

3. The method according to claim 2, characterized in that The feature extraction of the target video by the audio-visual feature extraction model to obtain the first semantic feature of the target video includes any one of the following: Performing convolution, residual connection, and full connection on at least one of the image and audio of the target video using the audio-visual feature extraction model to obtain the first semantic feature of the target video; Through the audio-visual feature extraction model, at least one of the image and audio of the target video is encoded based on the attention mechanism to obtain the first semantic feature of the target video.

4. The method according to claim 3, characterized in that The step of performing convolution, residual connection, and full connection on at least one of the image and audio of the target video by the audio-visual feature extraction model to obtain the first semantic feature of the target video includes: Performing convolution, residual connection, and full connection on the image of the target video through the audio-visual feature extraction model to obtain the image semantic features of the target video; Performing convolution, residual connection, and full connection on the audio of the target video through the audio-visual feature extraction model to obtain audio semantic features of the target video; The image semantic feature and the audio semantic feature are fused to obtain the first semantic feature of the target video.

5. The method according to claim 3, characterized in that The step of encoding at least one of the image and audio of the target video by the audio-visual feature extraction model based on the attention mechanism to obtain the first semantic feature of the target video includes: Encoding the image of the target video based on the attention mechanism through the audio-visual feature extraction model to obtain the image attention feature of the target video; Encoding the audio of the target video based on the attention mechanism through the audio-visual feature extraction model to obtain the audio attention feature of the target video; The image attention feature and the audio attention feature are fused to obtain the first semantic feature of the target video.

6. The method according to claim 2, characterized in that The feature extraction of the description text by the text feature extraction model to obtain the second semantic feature of the description text includes any one of the following: Performing sequence encoding on the description text by using the text feature extraction model to obtain the second semantic feature of the target video; Encoding the description text based on the attention mechanism through the text feature extraction model to obtain the second semantic feature of the target video; The description text is embedded and encoded using the text feature extraction model to obtain the second semantic feature of the target video.

7. The method according to claim 1, characterized in that The step of determining the label of the target video based on the fused semantic features includes: Performing full connection on the fused semantic features to obtain multiple prediction values ​​corresponding to multiple candidate labels; A candidate tag whose predicted value meets the target condition among the multiple candidate tags is determined as the tag of the target video.

8. The method according to claim 7, characterized in that The step of determining the candidate tag whose predicted value meets the target condition among the multiple candidate tags as the tag of the target video includes: Sort the multiple candidate tags in descending order of predicted values; The first target number of candidate tags among the sorted plurality of candidate tags are determined as tags of the target video.

9. The method according to claim 1, characterized in that The product of the first fusion parameter and the transposition of the second fusion parameter is the target fusion parameter.

10. The method according to claim 1, characterized in that The method for obtaining the target fusion parameters includes: Obtaining a sample video and a sample description text of the sample video; Performing feature extraction on the sample video and the sample description text to obtain a first sample semantic feature of the sample video and a second sample semantic feature of the sample description text; fusing the first sample semantic feature and the second sample semantic feature using the target fusion parameter to obtain a sample fusion semantic feature of the sample video; Determining a predicted label of the sample video based on the sample fusion semantic features; The target fusion parameter is updated based on difference information between the predicted label of the sample video and the annotated label of the sample video.

11. A label determination device, characterized in that: The device comprises: An acquisition module, configured to acquire a target video and a description text of the target video; A feature extraction module is used to extract features from the target video and the description text to obtain a first semantic feature of the target video and a second semantic feature of the description text; A feature fusion module is configured to multiply a first fusion parameter by the first semantic feature to obtain a first fusion feature; multiply a second fusion parameter by the second semantic feature to obtain a second fusion feature; and multiply the first fusion feature by the second fusion feature to obtain a fusion semantic feature of the target video; the first fusion parameter and the second fusion parameter are parameters obtained by performing tensor decomposition on the target fusion parameter, the sum of the parameter amounts of the first fusion parameter and the second fusion parameter is less than the parameter amount of the target fusion parameter, and the target fusion parameter is a fusion parameter obtained by training based on a sample video, a sample description text of the sample video, and a label of the sample video; The label determination module is used to determine the label of the target video based on the fused semantic features.

12. The device according to claim 11, characterized in that The feature extraction module is used to: Inputting the target video into an audio-visual feature extraction model, performing feature extraction on the target video through the audio-visual feature extraction model to obtain the first semantic feature of the target video; The description text is input into a text feature extraction model, and feature extraction is performed on the description text by the text feature extraction model to obtain the second semantic feature of the description text.

13. The device according to claim 12, characterized in that The feature extraction module is configured to perform any of the following: Performing convolution, residual connection, and full connection on at least one of the image and audio of the target video using the audio-visual feature extraction model to obtain the first semantic feature of the target video; Through the audio-visual feature extraction model, at least one of the image and audio of the target video is encoded based on the attention mechanism to obtain the first semantic feature of the target video.

14. The device according to claim 13, characterized in that The feature extraction module is used to: Performing convolution, residual connection, and full connection on the image of the target video through the audio-visual feature extraction model to obtain the image semantic features of the target video; Performing convolution, residual connection, and full connection on the audio of the target video through the audio-visual feature extraction model to obtain audio semantic features of the target video; The image semantic feature and the audio semantic feature are fused to obtain the first semantic feature of the target video.

15. The device according to claim 13, characterized in that The feature extraction module is used to: Encoding the image of the target video based on the attention mechanism through the audio-visual feature extraction model to obtain the image attention feature of the target video; Encoding the audio of the target video based on the attention mechanism through the audio-visual feature extraction model to obtain the audio attention feature of the target video; The image attention feature and the audio attention feature are fused to obtain the first semantic feature of the target video.

16. The device according to claim 12, characterized in that The feature extraction module is configured to perform any of the following: Performing sequence encoding on the description text by using the text feature extraction model to obtain the second semantic feature of the target video; Encoding the description text based on the attention mechanism through the text feature extraction model to obtain the second semantic feature of the target video; The description text is embedded and encoded using the text feature extraction model to obtain the second semantic feature of the target video.

17. The device according to claim 11, characterized in that The label determination module is used to: Performing full connection on the fused semantic features to obtain multiple prediction values ​​corresponding to multiple candidate labels; A candidate tag whose predicted value meets the target condition among the multiple candidate tags is determined as the tag of the target video.

18. The device according to claim 17, characterized in that The label determination module is used to: Sort the multiple candidate tags in descending order of predicted values; The first target number of candidate tags among the sorted plurality of candidate tags are determined as tags of the target video.

19. The device according to claim 11, characterized in that The product of the first fusion parameter and the transposition of the second fusion parameter is the target fusion parameter.

20. The device according to claim 11, characterized in that The device further includes a target fusion parameter acquisition module, which is used to: Obtaining a sample video and a sample description text of the sample video; Performing feature extraction on the sample video and the sample description text to obtain a first sample semantic feature of the sample video and a second sample semantic feature of the sample description text; fusing the first sample semantic feature and the second sample semantic feature using the target fusion parameter to obtain a sample fusion semantic feature of the sample video; Determining a predicted label of the sample video based on the sample fusion semantic features; The target fusion parameter is updated based on difference information between the predicted label of the sample video and the annotated label of the sample video.

21. A computer device, characterized in that: The computer device includes one or more processors and one or more memories, wherein at least one computer program is stored in the one or more memories, and the computer program is loaded and executed by the one or more processors to implement the label determination method according to any one of claims 1 to 10.

22. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one computer program, and the computer program is loaded and executed by a processor to implement the label determination method according to any one of claims 1 to 10.

23. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the label determination method according to any one of claims 1 to 10 is implemented.

Images