Video data processing method and apparatus, electronic device, and storage medium

Abstract

Embodiments of the present disclosure provide a video data processing method and device, electronic equipment and storage medium. The method comprises: obtaining a clustering representation corresponding to a target topic label and to-be-processed video data comprising the target topic label, wherein the clustering representation is obtained by clustering a plurality of target video data comprising the target topic label; performing multi-modal feature extraction on the to-be-processed video data to obtain a multi-modal representation of the to-be-processed video data; and determining the relevance of the target topic label and the to-be-processed video data according to the clustering representation corresponding to the target topic label and the multi-modal representation of the to-be-processed video.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more specifically, to a video data processing method, a video data processing apparatus, an electronic device, and a computer-readable storage medium. Background Technology

[0002] Hashtags are identifiers used on content platforms to represent video content. They are commonly used to generate unique topics and concisely express the information contained in published content. On content platforms, appropriate hashtags can effectively attract content consumers to actively participate in related topics, thereby increasing video views.

[0003] However, in reality, some video creators add trending hashtags unrelated to the video content to their videos in order to attract users to click on them and gain popularity.

[0004] Among related technologies, judging the relevance between video content and hashtags through manual viewing is a method with poor accuracy and low efficiency. Summary of the Invention

[0005] This disclosure provides a video data processing method, a video data processing apparatus, a video characterization device, an electronic device, and a computer-readable storage medium. The method improves the efficiency and accuracy of relevance judgment between target topic tags and video data to be processed, thereby enabling the selection of video data to be processed with lower relevance to the target topic tags based on relevance, thus improving the accuracy of matching topic tags and video data.

[0006] This disclosure provides a video data processing method, which includes: obtaining a clustering representation corresponding to a target topic tag and video data to be processed including the target topic tag, wherein the clustering representation is obtained by clustering multiple target video data including the target topic tag; performing multimodal feature extraction on the video data to be processed to obtain a multimodal representation of the video data to be processed; and determining the correlation between the target topic tag and the video data to be processed based on the clustering representation corresponding to the target topic tag and the multimodal representation of the video data to be processed.

[0007] In some exemplary embodiments of this disclosure, obtaining the cluster representation corresponding to the target topic tag includes: obtaining multiple target video data including the target topic tag; performing multimodal feature extraction on each target video data to obtain the multimodal representation of each target video data; and clustering the multimodal representation of each target video data to obtain the cluster representation corresponding to the target topic tag.

[0008] In some exemplary embodiments of this disclosure, the method further includes: acquiring the target topic tag and multiple training video data including the target topic tag; extracting multimodal features from each training video data using a multimodal representation model to obtain multimodal representations of each training video data; obtaining predicted topic tags based on the multimodal representations of each training video data; and training the multimodal representation model based on the predicted topic tags and the target topic tag, so as to extract multimodal features from the video data to be processed or each target video data based on the trained multimodal representation model.

[0009] In some exemplary embodiments of this disclosure, the training video data includes training image data, training text data, and training audio data. The multimodal representation model includes a video representation module, a text representation module, an audio representation module, and a feature fusion module. The step of extracting multimodal features from each training video data using the multimodal representation model to obtain a multimodal representation of each training video data includes: extracting video features from each training image data using the video representation module to obtain video features of each training image data; extracting text features from each training text data using the text representation module to obtain text features of each training text data; extracting audio features from each training audio data using the audio representation module to obtain audio features of each training audio data; and fusing the video features of each training image data, the text features of each training text data, and the audio features of each training audio data using the feature fusion module to obtain a multimodal representation of each training video data.

[0010] In some exemplary embodiments of this disclosure, determining the relevance between the target topic tag and the video data to be processed based on the cluster representation corresponding to the target topic tag and the multimodal representation of the video to be processed includes: determining the similarity between the cluster representation corresponding to the target topic tag and the multimodal representation of the video to be processed; when the similarity is less than a similarity threshold, determining that the target topic tag and the video data to be processed are unrelated.

[0011] In some exemplary embodiments of this disclosure, the method further includes: acquiring candidate video data that does not include a target topic tag; performing multimodal feature extraction on the candidate video data to obtain a multimodal representation of the candidate video data; and determining the correlation between the target topic tag and the candidate video data based on the clustering representation corresponding to the target topic tag and the multimodal representation of the candidate video data.

[0012] In some exemplary embodiments of this disclosure, the method further includes: if the target topic tag is related to the candidate video data, then adding the target topic tag to the candidate video data.

[0013] This disclosure provides a video data processing apparatus, comprising: an acquisition module configured to acquire a clustering representation corresponding to a target topic tag and video data to be processed including the target topic tag, wherein the clustering representation is obtained by clustering multiple target video data including the target topic tag; an acquisition module configured to perform multimodal feature extraction on the video data to be processed to obtain a multimodal representation of the video data to be processed; and a determination module configured to determine the correlation between the target topic tag and the video data to be processed based on the clustering representation corresponding to the target topic tag and the multimodal representation of the video data to be processed.

[0014] This disclosure provides an electronic device, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the executable instructions to implement the video data processing method as described above.

[0015] This disclosure provides a computer-readable storage medium that, when executed by a processor of an electronic device, enables the electronic device to perform a video data processing method as described above.

[0016] This disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the video data processing method described above.

[0017] The video data processing method provided in this disclosure uses cluster representations of target topic tags obtained by clustering multiple target video data including target topic tags. Based on the cluster representations of target topic tags and the multimodal representations of the videos to be processed, the correlation between target topic tags and video data to be processed is determined, which improves the efficiency and accuracy of correlation judgment. Thus, video data to be processed with low correlation to target topic tags can be obtained by filtering based on correlation, thereby improving the accuracy of matching topic tags and video data and increasing the overall viewership of video data.

[0018] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0020] Figure 1 A schematic diagram of an exemplary system architecture to which the video data processing method of the present disclosure can be applied is shown.

[0021] Figure 2 This is a flowchart illustrating a video data processing method according to an exemplary embodiment.

[0022] Figure 3 This is a flowchart illustrating another video data processing method according to an exemplary embodiment.

[0023] Figure 4 This is a schematic diagram illustrating feature extraction of target video data and video data to be processed using a multimodal representation model, as shown in an example.

[0024] Figure 5 This is a flowchart illustrating another video data processing method according to an exemplary embodiment.

[0025] Figure 6 This is a schematic diagram of a multimodal representation model illustrated by an example.

[0026] Figure 7 This is a flowchart illustrating another video data processing method according to an exemplary embodiment.

[0027] Figure 8 This is a block diagram illustrating a video data processing apparatus according to an exemplary embodiment.

[0028] Figure 9 This is a schematic diagram illustrating the structure of an electronic device suitable for implementing exemplary embodiments of the present disclosure, according to an exemplary embodiment. Detailed Implementation

[0029] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0030] The features, structures, or characteristics described in this disclosure can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more specific details omitted, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0031] The accompanying drawings are merely illustrative of this disclosure, and the same reference numerals in the drawings denote the same or similar parts, thus omitting repeated descriptions of them. Some block diagrams shown in the drawings do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in at least one hardware module or integrated circuit, or in different network and / or processor devices and / or microcontroller devices.

[0032] The flowchart shown in the accompanying drawings is merely illustrative and does not necessarily include all content and steps, nor does it require execution in the described order. For example, some steps may be broken down, while others may be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0033] In this specification, the terms “a,” “an,” “the,” “the,” and “at least one” are used to indicate the presence of at least one element / component / etc.; the terms “comprising,” “including,” and “having” are used to indicate an open-ended inclusion and to mean that there may be other elements / components / etc. in addition to the listed elements / components / etc.; the terms “first,” “second,” and “third,” etc., are used only as markings and are not a limitation on the number of objects.

[0034] Figure 1 A schematic diagram of an exemplary system architecture to which the video data processing method of the present disclosure can be applied is shown.

[0035] like Figure 1 As shown, the system architecture may include server 101, network 102, terminal device 103, terminal device 104, and terminal device 105. Network 102 serves as the medium for providing a communication link between terminal device 103, terminal device 104, or terminal device 105 and server 101. Network 102 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0036] Server 101 can be a server that provides various services, such as a back-end management server that supports the devices operated by users using terminal devices 103, 104, or 105. The back-end management server can analyze and process received requests and other data, and feed back the processing results to terminal devices 103, 104, or 105.

[0037] Terminal devices 103, 104, and 105 can be smartphones, tablets, laptops, desktop computers, smart speakers, wearable smart devices, virtual reality devices, augmented reality devices, etc., but are not limited to these.

[0038] In this embodiment of the disclosure, server 101 can: obtain cluster representations corresponding to target topic tags and video data to be processed including target topic tags, wherein the cluster representation is obtained by clustering multiple target video data including target topic tags; perform multimodal feature extraction on the video data to be processed to obtain multimodal representations of the video data to be processed; and determine the correlation between the target topic tags and the video data to be processed based on the cluster representations corresponding to the target topic tags and the multimodal representations of the video data to be processed.

[0039] It should be understood that Figure 1 The number of terminal devices 103, 104, 105, network 102, and server 101 in the diagram is merely illustrative. Server 101 can be a single physical server, a server cluster consisting of multiple servers, or a cloud server. Depending on actual needs, it can have any number of terminal devices, networks, and servers.

[0040] The following will describe in more detail the various steps of the video data processing method in the exemplary embodiments of this disclosure, with reference to the accompanying drawings and examples. The method provided in the embodiments of this disclosure can be executed by any electronic device, such as the one described above. Figure 1 The server and / or terminal equipment in the process, but this disclosure does not limit this.

[0041] Figure 2 This is a flowchart illustrating a video data processing method according to an exemplary embodiment. Figure 2 As shown, the video data processing method provided in this embodiment may include the following steps.

[0042] In step S210, the cluster representation corresponding to the target topic tag and the video data to be processed including the target topic tag are obtained, wherein the cluster representation is obtained by clustering multiple target video data including the target topic tag.

[0043] In this embodiment of the disclosure, hashtags are information used on the content platform to represent the theme of a video. Videos published on the content platform may or may not include hashtags. Hashtags may be added by the video publisher or automatically added by the content platform based on the video content. This disclosure does not limit this.

[0044] In this embodiment of the disclosure, the target topic tag refers to any specified topic tag, such as a popular topic tag with high discussion volume; the target video data refers to video data including the target topic tag used to generate the cluster representation of the target topic tag. For the target topic tag, a preset number of target video data including the target topic tag can be selected; the target video data can usually be high-quality videos with a large number of views including the target topic tag. The number of target video data can be set according to actual needs, such as 1000; the video data to be processed refers to video data including the target topic tag that needs to have its correlation with the target topic tag calculated, such as video data just uploaded by the video creator.

[0045] In this embodiment of the disclosure, multiple target video data including target topic tags can be clustered to obtain cluster representations corresponding to the target topic tags.

[0046] In this embodiment of the disclosure, a clustering algorithm can be used to cluster a preset number (e.g., 1000) of target video data, and the cluster center point can be selected as the cluster representation corresponding to the target topic tag.

[0047] In an exemplary embodiment, obtaining the cluster representation corresponding to the target topic tag includes: obtaining multiple target video data including the target topic tag; performing multimodal feature extraction on each target video data to obtain the multimodal representation of each target video data; and clustering the multimodal representation of each target video data to obtain the cluster representation corresponding to the target topic tag.

[0048] Multimodal feature extraction can include, but is not limited to, video feature extraction, text feature extraction, and audio feature extraction.

[0049] For example, multimodal feature extraction can be performed on 1,000 target video data to obtain a fixed-dimensional (e.g., 512-dimensional) multimodal representation of each target video data; the fixed-dimensional multimodal representation of these 1,000 target video data can be clustered, and the resulting cluster representation can be used as the cluster representation corresponding to the target topic tag.

[0050] In this embodiment of the disclosure, a trained multimodal representation model can be used to extract multimodal features from each target video data. The training process of the multimodal representation model can be referred to... Figure 3 Detailed description of the embodiments.

[0051] In step S220, multimodal feature extraction is performed on the video data to be processed to obtain a multimodal representation of the video data to be processed.

[0052] In this embodiment of the disclosure, there may be one or more video data to be processed. Before performing multimodal feature extraction on the video data to be processed, the video data to be processed can be filtered according to the original information (video meta data), such as the number of plays, author level, etc., to remove low-quality videos and reduce the amount of computation.

[0053] In this embodiment of the disclosure, a trained multimodal representation model can be used to extract multimodal features from the video data to be processed. The training process of the multimodal representation model can be referred to... Figure 3 Detailed description of the embodiments.

[0054] In step S230, the correlation between the target topic tag and the video data to be processed is determined based on the clustering representation corresponding to the target topic tag and the multimodal representation of the video to be processed.

[0055] In this embodiment of the disclosure, the clustering representation corresponding to the target topic tag can represent the commonalities of videos including the target topic tag. Based on the clustering representation corresponding to the target topic tag and the multimodal representation of the video to be processed, the relevance between the video to be processed and the target topic tag can be determined, that is, whether the video to be processed is suitable for the target topic tag. This allows for the filtering out of videos to be processed that have low relevance to the target topic tag content, and the identification of the behavior of adding irrelevant hot topic tags to the video to be processed in order to gain more views (i.e., the behavior of riding the hot topic).

[0056] In an exemplary embodiment, the correlation between the target topic tag and the video data to be processed is determined based on the cluster representation corresponding to the target topic tag and the multimodal representation of the video to be processed, including: determining the similarity between the cluster representation corresponding to the target topic tag and the multimodal representation of the video to be processed; when the similarity is less than the similarity threshold, it is determined that the target topic tag and the video data to be processed are not related.

[0057] Specifically, the cosine similarity between the cluster representation corresponding to the target topic tag and the multimodal representation of the video to be processed can be calculated. If the similarity is greater than or equal to the similarity threshold, the target topic tag and the video data to be processed are determined to be related; if the similarity is less than the similarity threshold, the target topic tag and the video data to be processed are determined to be unrelated. The similarity threshold can be set according to actual needs, and this disclosure does not limit it.

[0058] In this embodiment of the disclosure, after determining that the target topic tag and the video data to be processed are irrelevant, the target topic tag can be automatically removed from the video data to be processed, so as to improve the accuracy of the topic tag in the video data and thus improve the accuracy of video data recommendation.

[0059] The video data processing method provided in this disclosure uses cluster representations of target topic tags obtained by clustering multiple target video data including target topic tags. Based on the cluster representations of target topic tags and the multimodal representations of the videos to be processed, the correlation between target topic tags and video data to be processed is determined, which improves the efficiency and accuracy of correlation judgment. Thus, video data to be processed with low correlation to target topic tags can be obtained by filtering based on correlation, thereby improving the accuracy of matching topic tags and video data and increasing the overall viewership of video data.

[0060] Figure 3 This is a flowchart illustrating another video data processing method according to an exemplary embodiment.

[0061] In this embodiment of the disclosure, Figure 3 The example illustrates the training process of a multimodal representation model; in Figure 2 Before step S220 of the video data processing method shown, Figure 3 The video data processing method shown may also include the following steps.

[0062] In step S310, target topic tags and multiple training video data including target topic tags are obtained.

[0063] In this embodiment of the disclosure, training video data refers to video data used for training a multimodal representation model that includes the target topic tag; for the target topic tag, a preset number of training video data including the target topic tag can be selected; the training video data and the target video data can be the same video data, partially overlapping video data, or different video data.

[0064] In step S320, multimodal features are extracted from each training video data using a multimodal representation model to obtain the multimodal representation of each training video data.

[0065] In this embodiment of the disclosure, each training video data can be input into a multimodal representation model, and the multimodal representation model can extract multimodal features from each training video data and output the multimodal representation of each training video data.

[0066] In this embodiment of the disclosure, the multimodal representation model can be a Bert (Bidirectional Encoder Representation from Transformer) model, or other neural network models capable of feature extraction.

[0067] In step S330, predicted topic tags are obtained based on the multimodal representations of each training video data.

[0068] In this embodiment of the disclosure, the multimodal representations of each training video data can be input into an MLP (Multilayer Perceptron) structure for processing to obtain predicted topic tags.

[0069] In step S340, a multimodal representation model is trained based on the predicted topic tags and the target topic tags, so as to extract multimodal features from the video data to be processed or each target video data according to the trained multimodal representation model.

[0070] In this embodiment of the disclosure, a loss function can be determined based on the predicted topic tag and the target topic tag, the loss value between the predicted topic tag and the target topic tag can be calculated based on the loss function, and the model parameters of the multimodal representation model can be adjusted based on the loss value so that the loss value between the predicted topic tag and the target topic tag is less than a preset value, thereby completing the training of the multimodal representation model.

[0071] Specifically, taking a target topic tag of "XXX" and multiple training video data including video data A and video data B as an example, a multimodal representation model is used to extract multimodal features from video data A to obtain its multimodal representation. Based on this multimodal representation, a predicted topic tag is obtained, and the multimodal representation model is trained using both the predicted and target topic tags "XXX". Similarly, the same multimodal representation model is used to extract multimodal features from video data B to obtain its multimodal representation. Based on this representation, a predicted topic tag is obtained, and the multimodal representation model is trained using both the predicted and target topic tags "XXX". In other words, for both video data A and video data B, the target topic tag is used as the training tag for the multimodal representation model. Since the actual topic tags for both video data A and video data B are the target topic tag, the resulting multimodal representation model is more accurate.

[0072] In this embodiment of the disclosure, after the multimodal representation model is trained, the trained multimodal representation model can be used to extract multimodal features from the video data to be processed, or the trained multimodal representation model can be used to extract multimodal features from each target video data.

[0073] Figure 4 This is a schematic diagram illustrating feature extraction of target video data and video data to be processed using a multimodal representation model, as shown in an example.

[0074] refer to Figure 4 The trained multimodal representation model 401 can be used to extract multimodal features from the video data to be processed 1 to the video data to be processed N, respectively, to obtain multimodal representation 1 to multimodal representation N; clustering is performed on multimodal representation 1 to multimodal representation N to obtain the clustered representation corresponding to the target topic tag.

[0075] refer to Figure 4 The trained multimodal representation model 401 can extract multimodal features from the video to be processed to obtain the multimodal representation of the video to be processed; the multimodal representation of the video to be processed and the original information of the video data to be processed can be fused to obtain the fused multimodal representation of the video to be processed.

[0076] refer to Figure 4 The similarity can be calculated based on the cluster representation corresponding to the target topic tag and the fused multimodal representation of the video to be processed, so as to determine the relevance between the video to be processed and the target topic tag.

[0077] The video data processing method provided in this embodiment uses multiple training video data including target topic tags as training data for a multimodal representation model, and uses the target topic tags as training tags for the multimodal representation model. Since the topic tags actually included in the training video data are all target topic tags, the multimodal representation model obtained through training is more accurate.

[0078] Figure 5 This is a flowchart illustrating another video data processing method according to an exemplary embodiment. Figure 6 This is a schematic diagram of a multimodal representation model illustrated by an example.

[0079] In an exemplary embodiment, the training video data includes training image data, training text data, and training audio data, and the multimodal representation model includes a video representation module, a text representation module, an audio representation module, and a feature fusion module.

[0080] refer to Figure 6 The multimodal representation model 401 may include a video representation module 601, a text representation module 602, an audio representation module 603, and a feature fusion module 604.

[0081] exist Figure 5 In the embodiments, the above Figure 3 Step S320 in the embodiment may further include the following steps.

[0082] In step S510, video features are extracted from each training image data through the video representation module to obtain the video features of each training image data.

[0083] In this embodiment of the disclosure, the original video corresponding to the target topic tag can be obtained, and image frames can be extracted from the original video. For example, 1 video image can be extracted per second to obtain training image data. Each training image data can include n video images, which are represented by video image 1 (Frame-1), video image 2 (Frame-2), ..., video image n (Frame-n).

[0084] In this embodiment of the disclosure, video images 1 to n can be encoded by the video encoder in the video representation module 601 to obtain video feature 1 (Emb-1) of video image 1, video feature 2 (Emb-2) of video image 2, ..., video feature n (Emb-n) of video image n; and video features 1 to n can be fused by the attention fusion structure in the video representation module 501 to obtain the video features (Vision-Embedding) of the training image data. The video features can also be called visual features, which is not limited in this disclosure.

[0085] In step S520, text features are extracted from each training text data through the text representation module to obtain the text features of each training text data.

[0086] In this embodiment of the disclosure, training text data corresponding to the target topic tag can be obtained. The training text data may include, but is not limited to, the title of the video, the video description added by the video publisher, and the text extracted from the video by OCR (Optical Character Recognition). The training text data may include text information 1 (Text-1), text information 2 (Text-2), ..., text information m (Text-m).

[0087] In this embodiment of the disclosure, the text information 1 to the text information m can be encoded by the multi-lingual encoder in the text representation module 602 to obtain the text features (Text-Embedding) of the training text data.

[0088] In step S530, the audio features of each training audio data are extracted by the audio representation module to obtain the audio features of each training audio data.

[0089] In this embodiment of the disclosure, the training audio data refers to the audio data contained in the training video data; a fixed-dimensional initial audio feature can be extracted from the training audio data using a deep learning algorithm; the initial audio feature is processed by the MLP structure in the audio representation module 603 to obtain the audio feature of the training audio data.

[0090] In step S540, the feature fusion module performs feature fusion on the video features of each training image data, the text features of each training text data, and the audio features of each training audio data to obtain the multimodal representation of each training video data.

[0091] In this embodiment of the disclosure, the feature fusion module 604 can perform feature fusion on the video features of each training image data, the text features of each training text data, and the audio features of each training audio data to obtain the multimodal representation of each training video data. The multimodal representation of each training video data is then input into the MLP structure for processing to obtain the predicted topic tags.

[0092] The video data processing method provided in this disclosure comprehensively considers multiple modalities of data, including image data, text data, and audio data, as well as the corresponding feature extraction process during the training of the multimodal representation model. The multimodal representation model trained in this way can extract multimodal features from video data, resulting in a more accurate multimodal representation.

[0093] Figure 7 This is a flowchart illustrating another video data processing method according to an exemplary embodiment.

[0094] In this embodiment of the disclosure, Figure 2 Based on the video data processing method shown, Figure 7 The video data processing method shown may also include the following steps.

[0095] In step S710, candidate video data excluding the target topic tag is obtained.

[0096] In this embodiment of the disclosure, candidate video data refers to video data that does not include the target topic tag but may be related to the target topic tag; for example, candidate video data may be video data with more than a preset number of plays within a preset time period.

[0097] In step S720, multimodal feature extraction is performed on the candidate video data to obtain the multimodal representation of the candidate video data.

[0098] In this embodiment of the disclosure, a trained multimodal representation model can be used to extract multimodal features from candidate video data to obtain multimodal representations of the candidate video data.

[0099] In step S730, the correlation between the target topic tag and the candidate video data is determined based on the clustering representation corresponding to the target topic tag and the multimodal representation of the candidate video data.

[0100] Specifically, the cosine similarity between the cluster representation corresponding to the target topic tag and the multimodal representation of the candidate video data can be calculated. If the similarity is greater than or equal to the similarity threshold, the target topic tag and the candidate video data are determined to be related; if the similarity is less than the similarity threshold, the target topic tag and the candidate video data are determined to be unrelated.

[0101] In an exemplary embodiment, the method may further include: if the target topic tag is related to the candidate video data, then adding the target topic tag to the candidate video data.

[0102] In this embodiment of the disclosure, if the target topic tag is related to the candidate video data, the target topic tag can be automatically added to the candidate video data, which can further increase the playback volume of the candidate video data and the accuracy of recommending candidate video data; at the same time, it can improve the matching degree between video data and topic tags.

[0103] The video data processing method provided in this disclosure, for candidate video data that does not include target topic tags, determines the correlation between target topic tags and candidate video data based on the clustering representation corresponding to the target topic tags and the multimodal representation of the candidate video data. This improves the efficiency and accuracy of correlation judgment, thereby allowing candidate video data with a high degree of correlation with the target topic tags to be obtained through correlation screening, thus improving the accuracy of matching topic tags and video data and increasing the overall viewership of video data.

[0104] It should also be understood that the above is only to help those skilled in the art better understand the embodiments of this disclosure, and is not intended to limit the scope of the embodiments of this disclosure. Those skilled in the art can obviously make various equivalent modifications or changes based on the examples given above. For example, some steps in the above methods may be unnecessary, or new steps may be added, etc. Alternatively, any combination of any two or more of the above embodiments may be used. Such modifications, changes, or combinations also fall within the scope of the embodiments of this disclosure.

[0105] It should also be understood that the above description of the embodiments of this disclosure focuses on highlighting the differences between the various embodiments. Similarities or differences not mentioned can be referred to each other, and for the sake of brevity, they will not be repeated here.

[0106] It should also be understood that the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this disclosure.

[0107] It should also be understood that, in the various embodiments of this disclosure, unless otherwise specified or in case of logical conflict, the terminology and / or descriptions between different embodiments are consistent and can be referenced by each other, and the technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationships.

[0108] The foregoing section provided a detailed example of the video data processing method disclosed herein. It is understood that, in order to implement the above functions, the computer device includes corresponding hardware structures and / or software modules for performing each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this disclosure can be implemented in hardware or a combination of hardware and computer software. Whether a function is implemented in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0109] The following are embodiments of the apparatus disclosed herein, which can be used to execute embodiments of the method disclosed herein. For details not disclosed in the apparatus embodiments of this disclosure, please refer to the embodiments of the method disclosed herein.

[0110] Figure 8 This is a block diagram illustrating a video data processing apparatus according to an exemplary embodiment. (Refer to...) Figure 8 The device 800 may include an acquisition module 810, a obtaining module 820, and a determining module 830.

[0111] The acquisition module 810 is configured to acquire the clustering representation corresponding to the target topic tag and the video data to be processed including the target topic tag, wherein the clustering representation is obtained by clustering multiple target video data including the target topic tag; the acquisition module 820 is configured to perform multimodal feature extraction on the video data to be processed to obtain the multimodal representation of the video data to be processed; and the determination module 830 is configured to determine the correlation between the target topic tag and the video data to be processed based on the clustering representation corresponding to the target topic tag and the multimodal representation of the video data to be processed.

[0112] In some exemplary embodiments of this disclosure, the acquisition module 810 is configured to perform: acquiring multiple target video data including the target topic tag; performing multimodal feature extraction on each target video data to obtain a multimodal representation of each target video data; and clustering the multimodal representations of each target video data to obtain a clustering representation corresponding to the target topic tag.

[0113] In some exemplary embodiments of this disclosure, the acquisition module 810 is configured to acquire the target topic tag and a plurality of training video data including the target topic tag; the acquisition module 820 is configured to perform: multimodal feature extraction on each training video data through a multimodal representation model to obtain multimodal representations of each training video data; obtain predicted topic tags based on the multimodal representations of each training video data; train the multimodal representation model based on the predicted topic tags and the target topic tag, so as to perform multimodal feature extraction on the video data to be processed or each target video data based on the trained multimodal representation model.

[0114] In some exemplary embodiments of this disclosure, the training video data includes training image data, training text data, and training audio data; the multimodal representation model includes a video representation module, a text representation module, an audio representation module, and a feature fusion module; the obtaining module 820 is configured to perform: extracting video features from each training image data through the video representation module to obtain video features of each training image data; extracting text features from each training text data through the text representation module to obtain text features of each training text data; extracting audio features from each training audio data through the audio representation module to obtain audio features of each training audio data; and fusing the video features of each training image data, the text features of each training text data, and the audio features of each training audio data through the feature fusion module to obtain a multimodal representation of each training video data.

[0115] In some exemplary embodiments of this disclosure, the determining module 830 is configured to perform: determining the similarity between the cluster representation corresponding to the target topic tag and the multimodal representation of the video to be processed; and determining that the target topic tag and the video data to be processed are unrelated when the similarity is less than a similarity threshold.

[0116] In some exemplary embodiments of this disclosure, the acquisition module 810 is configured to acquire candidate video data that does not include the target topic tag; perform multimodal feature extraction on the candidate video data to obtain the multimodal representation of the candidate video data; and the determination module 830 is configured to determine the correlation between the target topic tag and the candidate video data based on the clustering representation corresponding to the target topic tag and the multimodal representation of the candidate video data.

[0117] In some exemplary embodiments of this disclosure, the apparatus further includes an adding module configured to add the target topic tag to the candidate video data if the target topic tag is related to the candidate video data.

[0118] It should be noted that the block diagrams shown in the above figures are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor terminal devices and / or microcontroller terminal devices.

[0119] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0120] The following reference Figure 9 To describe an electronic device 900 according to such an embodiment of the present disclosure. Figure 9 The electronic device 900 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0121] like Figure 9 As shown, the electronic device 900 is presented in the form of a general-purpose computing device. The components of the electronic device 900 may include, but are not limited to: at least one processing unit 910, at least one storage unit 920, a bus 930 connecting different system components (including storage unit 920 and processing unit 910), and a display unit 940.

[0122] The storage unit stores program code, which can be executed by the processing unit 910 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. For example, the processing unit 910 can perform actions such as... Figure 2 The steps shown are as follows.

[0123] For example, electronic devices can achieve such Figure 2 The steps shown.

[0124] Storage unit 920 may include readable media in the form of volatile storage units, such as random access memory (RAM) 921 and / or cache memory 922, and may further include read-only memory (ROM) 923.

[0125] The storage unit 920 may also include a program / utility 924 having a set (at least one) of program modules 925, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0126] Bus 930 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0127] Electronic device 900 can also communicate with one or more external devices 970 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 900, and / or with any device that enables electronic device 900 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 950. Furthermore, electronic device 900 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 960. As shown, network adapter 960 communicates with other modules of electronic device 900 via bus 930. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 900, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0128] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0129] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory including instructions that can be executed by a processor of the device to perform the described method. Optionally, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0130] In an exemplary embodiment, a computer program product is also provided, including a computer program / instructions, which, when executed by a processor, implement the video data processing method described above.

[0131] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0132] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A video data processing method, characterized in that, include: Obtain the clustering representation corresponding to the target topic tag and the video data to be processed including the target topic tag, wherein the clustering representation is obtained by clustering multiple target video data including the target topic tag; Multimodal feature extraction is performed on the video data to be processed to obtain a multimodal representation of the video data to be processed; The correlation between the target topic tags and the video data to be processed is determined based on the clustering representation corresponding to the target topic tags and the multimodal representation of the video to be processed. Obtain the clustering representation corresponding to the target topic tags, including: Acquire multiple target video data including the target topic tags; Multimodal feature extraction is performed on each target video data to obtain a multimodal representation of each target video data; Clustering is performed on the multimodal representations of each target video data to obtain the clustering representations corresponding to the target topic tags; Based on the clustering representation corresponding to the target topic tag and the multimodal representation of the video to be processed, the correlation between the target topic tag and the video data to be processed is determined, including: Determine the similarity between the cluster representation corresponding to the target topic tag and the multimodal representation of the video to be processed; When the similarity is less than the similarity threshold, it is determined that the target topic tag and the video data to be processed are unrelated.

2. The video data processing method according to claim 1, characterized in that, Also includes: Obtain the target topic tags and multiple training video data including the target topic tags; Multimodal features are extracted from each training video data using a multimodal representation model to obtain the multimodal representation of each training video data. Predicted topic tags are obtained based on the multimodal representations of each training video data; The multimodal representation model is trained based on the predicted topic tags and the target topic tags, so as to extract multimodal features from the video data to be processed or each target video data based on the trained multimodal representation model.

3. The video data processing method according to claim 2, characterized in that, The training video data includes training image data, training text data, and training audio data, and the multimodal representation model includes a video representation module, a text representation module, an audio representation module, and a feature fusion module. The step of extracting multimodal features from each training video data using a multimodal representation model to obtain multimodal representations of each training video data includes: The video feature extraction of each training image data is performed by the video representation module to obtain the video features of each training image data. The text feature extraction is performed on each training text data by the text representation module to obtain the text features of each training text data. The audio feature extraction of each training audio data is performed by the audio representation module to obtain the audio features of each training audio data. The feature fusion module performs feature fusion on the video features of each training image data, the text features of each training text data, and the audio features of each training audio data to obtain a multimodal representation of each training video data.

4. The video data processing method according to claim 1, characterized in that, Also includes: Obtain candidate video data that does not include the target topic tag; Multimodal feature extraction is performed on the candidate video data to obtain a multimodal representation of the candidate video data; The correlation between the target topic tag and the candidate video data is determined based on the clustering representation corresponding to the target topic tag and the multimodal representation of the candidate video data.

5. The video data processing method according to claim 4, characterized in that, Also includes: If the target topic tag is related to the candidate video data, then the target topic tag is added to the candidate video data.

6. A video data processing apparatus, characterized in that, include: The acquisition module is configured to acquire the clustering representation corresponding to the target topic tag and the video data to be processed including the target topic tag, wherein the clustering representation is obtained by clustering multiple target video data including the target topic tag; The acquisition module is configured to perform multimodal feature extraction on the video data to be processed to obtain a multimodal representation of the video data to be processed; The determination module is configured to perform a determination of the correlation between the target topic tag and the video data to be processed based on the clustering representation corresponding to the target topic tag and the multimodal representation of the video to be processed; The acquisition module is configured to perform the following: acquire multiple target video data including the target topic tag; Multimodal feature extraction is performed on each target video data to obtain a multimodal representation of each target video data; clustering is performed on the multimodal representations of each target video data to obtain a clustering representation corresponding to the target topic tag; The determination module is configured to perform the following: determine the similarity between the cluster representation corresponding to the target topic tag and the multimodal representation of the video to be processed; when the similarity is less than a similarity threshold, determine that the target topic tag and the video data to be processed are unrelated.

7. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the executable instructions to implement the video data processing method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, wherein instructions in the computer-readable storage medium, when executed by a processor of an electronic device, enable the electronic device to perform the video data processing method as described in any one of claims 1 to 5.

Images