Data processing method and related apparatus

Abstract

The application relates to the technical field of computers, in particular to the field of artificial intelligence, and provides a data processing method and related device, which comprises the following steps: training an initial model based on a source domain video subset containing labeled videos and a target domain sample subset containing unlabeled videos; specifically, after the source domain videos and respective multi-modal features are obtained, a prediction branch network is used to obtain video understanding prediction results of the source domain videos, and based on the extracted multi-modal features, corresponding feature difference information is obtained, and respective domain prediction results of the source domain videos and the target domain samples are obtained, and then the initial model is subjected to parameter adjustment based on the video understanding prediction results, the feature difference information and the obtained respective domain prediction results. In this way, the distance between the source domain and the target domain can be shortened, so that the model trained based on the source domain videos can be applied to the target domain.

Description

Technical Field

[0001] This application relates to the field of computer technology, and provides a data processing method and related apparatus. Background Technology

[0002] In recent years, with the massive amount of video being published on various video media platforms, video understanding has become particularly important. Video understanding enables many basic functions of video processing, such as video segmentation, video classification, video information extraction, and video summarization; moreover, the accuracy of video understanding directly determines the accuracy of downstream tasks (such as automatic video tagging, video search, and video recommendation).

[0003] In the field of artificial intelligence, with the rapid development of deep learning in computer vision (CV) and natural language processing (NLP), most commonly used video understanding algorithms are currently based on multimodal supervised learning methods. Compared to unimodal methods, multimodal video understanding algorithms rely more heavily on labeled data. However, for videos in a specific domain, labeled data is scarce, leading to poor model performance and affecting the accuracy of video understanding. Summary of the Invention

[0004] This application provides a data processing method and related apparatus to improve the performance of video understanding models and increase the accuracy of video understanding.

[0005] In a first aspect, embodiments of this application provide a data processing method, including:

[0006] Obtain a training sample set, which includes: a source domain video subset containing labeled videos, and a target domain video subset containing unlabeled videos;

[0007] Based on the training sample set, an initial video understanding model, which includes at least a feature extraction network and a prediction branch network, is iteratively trained to obtain a target video understanding model. During each iteration, the following operations are performed:

[0008] The feature extraction network is used to extract multimodal features from the source domain video and the target domain video respectively, and the prediction branch network is used to obtain the video understanding prediction result corresponding to the multimodal features of the source domain video. Each multimodal feature represents multiple video content attributes.

[0009] Based on the extracted multimodal features, the corresponding feature difference information is obtained, as well as the domain prediction results of the source domain video and the target domain video respectively.

[0010] Based on the video understanding prediction results, the feature difference information, and the obtained prediction results for each domain, the parameters of the initial video understanding model are adjusted.

[0011] Secondly, embodiments of this application provide a data processing apparatus, including:

[0012] The sample acquisition unit is used to acquire a training sample set, which includes: a source domain video subset containing labeled videos and a target domain video subset containing unlabeled videos.

[0013] The model training unit is used to iteratively train an initial video understanding model, which includes at least a feature extraction network and a prediction branch network, based on the training sample set to obtain a target video understanding model. During one iteration, the following operations are performed:

[0014] The feature extraction network is used to extract multimodal features from the source domain video and the target domain video respectively, and the prediction branch network is used to obtain the video understanding prediction result corresponding to the multimodal features of the source domain video. Each multimodal feature represents multiple video content attributes.

[0015] Based on the extracted multimodal features, the corresponding feature difference information is obtained, as well as the domain prediction results of the source domain video and the target domain video respectively.

[0016] Based on the video understanding prediction results, the feature difference information, and the obtained prediction results for each domain, the parameters of the initial video understanding model are adjusted.

[0017] Optionally, the initial video understanding model further includes a domain difference branch network; then, when obtaining the corresponding feature difference information based on the extracted multimodal features, the model training unit is specifically used for:

[0018] The extracted multimodal features are input into the domain difference branch network to obtain feature difference information between the multimodal features.

[0019] Optionally, the initial video understanding model further includes a domain discrimination branch network; then, when obtaining the respective domain prediction results of the source domain video and the target domain video based on the extracted multimodal features, the model training unit is specifically used for:

[0020] The extracted multimodal features are input into the domain discrimination branch network to obtain the domain prediction results for the source domain video and the target domain video, respectively.

[0021] Optionally, the feature extraction network includes a feature extraction sub-network and a feature fusion sub-network; then, when using the feature extraction network to extract multimodal features from the source domain video, the model training unit is specifically used for:

[0022] The source domain video is input into the feature extraction subnetwork to obtain multiple unimodal features of the source domain video, each unimodal feature being used to characterize a video content attribute;

[0023] Multiple unimodal features of the source domain video are input into the feature fusion subnetwork to obtain the multimodal features of the source domain video.

[0024] Optionally, when the source domain video is input into the feature extraction subnetwork to obtain multiple unimodal features of the source domain video, the model training unit 1102 is used to perform at least two of the following operations:

[0025] The source domain video is input into the image feature extraction network in the feature extraction subnetwork to obtain the image features of the source domain video;

[0026] The source domain video is input into the audio feature extraction network in the feature extraction subnetwork to obtain the audio features of the source domain video;

[0027] The source domain video is input into the text feature extraction network in the feature extraction subnetwork to obtain the text features of the source domain video.

[0028] Optionally, when adjusting the parameters of the initial video understanding model based on the video understanding prediction results, the feature difference information, and the obtained prediction results for each domain, the model training unit is specifically used for:

[0029] Based on the video understanding prediction results, the feature difference information, and the obtained prediction results for each domain, the target loss value is determined;

[0030] Based on the target loss value, the parameters of the initial video understanding model are adjusted.

[0031] Optionally, when determining the target loss value based on the video understanding prediction result, the feature difference information, and the obtained prediction results for each domain, the model training unit is specifically used for:

[0032] Based on the video understanding prediction results, a video understanding sub-loss is determined, a feature difference sub-loss is determined based on the feature difference information, and a domain prediction sub-loss is determined based on the obtained domain prediction results.

[0033] The target loss value is determined based on the video understanding sub-loss, the feature difference sub-loss, and the domain prediction sub-loss.

[0034] Optionally, the data processing device further includes a model application unit, which is used for:

[0035] Obtain the videos to be processed associated with the target domain;

[0036] Based on the feature extraction network and prediction branch network in the target video understanding model, the video understanding prediction results corresponding to each video to be processed are obtained.

[0037] Optionally, when obtaining the video understanding prediction results corresponding to each video to be processed based on the feature extraction network and prediction branch network in the target video understanding model, the model application unit is used to perform at least one of the following operations:

[0038] Based on the feature extraction network and prediction branch network in the target video understanding model, the video category of the video to be processed is determined, and based on the determined video category, each video to be processed is archived in the target domain.

[0039] Based on the feature extraction network and prediction branch network in the target video understanding model, the video title of each video to be processed is determined, and videos in the target domain are recommended based on the determined video titles;

[0040] Based on the feature extraction network and prediction branch network in the target video understanding model, each video to be processed is segmented, and based on the video segments obtained after segmentation, a short video of the target domain is generated.

[0041] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the above-described data processing method.

[0042] Fourthly, embodiments of this application provide a computer-readable storage medium including a computer program, which, when run on an electronic device, causes the electronic device to perform the steps of the above-described data processing method.

[0043] Fifthly, embodiments of this application provide a computer program product comprising a computer program that, when executed by a computer device, implements the aforementioned data processing method.

[0044] In this embodiment, an unsupervised learning method is used to train an initial video understanding model based on a source domain video subset containing labeled videos and a target domain video subset containing unlabeled videos, thereby obtaining a target video understanding model. Specifically, during model training, a feature extraction network is used to extract the source domain videos and their respective multimodal features. Then, a prediction branch network is used to obtain the video understanding prediction results for the source domain videos. Based on the extracted multimodal features, corresponding feature difference information is obtained, as well as the domain prediction results for the source and target domain videos. Finally, based on the video understanding prediction results, feature difference information, and the obtained domain prediction results, the parameters of the initial video understanding model are adjusted.

[0045] In this way, by leveraging the feature differences between source and target domain videos, as well as the prediction results from each domain, the distance between the source and target domains can be narrowed, allowing the prediction branch network trained on the source domain video to be applied to the target domain. Simultaneously, training the prediction branch network based on labeled source domain videos can improve the accuracy and precision of video understanding.

[0046] Furthermore, the trained target video understanding model can be used to handle any task in video understanding, including but not limited to video segmentation, video classification, video information extraction, and video summarization, demonstrating strong versatility, flexibility, and ease of use.

[0047] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0048] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0049] Figure 1 This is a schematic diagram illustrating an application scenario provided in the embodiments of this application;

[0050] Figure 2 This is a schematic diagram of the structure of an initial video understanding model provided in an embodiment of this application;

[0051] Figure 3 A flowchart illustrating the data processing method provided in an embodiment of this application;

[0052] Figure 4 This is a schematic diagram of the feature extraction network provided in an embodiment of this application;

[0053] Figure 5 This is a schematic diagram illustrating the logic for obtaining feature difference information provided in an embodiment of this application.

[0054] Figure 6 A logical diagram illustrating the acquisition of domain prediction results provided in an embodiment of this application;

[0055] Figure 7 This is a schematic diagram of the structure of the target video understanding model provided in the embodiments of this application;

[0056] Figure 8 A flowchart illustrating a method for completing a video classification task using a target video understanding model, as provided in this application embodiment.

[0057] Figure 9 A flowchart illustrating a method for completing a video summarization task using a target video understanding model, as provided in this application embodiment.

[0058] Figure 10 A flowchart illustrating a method for completing video segmentation tasks using a target video understanding model, as provided in this application embodiment.

[0059] Figure 11 This is a schematic diagram of the structure of the data processing apparatus provided in the embodiments of this application;

[0060] Figure 12 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0062] For ease of understanding, the terms used in the embodiments of this application are explained below.

[0063] Cloud technology refers to a hosting technology that unifies a series of video resources, such as hardware, software, and networks, within a wide area network or local area network to enable the computation, storage, processing, and sharing of video data.

[0064] Cloud technology is a general term encompassing network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. It can form resource pools, providing flexible and convenient on-demand access. Cloud computing technology will become a crucial support. Backend services of technical network systems require substantial computing and storage resources, such as video websites. With the rapid development and application of the internet industry, each video may have its own tags in the future, requiring transmission to backend systems for logical processing. Videos with different tags will be processed separately, necessitating robust system support across various sectors, which can be achieved through cloud computing.

[0065] So-called artificial intelligence cloud services are generally also known as AI as a Service (AIaaS). This is currently a mainstream service model for artificial intelligence platforms. Specifically, AIaaS platforms break down several common AI services and provide them as independent or packaged services in the cloud. This service model is similar to opening an AI-themed marketplace: all developers can access and use one or more AI services provided by the platform through API interfaces. Some experienced developers can also use the AI ​​framework and AI infrastructure provided by the platform to deploy and maintain their own dedicated cloud artificial intelligence services.

[0066] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0067] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0068] The solution provided in this application relates to transfer learning technology in artificial intelligence. In this application embodiment, the source domain video subset contains labeled videos, and the target domain video subset contains unlabeled videos. Using the aforementioned training sample set, the model is trained to achieve good generalization ability in the target domain, that is, to accurately predict the labels of the videos to be processed in the target domain. Detailed data processing procedures are described below and will not be repeated here.

[0069] Currently, most commonly used video understanding algorithms employ multimodal supervised learning methods, such as Two-Stream Inflated 3D ConvNets (I3D) and Time Sensitive Networks (TSN). These multimodal supervised learning methods rely heavily on labeled videos. However, labeled videos are extremely scarce when handling domain-specific video understanding tasks, resulting in limited performance of video understanding algorithms and severely hindering their development.

[0070] For example, when dealing with video classification tasks, the largest existing video dataset is YouTube-8M, which contains more than 8 million video data, while the ImageNet dataset used for image classification tasks in the CV field contains 15 million image data. Therefore, the datasets available for video classification tasks are relatively scarce.

[0071] With the development of media technology, there is a massive amount of unlabeled video data in various video media platforms and applications (APPs). If we can effectively utilize this unlabeled video data, it will enrich the dataset used to process video understanding tasks, which is of great importance to the field of video understanding.

[0072] In this embodiment, an unsupervised learning method is used to train an initial video understanding model based on a source domain video subset containing labeled videos and a target domain video subset containing unlabeled videos, thereby obtaining a target video understanding model. Specifically, during model training, a feature extraction network is used to extract the source domain videos and their respective multimodal features. Then, a prediction branch network is used to obtain the video understanding prediction results for the source domain videos. Based on the extracted multimodal features, corresponding feature difference information is obtained, as well as the domain prediction results for the source and target domain videos. Finally, based on the video understanding prediction results, feature difference information, and the obtained domain prediction results, the parameters of the initial video understanding model are adjusted.

[0073] In this way, by leveraging the feature differences between source and target domain videos, as well as the prediction results from each domain, the distance between the source and target domains can be narrowed, allowing the prediction branch network trained on the source domain video to be applied to the target domain. Simultaneously, training the prediction branch network based on labeled source domain videos can improve the accuracy and precision of video understanding.

[0074] Furthermore, the trained target video understanding model can be used to handle any task in video understanding, including but not limited to video segmentation, video classification, video information extraction, and video summarization, demonstrating strong versatility, flexibility, and ease of use.

[0075] The preferred embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments and features in the embodiments of this application can be combined with each other without conflict.

[0076] refer to Figure 1 The diagram illustrates a possible application scenario provided by an embodiment of this application. This application scenario includes at least a terminal device 110 and a server 120. The number of terminal devices 110 can be one or more, and the number of servers 120 can also be one or more. This application does not specifically limit the number of terminal devices 110 and servers 120.

[0077] The terminal device 110 is equipped with a video understanding application, which can be a client application, a web application, a mini-program application, etc. The terminal device 110 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, smart home appliance, vehicle terminal, aircraft, etc., but is not limited to these.

[0078] Server 120 can be a backend server for video understanding applications, providing corresponding services for them. Server 120 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0079] Terminal device 110 and server 120 can be directly or indirectly connected via wired or wireless communication and communicate and transmit data through a network; this application does not impose any restrictions on this.

[0080] The data processing process in this embodiment can be divided into two stages: a model training stage and a model application stage. The model training stage is used to iteratively train an initial video understanding model, which includes at least a feature extraction network and a prediction branch network, based on a training sample set to obtain a target video understanding model. The model application stage is used to obtain the video understanding prediction result corresponding to the video to be processed based on the target video understanding model.

[0081] It should be noted that the data processing method in the embodiments of this application can be executed by the server or the terminal device alone, or by the server and the terminal device together.

[0082] For example, after a terminal device obtains a training sample set, it iteratively trains an initial video understanding model, which includes at least a feature extraction network and a prediction branch network, based on the training sample set to obtain a target video understanding model. Then, based on the target video understanding model, it obtains the video understanding prediction result corresponding to the video to be processed. Alternatively, after a server obtains a training sample set, it iteratively trains an initial video understanding model, which includes at least a feature extraction network and a prediction branch network, based on the training sample set to obtain a target video understanding model. Finally, based on the target video understanding model, it obtains the video understanding prediction result corresponding to the video to be processed.

[0083] See Figure 2 The diagram shown illustrates the structure of an initial video understanding model provided in this embodiment. This initial video understanding model is an adversarial learning adaptive model for handling video understanding tasks, and can also be referred to as a video understanding model. The initial video understanding model includes a feature extraction network and a domain adaptation module.

[0084] The feature extraction network is used to extract features from either the source or target domain video, obtaining multimodal features for both. These multimodal features can include any two of the following: text features, image features, and speech features. The feature extraction network used to extract source domain features shares weights with the feature extraction network used to extract target domain features.

[0085] The domain adaptation module includes a prediction branch network, a domain difference branch network, and a domain discrimination branch network. The prediction branch network can also be called the target task branch network, which is used to handle video understanding tasks. The domain difference branch network is used to narrow the feature differences between source domain videos and target domain videos. The domain discrimination branch network is used to obfuscate the content attributes between source domain videos and target domain videos.

[0086] Below, in conjunction with Figure 2 The initial video understanding model shown illustrates the data processing method proposed in the embodiments of this application.

[0087] See Figure 3 The diagram shown is a flowchart of a data processing method provided in an embodiment of this application. This method is applied to an electronic device, which may be a terminal device or a server. The specific flow of the method is as follows:

[0088] S301. Obtain the training sample set, which includes: a subset of source domain videos containing labeled videos, and a subset of target domain videos containing unlabeled videos.

[0089] In this embodiment, a domain consists of data and the probability distribution that generated that data, typically denoted by D, where P represents a probability distribution. Specifically, there are two basic domains: the source domain and the target domain. The source domain is the object to be transferred, and its data usually comes from labeled samples. The target domain is the object ultimately desired, and its data typically comes from a video understanding task, which may include unlabeled samples.

[0090] The source domain videos in the source domain video subset come from the source domain, and the target domain videos in the target domain video subset come from the target domain. Generally, the source domain has sufficient data to train a model for the source domain, while the target domain video data has less labeled target domain videos, making it difficult to directly train a model for the target domain. In this embodiment, by combining labeled source domain videos and unlabeled target domain videos, the knowledge learned in the source domain is transferred to the target domain to obtain the desired model.

[0091] It should be noted that, in this embodiment, there are no limiting requirements on the number of source domain videos and the number of target domain videos; these can be set according to actual needs. The source domain video subset and the target domain video subset have the same category. Taking video classification as an example of a video understanding task, if the video categories include running, long jump, and high jump, then the video categories of each source domain video in the source domain video subset include running, long jump, and high jump, and the video categories of each target domain video in the target domain video subset also include running, long jump, and high jump; that is, the classification categories of the two are consistent.

[0092] In this embodiment of the application, a video may correspond to one or more video categories. When a video corresponds to one video category, the video understanding task can be regarded as a single classification task. When a video corresponds to multiple video categories, the video understanding task can be regarded as a multi-classification task.

[0093] The source video can be a single video or a video segment from a single video; there are no restrictions on this.

[0094] S302. Based on the training sample set, iteratively train the initial video understanding model, which includes at least a feature extraction network and a prediction branch network, to obtain the target video understanding model.

[0095] Let's take one iteration as an example to illustrate S302. Specifically, in one iteration, the following operations are performed:

[0096] S3021. A feature extraction network is used to extract the multimodal features of the source domain video x and the target domain video y respectively. A prediction branch network is then used to obtain the video understanding prediction results corresponding to the multimodal features of the source domain video x. Each multimodal feature represents multiple video content attributes.

[0097] In practice, source domain videos in the source domain video subset can be paired with target domain videos in the target domain video subset to obtain multiple training sample pairs, and then model training can be performed on each training data pair.

[0098] It should be noted that in the embodiments of this application, a single iteration can process either a pair of source domain videos and target domain videos, or multiple pairs of source domain videos and target domain videos; there is no limitation on this. This document will only use a pair of source domain videos and target domain videos: source domain video x and target domain video y, as an example for illustration.

[0099] Wherein, source domain video x can be a source domain video in a subset of source domain videos, and target domain video y can be a target domain video in a subset of target domain videos.

[0100] Each multimodal feature is used to represent multiple video content attributes, including but not limited to images, audio, and text. A multimodal feature can refer to a dense vector with fixed dimensions, and the correlation between videos can be measured by the distance between these multimodal features. For example, each multimodal feature might be used to represent images, audio, and text; or, for instance, each multimodal feature might be used to represent images and audio.

[0101] It should be noted that in the embodiments of this application, either a single-modal analysis method (i.e., data processing based on single-modal features) or a multi-modal analysis method (i.e., data processing based on multi-modal features) can be used, and there is no limitation on which one is used. However, considering that video is a comprehensive media information containing images, audio, and text, it is difficult to obtain good video understanding and prediction results with a single feature. Therefore, in practical applications, this embodiment of the application will only use the multi-modal analysis method as an example for illustration.

[0102] See Figure 2As shown, the initial video understanding model can include a feature extraction network and a prediction branch network. The prediction branch network is used to perform the video understanding task and can also be called the target task branch network. The initial video understanding model can also include a domain difference branch network and a domain discrimination branch network. The domain difference branch network is used to narrow the feature differences between the source domain video and the target domain video, while the domain discrimination branch network is used to obfuscate the content attributes between the source domain video and the target domain video. Detailed introductions to the domain difference branch network and the domain discrimination branch network are provided below.

[0103] In some embodiments, see Figure 4 As shown, the feature extraction network includes a feature extraction subnetwork and a feature fusion subnetwork. Since the extraction process of multimodal features from source domain video x is the same as that from target domain video y, this paper will only use the extraction process of multimodal features from source domain video x as an example for explanation.

[0104] Specifically, a feature extraction network is used to extract multimodal features from the source domain video x, including:

[0105] The source domain video x is input into the feature extraction subnetwork to obtain multiple unimodal features of the source domain video x, each unimodal feature being used to characterize a video content attribute; the multiple unimodal features of the source domain video x are input into the feature fusion subnetwork to obtain the multimodal features of the source domain video x.

[0106] In this embodiment of the application, image features include, but are not limited to, color, texture, and shape, while audio features include, but are not limited to, zero-crossing rate, short-time energy, and short-time autocorrelation coefficient.

[0107] Taking source domain video as an example, please refer to Figure 4 As shown, the feature extraction network includes a feature extraction subnetwork and a feature fusion subnetwork. The source domain video is input into the feature extraction subnetwork to obtain the image features, audio features and text features of the source domain video. The image representation of the source domain video is that the target object is playing badminton, and the text representation is "serving the ball out of bounds". Then, the image features, audio features and text features of the source domain video are input into the feature fusion subnetwork to obtain the multimodal features of the source domain video.

[0108] Through the above implementation method, the extracted features contain video content attributes of each modality. Therefore, the content of each video can be effectively and accurately described from various dimensions such as images, audio, and text. In this way, more accurate feature difference information, domain discrimination results, and video understanding prediction results can be obtained, thereby improving the accuracy of domain transfer learning and thus improving the accuracy of video understanding.

[0109] One possible approach is to input the source domain video x into the feature extraction subnetwork to directly obtain multiple unimodal features of the source domain video x.

[0110] As another possible implementation, in order to improve the extraction accuracy of single-modal features, in some embodiments, the source domain video x is input into the feature extraction sub-network to obtain multiple single-modal features of the source domain video x, including at least two of the following operations:

[0111] Operation 1: Input the source domain video x into the image feature extraction network in the feature extraction sub-network to obtain the image features of the source domain video x.

[0112] It should be noted that in the embodiments of this application, a video contains multiple video frames, and each video frame corresponds to an image.

[0113] With the development of Transformer networks in various fields, in order to improve the extraction accuracy of single-modal features, a Transformer network corresponding to the content attributes can be used to extract the corresponding single-modal features. In the embodiments of this application, the image feature extraction network can use an image transformer (Vision Transformer), but is not limited to this.

[0114] For example, see Figure 4 As shown, after acquiring the source domain video, the source domain video is input into the image feature extraction network in the feature extraction subnetwork to obtain the image features of the source domain video. The image features of the source domain video represent that the target object in the image corresponding to the source domain video is playing badminton.

[0115] Operation 2: Input the source domain video x into the audio feature extraction network in the feature extraction sub-network to obtain the audio features of the source domain video x.

[0116] Audio feature extraction networks can employ, but are not limited to, audio transformers.

[0117] For example, see Figure 4 As shown, the source domain video is input into the text feature extraction network in the feature extraction subnetwork to obtain the text features of the source domain video. The text features of the source domain video represent the text corresponding to the source domain video 1 as "serve and step on the line".

[0118] Step 3: Input the source domain video x into the text feature extraction network in the feature extraction subnetwork to obtain the text features of the source domain video x.

[0119] Text feature extraction networks can employ, but are not limited to, Bidirectional Encoder Representation from Transformers (BERT).

[0120] For example, see Figure 4 As shown, the source domain video is input into the text feature extraction network in the feature extraction subnetwork to obtain the audio features of the source domain video. The audio features of the source domain video indicate that the audio corresponding to the source domain video is a certain background music.

[0121] By employing the above implementation method, single-modal feature extraction based on the feature extraction network corresponding to each content attribute can improve the representation accuracy of single-modal features. Furthermore, since the Transformer network can significantly improve model performance, obtaining single-modal features based on the Transformer network can significantly improve model training efficiency and simultaneously increase model accuracy.

[0122] After obtaining single-modal features, when obtaining multimodal features through a feature fusion subnetwork, the following methods can be used, but are not limited to:

[0123] The first approach is to directly fuse the features of each single modality. For example, audio features, image features, and text features can be fused to obtain the corresponding multimodal features.

[0124] The second approach involves weighting each unimodal feature separately and then summing the weighted features to obtain the joint feature, which is then used as the multimodal feature. For example, image features, audio features, and text features can be weighted separately, and the joint feature obtained after weighted summation can be used as the multimodal feature. The weight coefficients of each unimodal feature can be preset or obtained through model training.

[0125] To improve model accuracy, in this embodiment, the weight coefficients of each single-modal feature can be obtained through model training. Specifically, each single-modal feature can be assigned an initial weight. During the model training phase, the initial weight of each single-modal feature can be weighted and fused with the corresponding single-modal feature to obtain multimodal features. Then, through training, the target weight of each single-modal feature can be obtained. Thus, during the model application phase, the target weight of each single-modal feature can be weighted and fused with the corresponding single-modal feature to obtain multimodal features.

[0126] During model training, feature fusion can be performed based on an attention mechanism. Feature fusion methods based on an attention mechanism include the Transformer method based on self-attention and feature fusion methods based on soft-attention, but are not limited to these.

[0127] Since different modal features have different importance, the joint features obtained by weighted fusion of different modal features can be used as multimodal features to improve the accuracy of video understanding. At the same time, it can also improve the accuracy of domain difference and domain discrimination, thereby further reducing the feature space difference between the target domain and the source domain and improving the accuracy of video understanding.

[0128] S3022. Based on the extracted multimodal features, obtain the corresponding feature difference information, and obtain the respective domain prediction results for source domain video x and target domain video y.

[0129] In some embodiments, see Figure 2 As shown, the initial video understanding model also includes a domain difference branch network to bridge the feature differences between source domain videos and target domain videos. Specifically, based on the extracted multimodal features, corresponding feature difference information is obtained, including:

[0130] The extracted multimodal features are input into the domain difference branch network to obtain feature difference information between the multimodal features.

[0131] As a first possible implementation, the feature difference information can be obtained by using the Maximum Mean Discrepancy (MMD). Specifically, the MMD algorithm is used to determine the maximum mean difference between each multimodal feature, and the maximum mean difference is used as the feature difference information. The maximum mean difference is used to characterize the distance between two different but related distributions.

[0132] The maximum average difference can be calculated using, but is not limited to, the following formula (1):

[0133]

[0134] Where MMD represents the maximum mean difference. The maximum average difference is represented by The data is mapped to a reproducing kernel Hilbert space for measurement. This is used to characterize the feature mapping that maps multimodal features from the original space containing the source and target domain videos to the reproducing kernel Hilbert space. The distribution of multimodal features of the source domain video is S-distribution, and the distribution of multimodal features of the target domain video is T-distribution. E represents the expectation, which can be the average value of each element.

[0135] By matching the distributions of the source and target domains using the maximum mean difference, the feature space distance between the source and target domain videos is continuously reduced, thereby improving the performance of the target domain video understanding model.

[0136] As a second possible implementation, to facilitate computation and thus improve the efficiency of model training, the feature difference information can be represented in the form of the inner product of the regenerating kernel Hilbert space, that is, the square of the maximum average difference. The calculation method of the square of the maximum average difference will not be elaborated here.

[0137] As a third possible implementation, feature difference information can also be obtained using correlation alignment (CORAL). Specifically, the CORAL algorithm is used to determine the covariance distance between each multimodal feature, and the covariance distance is used as feature difference information.

[0138] Specifically, the covariance distance can be calculated using, but is not limited to, the following formula (2):

[0139]

[0140] Where CORAL represents the covariance distance, d represents the feature dimension of the multimodal feature, and ||·|| F Represents the Frobenius function, C S C represents the covariance of the multimodal features of the source domain video. T This represents the covariance of the multimodal features of the target domain video.

[0141] By measuring the difference between source domain features and target domain features using covariance distance, the feature space distance between source domain videos and target domain videos is continuously reduced, thereby improving the performance of the target domain video understanding model.

[0142] It should be noted that the embodiments in this application only use MMD and CORAL as examples for illustration, but are not limited to these.

[0143] For example, see Figure 5As shown, the source domain video is of basketball type and the target domain video is of football type. The multimodal features of the source domain video and the multimodal features of the target domain video are respectively input into the domain difference branch network to obtain the feature difference information between the multimodal features of the source domain video and the target domain video. This feature difference information is used to measure the difference between the multimodal features of the source domain video and the multimodal features of the target domain video.

[0144] Through the above implementation, since the domain discrimination branch network can narrow the feature differences between the source domain video and the target domain video, the features of the source domain and the target domain become very similar, reducing the feature space difference between the target domain and the source domain. As a result, when using the trained model to perform video understanding on the target domain video, the accuracy of video understanding can be improved.

[0145] In some embodiments, the initial video understanding model further includes a domain discrimination branch network for obfuscating content attributes between the source domain video and the target domain video; then, based on the extracted multimodal features, the respective domain prediction results for the source domain video and the target domain video are obtained, including:

[0146] The extracted multimodal features are input into the domain discrimination branch network to obtain the domain prediction results for the source domain video x and the target domain video y, respectively.

[0147] In this embodiment, for the domain discrimination branch, adversarial learning can be used to make the feature extraction network focus on information shared between domains and suppress information specific to each domain. The adversarial learning method is as follows: optimize the domain discrimination branch network so that it can correctly determine which domain its input belongs to; simultaneously, optimize the feature extraction network so that its output confuses the domain discrimination branch network. When the domain discrimination branch network cannot successfully determine which domain its input (i.e., multimodal features) comes from, it is considered that the output of the feature extraction network does not contain domain-specific information, that is, it eliminates the video content attributes unique to each domain and retains the video content attributes shared by the domains.

[0148] For example, see Figure 6 As shown, the source domain video is of basketball type, and the target domain video is of football type. The multimodal features of the source domain video are input into the domain discrimination branch network to obtain the domain prediction result of the source domain video. The domain prediction result of the source domain video indicates that the source domain video belongs to the target domain. Similarly, the multimodal features of the target domain video are input into the domain discrimination branch network to obtain the domain prediction result of the target domain video. The domain prediction result of the target domain video indicates that the target domain video belongs to the target domain. Based on the domain prediction result 1 and the domain prediction result 2, the domain discriminant sub-loss can be obtained. With continuous training, the domain discriminant sub-loss increases continuously, and the domain discrimination branch network will no longer be able to determine whether its input belongs to the target domain or the source domain.

[0149] Through the above implementation, since the domain discrimination branch network can confuse the content attributes between source domain videos and target domain videos, the model cannot determine whether the data comes from the source domain or the target domain. Therefore, when using the trained model to perform video understanding on target domain videos, the feature extraction network can focus on the information shared between domains and suppress the information specific to each domain, thereby improving the accuracy of video understanding when performing video understanding on target domain videos.

[0150] S3023. Based on the video understanding prediction results, feature difference information, and the obtained prediction results from various fields, adjust the parameters of the initial video understanding model.

[0151] Specifically, based on the video understanding prediction results, feature difference information, and the obtained prediction results from various domains, the parameters of the initial video understanding model are adjusted, including:

[0152] Based on the video understanding prediction results, feature difference information, and the obtained prediction results from various fields, the target loss value is determined; based on the target loss value, the parameters of the initial video understanding model are adjusted.

[0153] Using the above implementation method, based on the video understanding prediction results, feature difference information, and the obtained prediction results from various domains, the target loss value can be obtained, thereby adjusting the parameters of the video understanding model to be trained and obtaining a converged target video understanding model.

[0154] In some embodiments, the target loss value may be determined in, but is not limited to, the following ways:

[0155] Based on the video understanding prediction results, the video understanding sub-loss is determined, the feature difference sub-loss is determined based on feature difference information, and the domain prediction sub-loss is determined based on the obtained prediction results for each domain.

[0156] The target loss value is determined based on the video understanding sub-loss, feature difference sub-loss, and domain prediction sub-loss.

[0157] The video understanding sub-loss can be determined based on the video understanding prediction result of the source domain video x and the pre-annotated ground truth label of the source domain video x. The feature difference sub-loss can directly use feature difference information. The domain prediction sub-loss can be determined based on the obtained domain prediction results and the ground truth domain corresponding to the source domain video and the target domain video, respectively.

[0158] As one possible implementation, the target loss value can be obtained by weighting the target task sub-loss value, the domain discrimination sub-loss value, and the domain difference sub-loss value according to the preset weights of each sub-loss value.

[0159] Let Loss represent the target loss value, Loss_classification represent the target task sub-loss value, Loss_neighborhood difference represent the neighborhood discriminant sub-loss value, and Loss_discriminator represent the neighborhood difference sub-loss value. Then, the target loss value can be expressed by the following formula:

[0160] Loss = a1 * Loss_Classification + a2 * Loss_Domain Difference + a3 * Loss_Discriminator

[0161] Where a1, a2, and a3 are the weights of the sub-loss values ​​corresponding to the target task sub-loss value, the domain discrimination sub-loss value, and the domain difference sub-loss value, respectively.

[0162] For example, assuming a1, a2, and a3 are 0.4, 0.3, and 0.3 respectively, and the target task sub-loss value, the domain discriminant sub-loss value, and the domain difference sub-loss value are 1, 2, and 2 respectively, then the target loss value is 1.6.

[0163] As another possible implementation, the sum of the target task sub-loss value, the domain discrimination sub-loss value, and the domain difference sub-loss value can be directly used as the target loss value.

[0164] It should be noted that this implementation method can also be understood as the case where the values ​​of a1, a2, and a3 are all 1.

[0165] For example, assuming the target task sub-loss value, the domain discrimination sub-loss value, and the domain difference sub-loss value are 1, 2, and 2 respectively, then the target loss value is 5.

[0166] By adjusting the model parameters using the target loss value, the differences between the source domain video and the target domain video can be reduced, enabling the knowledge learned in the source domain to be accurately transferred to the target domain, thereby improving the accuracy of video understanding.

[0167] After obtaining the trained target video understanding model, target video understanding models for each video to be processed can be obtained based on the target video understanding model to process the target domain association. The target video understanding model can be deployed to the terminal or server to perform tasks such as video segmentation, video summarization and video classification.

[0168] Specifically, after obtaining the target video understanding model, the process also includes: acquiring each video to be processed associated with the target domain, and obtaining the video understanding prediction results corresponding to each video to be processed based on the feature extraction network and prediction branch network in the target video understanding model.

[0169] For example, see Figure 7As shown, the target video understanding model includes a feature extraction network and a prediction branch network. The video to be processed is input into the target video understanding model, and multimodal features of the video to be processed are obtained by extracting text features, image features and audio features of the video. Then, the multimodal features are input into the prediction branch network to obtain the video understanding prediction result of the video to be processed. The video understanding prediction result indicates that the video to be processed is classified as basketball.

[0170] By implementing the above methods and applying the target video understanding model provided in the embodiments of this application, the effect of video understanding can be improved, thereby enhancing the product experience.

[0171] Taking video understanding tasks as an example, specifically video classification, please refer to... Figure 8 As shown, the implementation process of the video understanding method mainly includes the following steps:

[0172] S801: Obtain the videos to be processed associated with the target domain.

[0173] The target audience uses a mobile device to capture footage, generating a video to be processed. This video is then published to the media platform corresponding to the target domain via a video application.

[0174] It should be noted that, in this embodiment of the application, there are no restrictive stipulations on the number of videos to be processed or the generation method. The videos to be processed can be newly shot or generated after cropping existing video resources.

[0175] S802: Based on the feature extraction network and prediction branch network in the target video understanding model, the video category of each video to be processed is determined.

[0176] Through transfer learning, the target video understanding model transfers the classification knowledge learned in the source domain to the target domain, thereby determining the video category of the video to be processed based on the multimodal features extracted from the video.

[0177] S803: Based on the determined video category, archive each video to be processed in the target domain.

[0178] For example, when the video category of the video to be processed is determined to be news, the video to be processed will be archived into the news video category of the target domain.

[0179] Taking video summarization as an example, see [link / reference]. Figure 9 As shown, the implementation process of the video understanding method mainly includes the following steps:

[0180] S901: Obtain the videos to be processed associated with the target domain. See S801 for details.

[0181] S902: Based on the feature extraction network and prediction branch network in the target video understanding model, determine the video title of each video to be processed.

[0182] Through transfer learning, the target video understanding model transfers the knowledge of extracting summaries learned in the source domain to the target domain, thereby determining the video title of each video based on the multimodal features extracted from each video to be processed.

[0183] S903: Recommend videos in the target domain based on a given video title.

[0184] For example, when the target video understanding model extracts the video title of the video to be processed as "Holiday self-driving tour route planning", it recommends self-driving tour route videos to different scenic spots from the target domain.

[0185] Taking the target task as video segmentation as an example, see [link / reference]. Figure 10 As shown, the implementation process of the video understanding method mainly includes the following steps:

[0186] S1001: Obtain the videos to be processed associated with the target domain.

[0187] For example, if the target domain is a TV series, the video to be processed could be an episode of a TV series.

[0188] S1002: Based on the feature extraction network and prediction branch network in the target video understanding model, each video to be processed is segmented.

[0189] Through transfer learning, the target video understanding model transfers the video segmentation knowledge learned in the source domain to the target domain. Based on the multimodal features extracted from the video to be processed, it segments the video into sub-segments with independent themes, thereby providing material for short video generation and video editing.

[0190] S1003: Generate a short video for the target domain based on the video segments obtained after segmentation.

[0191] For example, the series includes three scenes: a leader's visit, industrial transformation, and a family reunion. Each scene has an independent theme. By segmenting the scenes, the series is divided into three sub-segments. By further editing each sub-segment, short videos of the corresponding scenes are generated.

[0192] The target video understanding model provided in this application embodiment can be seamlessly integrated into all domain adaptation methods to complete various video understanding tasks. It is a pluggable component that does not require changes to the backbone network of the domain adaptation method. Without increasing the running time, it can significantly improve the performance of video understanding and has strong versatility, flexibility and ease of use.

[0193] In this application embodiment, a general and effective multimodal domain adaptation framework is proposed. With this framework, only the source domain and the target domain need to be constructed to complete a specific target task without any labeled data. At the same time, it can be applied to a variety of video understanding tasks and has strong versatility, flexibility and ease of use.

[0194] Specifically, after acquiring source and target domain videos, features of each modality are obtained through a network structure with shared weights. Then, multimodal feature fusion is performed on the features of each modality to obtain multimodal features. Finally, while performing the video understanding task, the goal of bridging the feature spaces of the source and target domains is achieved, thereby transferring knowledge from the source domain to the target domain and solving the problem of insufficient labeled data for the target task.

[0195] The target video understanding model provided in this application embodiment can be seamlessly integrated into all domain adaptive methods to complete various video understanding tasks. Since the target video understanding model is a pluggable component, there is no need to change the backbone network of the target video understanding model when completing various video understanding tasks. Thus, without increasing the running time, the performance of video understanding can be greatly improved, and it has strong versatility, flexibility and ease of use.

[0196] Based on the same inventive concept, embodiments of this application provide a data processing apparatus. For example... Figure 11 As shown, this is a structural schematic diagram of the data processing device 1100, which may include:

[0197] The sample acquisition unit 1101 is used to acquire a training sample set, which includes: a source domain video subset containing labeled videos and a target domain video subset containing unlabeled videos.

[0198] Model training unit 1102 is used to iteratively train an initial video understanding model, which includes at least a feature extraction network and a prediction branch network, based on the training sample set to obtain a target video understanding model. During one iteration, the following operations are performed:

[0199] The feature extraction network is used to extract multimodal features from the source domain video and the target domain video respectively, and the prediction branch network is used to obtain the video understanding prediction result corresponding to the multimodal features of the source domain video. Each multimodal feature represents multiple video content attributes.

[0200] Based on the extracted multimodal features, the corresponding feature difference information is obtained, as well as the domain prediction results of the source domain video and the target domain video respectively.

[0201] Based on the video understanding prediction results, the feature difference information, and the obtained prediction results for each domain, the parameters of the initial video understanding model are adjusted.

[0202] Optionally, the initial video understanding model further includes a domain difference branch network; then, when obtaining the corresponding feature difference information based on the extracted multimodal features, the model training unit 1102 is specifically used for:

[0203] The extracted multimodal features are input into the domain difference branch network to obtain feature difference information between the multimodal features.

[0204] Optionally, the initial video understanding model further includes a domain discrimination branch network; then, when obtaining the respective domain prediction results of the source domain video and the target domain video based on the extracted multimodal features, the model training unit 1102 is specifically used for:

[0205] The extracted multimodal features are input into the domain discrimination branch network to obtain the domain prediction results for the source domain video and the target domain video, respectively.

[0206] Optionally, the feature extraction network includes a feature extraction sub-network and a feature fusion sub-network; then, when using the feature extraction network to extract multimodal features from the source domain video, the model training unit 1102 is specifically used for:

[0207] The source domain video is input into the feature extraction subnetwork to obtain multiple unimodal features of the source domain video, each unimodal feature being used to characterize a video content attribute;

[0208] Multiple unimodal features of the source domain video are input into the feature fusion subnetwork to obtain the multimodal features of the source domain video.

[0209] Optionally, when the source domain video is input into the feature extraction subnetwork to obtain multiple unimodal features of the source domain video, the model training unit 1102 is used to perform at least two of the following operations:

[0210] The source domain video is input into the image feature extraction network in the feature extraction subnetwork to obtain the image features of the source domain video;

[0211] The source domain video is input into the audio feature extraction network in the feature extraction subnetwork to obtain the audio features of the source domain video;

[0212] The source domain video is input into the text feature extraction network in the feature extraction subnetwork to obtain the text features of the source domain video.

[0213] Optionally, when adjusting the parameters of the initial video understanding model based on the video understanding prediction results, the feature difference information, and the obtained prediction results for each domain, the model training unit 1102 is specifically used for:

[0214] Based on the video understanding prediction results, the feature difference information, and the obtained prediction results for each domain, the target loss value is determined;

[0215] Based on the target loss value, the parameters of the initial video understanding model are adjusted.

[0216] Optionally, when determining the target loss value based on the video understanding prediction result, the feature difference information, and the obtained prediction results for each domain, the model training unit 1102 is specifically used for:

[0217] Based on the video understanding prediction results, a video understanding sub-loss is determined, a feature difference sub-loss is determined based on the feature difference information, and a domain prediction sub-loss is determined based on the obtained domain prediction results.

[0218] The target loss value is determined based on the video understanding sub-loss, the feature difference sub-loss, and the domain prediction sub-loss.

[0219] Optionally, the data processing device further includes a model application unit 1103, which is used for:

[0220] Obtain each video to be processed associated with the target domain; based on the feature extraction network and prediction branch network in the target video understanding model, obtain the video understanding prediction result corresponding to each video to be processed.

[0221] Optionally, when using the target video understanding model to obtain the video understanding prediction results corresponding to each video to be processed, the model application unit 1103 is used to perform at least one of the following operations:

[0222] Based on the feature extraction network and prediction branch network in the target video understanding model, the video category of the video to be processed is determined, and based on the determined video category, each video to be processed is archived in the target domain.

[0223] Based on the feature extraction network and prediction branch network in the target video understanding model, the video title of each video to be processed is determined, and videos in the target domain are recommended based on the determined video titles;

[0224] Based on the feature extraction network and prediction branch network in the target video understanding model, each video to be processed is segmented, and based on the video segments obtained after segmentation, a short video of the target domain is generated.

[0225] For ease of description, the above sections are divided into modules (or units) according to their functions and described separately. Of course, in implementing this application, the functions of each module (or unit) can be implemented in one or more software or hardware components.

[0226] Regarding the apparatus in the above embodiments, the specific manner in which each unit executes the request has been described in detail in the embodiments related to the method, and will not be elaborated here.

[0227] Those skilled in the art will understand that various aspects of this application can be implemented as a system, method, or program product. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "system."

[0228] After introducing the data processing method and apparatus according to exemplary embodiments of this application, an electronic device according to another exemplary embodiment of this application will be described next.

[0229] Figure 12 This is a block diagram of an electronic device 1200 according to an exemplary embodiment. The device includes: a processor 1210; a memory 1220 for storing executable instructions of the processor 1210; and the processor 1210 and the processor 1210 are connected via a bus.

[0230] The processor 1210 is configured to execute instructions to implement the data processing method in this embodiment, for example... Figure 2 , Figure 8 , Figure 9 or Figure 10 The steps are shown in the figure.

[0231] Memory 1220 may be volatile memory, such as random-access memory (RAM); memory 1220 may also be non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); or memory 1220 may be any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 1220 may also be a combination of the above-described memories.

[0232] Processor 1210 may include one or more central processing units (CPUs) or digital processing units, etc. Processor 1210 implements the above-described data processing method when executing a computer program stored in memory 1220. In some embodiments, processor 1210 and memory 1220 may be implemented on the same chip; in some embodiments, they may be implemented on separate chips.

[0233] This application embodiment does not limit the specific connection medium between the processor 1210 and the memory 1220. This application embodiment takes the connection between the processor 1210 and the memory 1220 via a bus as an example. Figure 12 The diagram uses thick lines to describe the connections between other components; these are merely illustrative and not intended to be limiting. Buses can be categorized as address buses, data buses, control buses, etc. For ease of description, Figure 12 It is described using only a thick line, but does not indicate that there is only one bus or one type of bus.

[0234] Based on the same inventive concept, this application also provides a computer program product or computer program, which includes a computer program that, when run on a computer device, causes the computer device to perform the data processing methods provided in various exemplary embodiments of this application.

[0235] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, RAM, ROM, erasable programmable read-only memory (EPROM or flash memory), optical fibers, CD-ROMs, optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0236] The program product of the embodiments of this application may be in the form of a CD-ROM and include program code, and may run on a computing device. However, the program product of this application is not limited thereto. In this document, the readable storage medium may be any tangible medium that contains or stores a program that may be used by or in conjunction with a command execution system, apparatus, or device.

[0237] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. This propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with a command execution system, apparatus, or device.

[0238] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0239] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0240] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0241] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1The function specified in one or more boxes.

[0242] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0243] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A data processing method, characterized in that, The method includes: Obtain a training sample set, which includes: a source domain video subset containing labeled videos and a target domain video subset containing unlabeled videos, wherein the video categories of the source domain video subset and the target domain video subset are the same; Based on the training sample set, the video understanding model, which includes an initial domain difference branch network and a domain discrimination branch network comprising two feature extraction networks, a prediction branch network, and an initial domain difference branch network, is iteratively trained to obtain the target video understanding model. The two feature extraction networks share weights, and each feature extraction network includes a feature extraction sub-network and a feature fusion sub-network. During each iteration, the following operations are performed: Multiple unimodal features of the source domain video are extracted through a feature extraction subnetwork of a feature extraction network, and the multiple unimodal features of the source domain video are fused through a feature fusion subnetwork of the same feature extraction network to obtain the multimodal features of the source domain video. The multimodal features represent multiple video content attributes, including one of image, audio, and text. Multiple unimodal features of the target domain video are extracted by a feature extraction subnetwork of another feature extraction network, and the multiple unimodal features of the target domain video are fused by a feature fusion subnetwork of the other feature extraction network to obtain the multimodal features of the target domain video. Using the aforementioned prediction branch network, video understanding prediction results corresponding to the multimodal features of the source domain video are obtained; The extracted multimodal features are input into the domain difference branch network to obtain feature difference information between the multimodal features. The extracted multimodal features are also input into the domain discrimination branch network to obtain the domain prediction results for the source domain video and the target domain video, respectively. Based on the video understanding prediction results, the feature difference information, and the obtained prediction results for each domain, the parameters of the initial video understanding model are adjusted.

2. The method as described in claim 1, characterized in that, The step of inputting the source domain video into the feature extraction subnetwork to obtain multiple unimodal features of the source domain video includes at least two of the following operations: The source domain video is input into the image feature extraction network in the feature extraction subnetwork to obtain the image features of the source domain video; The source domain video is input into the audio feature extraction network in the feature extraction subnetwork to obtain the audio features of the source domain video; The source domain video is input into the text feature extraction network in the feature extraction subnetwork to obtain the text features of the source domain video.

3. The method as described in claim 1 or 2, characterized in that, The parameter adjustment of the initial video understanding model based on the video understanding prediction results, the feature difference information, and the obtained prediction results for each domain includes: Based on the video understanding prediction results, the feature difference information, and the obtained prediction results for each domain, the target loss value is determined; Based on the target loss value, the parameters of the initial video understanding model are adjusted.

4. The method as described in claim 3, characterized in that, The determination of the target loss value based on the video understanding prediction result, the feature difference information, and the obtained prediction results for each domain includes: Based on the video understanding prediction results, a video understanding sub-loss is determined, a feature difference sub-loss is determined based on the feature difference information, and a domain prediction sub-loss is determined based on the obtained domain prediction results. The target loss value is determined based on the video understanding sub-loss, the feature difference sub-loss, and the domain prediction sub-loss.

5. The method as described in claim 1 or 2, characterized in that, After obtaining the target video understanding model, the following is also included: Obtain the videos to be processed associated with the target domain; Based on the feature extraction network and prediction branch network in the target video understanding model, the video understanding prediction results corresponding to each video to be processed are obtained.

6. The method as described in claim 5, characterized in that, The step of obtaining video understanding prediction results for each video to be processed based on the feature extraction network and prediction branch network in the target video understanding model includes at least one of the following operations: Based on the feature extraction network and prediction branch network in the target video understanding model, the video category of the video to be processed is determined, and based on the determined video category, each video to be processed is archived in the target domain. Based on the feature extraction network and prediction branch network in the target video understanding model, the video title of each video to be processed is determined, and videos in the target domain are recommended based on the determined video titles; Based on the feature extraction network and prediction branch network in the target video understanding model, each video to be processed is segmented, and based on the video segments obtained after segmentation, a short video of the target domain is generated.

7. A data processing apparatus, characterized in that, include: The sample acquisition unit is used to acquire a training sample set, which includes: a source domain video subset containing labeled videos and a target domain video subset containing unlabeled videos, wherein the video categories of the source domain video subset and the target domain video subset are the same. The model training unit is used to iteratively train an initial video understanding model, which includes at least two feature extraction networks and a prediction branch network, based on the training sample set, to obtain a target video understanding model. The two feature extraction networks share weights, and each feature extraction network includes a feature extraction sub-network and a feature fusion sub-network. During one iteration, the following operations are performed: Multiple unimodal features of the source domain video are extracted through a feature extraction subnetwork of a feature extraction network, and the multiple unimodal features of the source domain video are fused through a feature fusion subnetwork of the same feature extraction network to obtain the multimodal features of the source domain video; wherein each unimodal feature is used to characterize a video content attribute, and the video content attribute includes one of image, audio, and text. Multiple unimodal features of the target domain video are extracted by a feature extraction subnetwork of another feature extraction network, and the multiple unimodal features of the target domain video are fused by a feature fusion subnetwork of the other feature extraction network to obtain the multimodal features of the target domain video. Using the aforementioned prediction branch network, video understanding prediction results corresponding to the multimodal features of the source domain video are obtained; The extracted multimodal features are input into the domain difference branch network to obtain feature difference information between the multimodal features. The extracted multimodal features are also input into the domain discrimination branch network to obtain the domain prediction results of the source domain video and the target domain video respectively. Based on the video understanding prediction results, the feature difference information, and the obtained prediction results for each domain, the parameters of the initial video understanding model are adjusted.

8. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of any of the methods described in claims 1 to 6.

9. A computer-readable storage medium, characterized in that, It includes a computer program that, when run on an electronic device, causes the electronic device to perform the steps of any of the methods described in claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, When the computer program is executed by a computer device, it implements the method of any one of claims 1-6.

Images