Video processing method and device, computer device and storage medium

Abstract

The application relates to a video processing method and device, computer equipment and a storage medium. The method comprises the following steps: extracting deep features of video samples in a source domain and a target domain respectively by an identification model to be trained; performing multi-time scale feature extraction on the deep features by a domain adaptation trainer to obtain video features in the source domain and the target domain in a multi-time scale; grouping and aligning the video features in the source domain and the target domain according to corresponding time nodes and time scale weights of the video features; the time scale weight is positively correlated with the amount of information expressed by the corresponding video features; adjusting model parameters of the identification model and continuing the adversarial training according to the adversarial loss between the video features in the source domain and the target domain in the same group and the category loss between the predicted category of the video sample in the source domain and the corresponding sample label until the training is ended when the training stop condition is met. The method can effectively improve the accuracy of video recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a video processing method and device, computer equipment and a storage medium. BACKGROUND

[0002] With the rapid development of image processing technology and artificial intelligence technology, video recognition technology has emerged, such as detecting and recognizing the behavior of objects in video content, to realize automatic recognition of the category of the video.

[0003] In related technologies, a network model is usually trained using a large number of labeled sample data, so as to classify and identify each static frame in the video of the labeled field through the trained network model. However, this method can only be used for classification and identification of videos in the labeled field, and cannot accurately identify videos in other fields, resulting in low accuracy of video recognition in other fields. SUMMARY

[0004] Therefore, it is necessary to provide a video processing method, device, computer equipment and storage medium capable of effectively improving the accuracy of video recognition to solve the above technical problems.

[0005] A video processing method, the method comprising:

[0006] extracting deep features of video samples in a source domain and a target domain respectively by a to-be-trained recognition model; the video samples in the source domain carrying sample labels;

[0007] extracting multi-time scale features of the deep features by a domain adaptation trainer to obtain video features in the source domain and the target domain in the multi-time scale respectively;

[0008] grouping and aligning the video features in the source domain and the target domain according to time nodes and time scale weights corresponding to the video features; the time scale weight being positively correlated with an information amount expressed by the corresponding video feature;

[0009] determining an adversarial loss according to differences between the video features in the source domain and the target domain in the same group;

[0010] determining a category loss based on differences between predicted categories of the video samples in the source domain and corresponding sample labels; the predicted categories being obtained by classifying the video features of the video samples in the source domain;

[0011] adjusting model parameters of the recognition model and continuing adversarial training according to the adversarial loss and the category loss until a training stop condition is met.

[0012] A video processing device, the device comprising:

[0013] a feature extraction module configured to extract deep features of video samples in a source domain and a target domain respectively by an identification model to be trained, the video samples in the source domain carrying sample labels;

[0014] a domain adaptation training module configured to perform multi-time scale feature extraction on the deep features by a domain adaptation trainer to obtain video features in the source domain and the target domain in multiple time scales respectively, and group and align the video features in the source domain and the target domain according to time nodes and time scale weights corresponding to the video features, the time scale weights being positively correlated with information amounts expressed by the corresponding video features;

[0015] a loss determination module configured to determine an adversarial loss according to differences between the video features in the source domain and the target domain in the same group, and determine a category loss based on differences between predicted categories of the video samples in the source domain and corresponding sample labels, the predicted categories being obtained by classifying the video features of the video samples in the source domain;

[0016] a parameter adjustment module configured to adjust model parameters of the identification model according to the adversarial loss and the category loss, and continue adversarial training until a training stop condition is met.

[0017] In an embodiment, the domain adaptation training module is further configured to perform multi-time scale convolution processing on the deep features by the domain adaptation trainer respectively to obtain convolution results corresponding to the deep features, and obtain the video features in the source domain and the target domain in multiple time scales respectively according to time node weights corresponding to the deep features and the corresponding convolution results.

[0018] In an embodiment, the domain adaptation trainer performs multi-time scale convolution processing by a convolution layer; the domain adaptation trainer further includes a time node attention layer; and the domain adaptation training module is further configured to assign corresponding time node weights to time nodes corresponding to the deep features respectively according to information amounts expressed by the deep features in the time nodes by the time node attention layer, the time node weights being positively correlated with the information amounts expressed by the deep features in the corresponding time nodes.

[0019] In an embodiment, the domain adaptation training module is further configured to determine information entropy corresponding to the video features in each time scale in the source domain and the target domain by a time scale attention layer of the domain adaptation trainer, the information entropy representing information amounts expressed by the corresponding video features, and assign corresponding time scale weights to the video features in each time scale respectively according to the information entropy.

[0020] In an embodiment, the domain adaptation training module is further configured to determine, by the domain adaptation trainer, video features to be aligned in the source domain and the target domain according to time nodes and time scale weights corresponding to the video features; divide the video features to be aligned into a group to obtain a plurality of groups of aligned video features; and the video features in each group include video features in the source domain and the target domain with the same time scale.

[0021] In an embodiment, the domain adaptation training module is further configured to determine time node weights of the video features in the source domain and the target domain at corresponding time nodes; and determine, from the video features in the source domain and the target domain, video features with matching time node weights and time scale weights in different domains as the video features to be aligned in the source domain and the target domain.

[0022] In an embodiment, the feature extraction module is further configured to extract initial features of the video samples in the source domain and the target domain by an initial feature extractor in the identification model to be trained; and extract deep features of the video samples in the source domain and the target domain by a target feature extractor in the identification model.

[0023] In an embodiment, the video processing apparatus further includes a classification module configured to perform classification based on the video features of the video samples in the source domain by a classifier in the identification model to obtain predicted categories of the video samples in the source domain; and the parameter adjustment module is further configured to adjust model parameters of the target feature extractor and the classifier in the identification model according to the adversarial loss and the category loss and continue adversarial training, so that the identification model reduces differences between the video features in the source domain and the target domain in the same group in the process of iterative training until a training stop condition is met.

[0024] In an embodiment, the loss determination module is further configured to determine a cross-entropy loss based on differences between predicted categories of the video samples in the target domain and predicted categories of the video samples in the source domain; and the parameter adjustment module is further configured to adjust model parameters of the identification model according to the adversarial loss, the category loss, and the cross-entropy loss and continue adversarial training, so that the identification model reduces differences between the video features in the source domain and the target domain in the same group in the process of iterative training until a training stop condition is met.

[0025] In an embodiment, after the training ends until the training stop condition is met, the video processing apparatus further comprises a video recognition module configured to perform deep feature extraction on a to-be-processed video in the target domain by using the trained recognition model to obtain target video features aligned in the source domain and the target domain; and perform classification and recognition processing on the to-be-processed video based on the target video features by using the recognition model.

[0026] In an embodiment, the video recognition module is further configured to perform preliminary feature extraction on the to-be-processed video by using the trained recognition model to obtain initial features of the to-be-processed video; and perform deep feature extraction on the initial features based on time nodes corresponding to the initial features and time scale weights to obtain target video features aligned in the source domain and the target domain.

[0027] In an embodiment, the target video features are target video behavior features; and the video recognition module is further configured to perform behavior recognition on an object in the to-be-processed video based on the target video behavior features by using the recognition model.

[0028] A computer device comprising a memory and a processor, the memory storing a computer program, and the processor implementing steps in a video processing method of embodiments of the present application when executing the computer program.

[0029] A computer readable storage medium storing a computer program, the computer program being executed by a processor to implement steps in a video processing method of embodiments of the present application.

[0030] A computer program product or computer program comprising computer instructions stored in a computer readable storage medium; a processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor implements steps in a video processing method of embodiments of the present application when executing the computer instructions.

[0031] The aforementioned video processing method, apparatus, computer equipment, and storage medium extract depth features from video samples in the source and target domains using a recognition model to be trained. A domain adaptation trainer then performs multi-temporal-scale feature extraction on these depth features, yielding multi-temporal-scale video features in both the source and target domains. These video features are grouped and aligned according to their corresponding time nodes and time-scale weights. Since the time-scale weights are positively correlated with the information content expressed by the corresponding video features, the extracted features can be grouped based on time nodes and time-scale weights. Furthermore, each group of features can undergo separate domain adversarial training, resulting in a more accurate alignment of the distribution of video features in the source and target domains. Then, based on the adversarial loss between video features in the source and target domains within the same group, and the class loss between the predicted class of the video sample in the source domain and the sample label carried by the video sample in the source domain, the model parameters of the recognition model are adjusted and adversarial training continues. This allows for a more accurate alignment of the distribution of video features in the source and target domains, enabling the trained recognition model to extract video features aligned in the source and target domains, thereby effectively improving the accuracy of classifying videos in the target domain. Attached Figure Description

[0032] Figure 1 This is an application environment diagram of a video processing method in one embodiment;

[0033] Figure 2 This is a flowchart illustrating a video processing method in one embodiment;

[0034] Figure 3 This is a block diagram of the domain adaptation trainer in one embodiment;

[0035] Figure 4 This is a block diagram of the structure of the convolutional layer in the domain adaptation trainer in one embodiment;

[0036] Figure 5 This is a flowchart illustrating a video processing method in another embodiment;

[0037] Figure 6 This is an overall architecture diagram of the recognition model during the training process in one embodiment;

[0038] Figure 7 This is a diagram showing the overall architecture of the recognition model during the training process in a specific embodiment.

[0039] Figure 8 This is a flowchart illustrating the video recognition processing steps in one embodiment;

[0040] Figure 9 This is a structural block diagram of a video processing device in one embodiment;

[0041] Figure 10 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0043] The video processing method provided in this application can be applied to computer devices. The computer device can be a terminal or a server. It is understood that the video processing method provided in this application can be applied to a terminal, a server, or a system including both a terminal and a server, and is implemented through interaction between the terminal and the server.

[0044] The video processing method provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. Terminal 102 first obtains video samples in the source and target domains from server 104 for training the recognition model. Then, terminal 102 extracts depth features from the video samples in the source and target domains using the recognition model to be trained; it performs multi-timescale feature extraction on the depth features using a domain adaptation trainer, obtaining multi-timescale video features in both the source and target domains; it groups and aligns the video features in the source and target domains according to the time nodes and timescale weights corresponding to the video features; it determines the adversarial loss based on the differences between the video features in the same group; it determines the category loss based on the difference between the predicted category and the corresponding sample label of the video sample in the source domain; and it adjusts the model parameters of the recognition model based on the adversarial loss and the category loss, continuing adversarial training until the training stopping condition is met.

[0045] The server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal 102 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, and this application does not impose any restrictions.

[0046] Cloud Computing is a delivery and usage mode of IT (Internet Technology) infrastructure, which refers to obtaining required resources in a scalable manner on demand through a network; and broad-sense cloud computing refers to a delivery and usage mode of services, which refers to obtaining required services in a scalable manner on demand through a network. Such services can be IT and software, Internet related, or other services. Cloud computing is a product of the development of grid computing, distributed computing, parallel computing, utility computing, network storage technologies, virtualization, load balancing, and other traditional computer and network technologies. With the development of the Internet, real-time data flow, and diversified connected devices, and the promotion of requirements for search services, social networks, mobile commerce, and open collaboration, cloud computing has rapidly developed. Unlike previous parallel distributed computing, the generation of cloud computing will revolutionize the entire Internet mode and enterprise management mode from the concept.

[0047] It can be understood that the video processing method in each embodiment of the present application adopts machine learning technology in artificial intelligence technology, and can train an identification model capable of accurately identifying and classifying videos. Artificial intelligence (AI) is the use of digital computers or machine controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, obtain knowledge and use the knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision making.

[0048] Machine Learning (ML) is a multi-disciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithmic complexity theory, etc. It is a specialized study of how computers simulate or implement human learning behavior to acquire new knowledge or skills, reorganize existing knowledge structure, and continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental approach to making computers intelligent. Its applications are widespread in various fields of artificial intelligence. Machine learning and deep learning generally include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and rule-based learning. It can be understood that the recognition model trained in some embodiments of the present application is trained using machine learning technology. Based on the machine learning technology, the recognition model can be trained to have high accuracy in classifying and recognizing videos in the target domain.

[0049] In one embodiment, as shown in Figure 2 , a video processing method is provided. Taking a computer device as an example, the computer device can be a terminal or a server in Figure 1 , including the following steps:

[0050] Step S202, extracting the deep features of the video samples in the source domain and the target domain by the to-be-trained recognition model.

[0051] It can be understood that a video refers to a continuous image frame, i.e., a video includes multiple image frames with time sequence, which can be a series of continuous static images generated by capturing dynamic images using photography.

[0052] The to-be-trained recognition model is an initial neural network model that needs to be trained. The recognition model is trained by video samples in the source domain and the target domain, so that the trained recognition model has the ability to accurately classify videos in the target domain.

[0053] The domain refers to a specific range or attribute. The source domain and the target domain both refer to the domain of the video. The domain of the video refers to the attribute range of the video. Different domain data refers to data with large appearance differences of the same category of objects. Videos in different domains usually have differences in appearance. For example, the domain of the video can include various scene domains and various style domains. Various scene domains include images generated in various environmental scenes. For example, videos captured in daytime scenes and videos captured in nighttime scenes, videos captured in indoor scenes and videos captured in outdoor scenes, i.e., videos in different scene domains.

[0054] It can be understood that the source domain, also referred to as the source field, is a video under the source domain belonging to one field range. The video under the target domain is a video belonging to another field range. In transfer learning, the field with existing knowledge is the source domain, and the field to be learned new knowledge is the target domain. The goal of transfer learning is to extract useful knowledge information from one or more source domain tasks and apply it to new target domain tasks, which can be specifically applied to cross-field video recognition tasks. When the data distribution of the source domain and the target domain is different, but the tasks corresponding to the two kinds of data are the same, this special transfer learning is domain adaptation.

[0055] The video samples include video samples under the source domain and video samples under the target domain. The video samples under the source domain carry sample labels. The sample labels are labels marked for the categories of the videos under the source domain, and are used for difference comparison with predicted categories output by the recognition model to adjust the model parameters of the recognition model. The sample labels can be specifically generated by manual labeling.

[0056] The computer device first acquires video samples under the source domain and the target domain, and then extracts deep features of the video samples under the source domain and the target domain through the to-be-trained recognition model. Specifically, the deep features of the video samples are information expressions of the depths of the video samples, that is, feature representations. The deep features of the video samples can include frame-level features and can also include segment-level features. The frame-level features refer to features corresponding to each frame in the video samples. The segment-level features refer to features corresponding to one segment in the video samples.

[0057] In some embodiments, the computer device inputs one video sample under the source domain and one video sample under the target domain into the to-be-trained recognition model in each round of training, and extracts deep features of the video samples under the source domain and the target domain through the to-be-trained recognition model to perform adversarial training.

[0058] In step S204, the domain adaptation trainer performs multi-time scale feature extraction on the deep features to obtain multi-time scale video features under the source domain and the target domain, respectively.

[0059] The domain adaptation trainer is a neural network used for feature extraction on the video samples under the source domain and the target domain, and aligns the features of the video samples under the source domain with the features of the video samples under the target domain, to perform adversarial training on the recognition model.

[0060] It can be understood that the time scale refers to a measure for measuring the time range. The time scale of the video can refer to the length of the video segment in the video sample. Specifically, the time scale can be determined according to the time length of the video segment, or can be determined according to the number of frames included in the video segment. For example, the time scale can be the video segment time length, such as 3 seconds, 4 seconds, 5 seconds, etc.; the time scale can also be the number of frames of the video segment, such as 3 frames, 4 frames, 5 frames, etc.

[0061] The computer device obtains video samples under the source domain and the target domain, and respectively extracts deep features of the video samples under the source domain and the target domain. After obtaining the deep features of the video samples under the source domain and the target domain, the computer device further inputs the deep features of the video samples under the source domain and the target domain into the domain adaptation trainer, and respectively further extracts multi-time scale features of the deep features of the video samples under the source domain and the target domain through the domain adaptation trainer to obtain multi-time scale video features under the source domain and the target domain.

[0062] It can be understood that the multi-time scale video features are video features corresponding to video frames of various time scales.

[0063] In step S206, the video features under the source domain and the target domain are grouped and aligned according to the time nodes corresponding to the video features and the time scale weights.

[0064] It can be understood that the time node of the video refers to a time point or a time period in the video, which can specifically refer to one or more frames in the video and the time sequence position in the entire video. The time node corresponding to the video feature is the time sequence position of the corresponding video frame in the corresponding video sample.

[0065] The time scale weight is positively correlated with the amount of information expressed by the corresponding video feature, that is, the more the amount of information expressed by the video feature, the higher the time scale weight of the video feature. It can be understood that the time scale weight of the video feature refers to the weight of the video feature itself of various time scales, rather than the weight of the time scale itself.

[0066] For video data, the most important feature is that it introduces a time dimension, and the information of the previous and subsequent frames needs to be combined to accurately analyze the content of the video. Therefore, for the video recognition task, the time sequence information needs to be combined as much as possible during the domain adaptation training process, the video features are reasonably modeled, and the modeled features are efficiently aligned.

[0067] After the computer device extracts the multi-time scale video features under the source domain and the target domain through the domain adaptation trainer, the computer device further groups and aligns the video features under the source domain and the video features under the target domain through the domain adaptation trainer.

[0068] Specifically, the computer device can first preliminarily align the video samples under the source domain and the video samples under the target domain according to the corresponding time nodes by the domain adaptation trainer, and then respectively group and align the video features under the source domain and the video features under the target domain according to the time scale weights corresponding to the video features under various time scales of the source domain and the target domain, to obtain multiple groups of aligned video features.

[0069] It can be understood that each group of aligned video features respectively includes the video features under the source domain and the video features under the target domain, for example, each group of aligned video features respectively includes one video feature under the source domain and one video feature under the target domain. The time nodes of the video features under the source domain and the target domain in each group of aligned video features can be corresponding or not corresponding.

[0070] Step S208, determining an adversarial loss according to the difference between the video features under the source domain and the video features under the target domain in the same group.

[0071] The adversarial loss refers to the distribution difference between the video features under the source domain and the video features under the target domain in the same group.

[0072] After the computer device groups and aligns the video features under the source domain and the target domain to obtain multiple groups of aligned video features, the computer device respectively calculates the difference between the video features under the source domain and the video features under the target domain in the same group, to obtain the corresponding adversarial loss.

[0073] Step S210, determining a category loss based on the difference between the predicted category of the video sample under the source domain and the corresponding sample label.

[0074] It can be understood that the predicted category refers to the category of the object in the video obtained by classifying the video. The predicted category of the video sample under the source domain is obtained based on the video features of the video sample under the source domain.

[0075] Specifically, after the computer device extracts the video features under multiple time scales of the source domain and the target domain by the domain adaptation trainer, the computer device further classifies the video samples under the source domain based on the video features under the source domain by the to-be-trained recognition model, to obtain the predicted category of the video sample under the source domain.

[0076] Then, the computer device compares and contrasts the predicted category of the video sample under the source domain with the corresponding sample label carried by the video sample under the source domain, and calculates the category loss between the predicted category of the video sample under the source domain and the corresponding sample label.

[0077] Step S212, according to the adversarial loss and the category loss, the model parameters of the recognition model are adjusted and the adversarial training is continued until the training stopping condition is met.

[0078] It can be understood that in the process of training the recognition model, the recognition model needs to be trained for multiple rounds of iteration. Among them, the current round is the current round of model training. Each round of iteration training of each round is to adjust the model parameters of the recognition model of the current round, so that the recognition model gradually converges to obtain the final recognition model.

[0079] Among them, the training stopping condition refers to the end condition of model training, for example, the training stopping condition can be that the preset number of iterations is reached, or the classification performance index of the recognition model after adjusting the parameters reaches the preset index.

[0080] Specifically, after the computer device obtains the adversarial loss between the video features in the source domain and the target domain in the same group, and the category loss between the predicted category of the video sample in the source domain and the corresponding sample label, then according to the adversarial loss and the category loss, the model parameters of the recognition model are adjusted in the direction of reducing the loss difference.

[0081] When the current round does not meet the training stopping condition, the computer device returns to the step of extracting the deep features of the video samples in the source domain and the target domain by the to-be-trained recognition model, so as to enter the next round. The computer device continues to perform multi-time scale feature extraction and grouping processing on the deep features of the video samples in another group of source domain and target domain, so as to perform iterative adversarial training. When the training stopping condition is met, the iterative training is stopped, thereby obtaining the trained recognition model.

[0082] It can be understood that the trained recognition model is a machine learning model with the ability to extract field-independent video features aligned in the source domain and the target domain from the video in the target domain, and accurately classify the video in the target domain. It can directly extract deep features and classify the video in the target domain to accurately identify the category of the object in the video.

[0083] In one of the embodiments, the difference between the predicted category of the video sample in the source domain and the corresponding sample label can be measured by a loss function, for example, a cross-entropy or mean square error function can be selected as the loss function. In the process of iterative training of the recognition model, the back propagation algorithm can be used to update the parameters in the direction of gradient descent, adjust the weights and biases to minimize the overall error, and gradually adjust the parameters of the recognition extraction model to iteratively train the recognition model. For example, the training can be ended when the value of the loss function is less than the pre-set value, and the recognition model capable of accurately and effectively classifying the video in the target domain is obtained.

[0084] In a conventional video domain adaptation processing manner, usually only a single frame feature weight is modeled, i.e., only the time sequence feature of a single frame can be identified and the corresponding importance weight is calculated, and then the features of all time nodes are uniformly aligned or each frame of video is aligned one by one according to the time node sequence. Due to the complexity of the video, the information expressed by different time nodes and different time scales and their importance to the domain adaptation task may be different. Although the conventional method has certain improvement on the video domain adaptation task, the mining of time sequence information in the video task is not sufficient, thereby resulting in low accuracy of video recognition.

[0085] In the embodiment, the importance of different time nodes can be calculated at different time scales, i.e., the importance of video features at corresponding time nodes according to the video time sequence is calculated. In addition, the important time scale in the video feature can be assigned a greater weight, thereby through introducing the time dimension, combining the time node information and the time scale information, the video content features are accurately analyzed, and the distribution of the video features in the source domain and the target domain can be more accurately aligned.

[0086] In the above video processing method, after the deep features of the video samples in the source domain and the target domain are extracted by the to-be-trained recognition model, the deep features are subjected to multi-time scale feature extraction by a domain adaptation trainer to obtain the multi-time scale video features in the source domain and the target domain. Then, the video features in the source domain and the target domain are grouped and aligned according to the time node and time scale weight corresponding to the video features. Since the time scale weight is positively correlated with the amount of information expressed by the corresponding video features, the extracted video features can be grouped according to the time node and time scale weight, and the features in each group are subjected to domain adversarial training separately, thereby the distribution of the video features in the source domain and the target domain can be more accurately aligned. According to the adversarial loss between the video features in the source domain and the target domain in the same group and the category loss between the predicted category of the video sample in the source domain and the corresponding sample label, the model parameters of the recognition model are adjusted and the adversarial training is continued, thereby the distribution of the video features in the source domain and the target domain can be more accurately aligned, and the trained recognition model has the ability to extract the aligned video features in the source domain and the target domain, thereby the accuracy of classifying the video in the target domain can be effectively improved.

[0087] In an embodiment, the step of performing multi-time scale feature extraction on the deep features by the domain adaptation trainer to obtain the multi-time scale video features in the source domain and the target domain respectively includes: performing multi-time scale convolution processing on the deep features by the domain adaptation trainer respectively to obtain the convolution results corresponding to the deep features; and obtaining the multi-time scale video features in the source domain and the target domain respectively according to the time node weights corresponding to the deep features and the corresponding convolution results.

[0088] It can be understood that in the process of feature extraction on the video, the feature extraction can be performed on image frames in the video. Specifically, the deep features of the video samples in the source domain and the target domain are extracted by the to-be-trained recognition model, which can be deep video features at the frame level, i.e., the deep features corresponding to each image frame in the video samples.

[0089] In image processing, the convolution operation refers to a series of operations on each pixel in the image using a convolution kernel, i.e., a convolution template. That is, the pixel gray value on the image point is multiplied by the corresponding value on the convolution kernel, and then all the multiplied values are added as the gray value of the pixel on the image corresponding to the middle pixel of the convolution kernel, and finally all the images are slid.

[0090] It can be understood that the time node weight refers to the weight of the video feature at the corresponding time node, i.e., the weight of the video feature itself at the corresponding time node, rather than the weight of the time node itself.

[0091] After the computer device extracts the deep features of the video samples in the source domain and the target domain by the to-be-trained recognition model, the computer device further inputs the deep features of the video samples in the source domain and the target domain into the domain adaptation trainer, so that the domain adaptation trainer performs multi-time scale convolution processing on the deep features in the source domain and the target domain respectively to obtain the convolution results corresponding to the deep features.

[0092] Specifically, the domain adaptation trainer can first perform multi-scale modeling on the input deep features, and extract the multi-time scale video features corresponding to the deep features in the source domain and the deep features in the target domain respectively by using multi-layer atrous convolution, so as to obtain the corresponding convolution results.

[0093] The computer device further determines the time node weights of the deep features in the source domain and the target domain at the corresponding time nodes by the domain adaptation trainer. Then, the computer device obtains the multi-time scale video features in the source domain and the target domain respectively according to the time node weights corresponding to the deep features and the corresponding convolution results.

[0094] In this embodiment, the domain adaptation trainer respectively performs multi-time scale convolution processing on the deep features in the source domain and the target domain, and according to the time node weights corresponding to the deep features and the corresponding convolution results, the multi-time scale video features in the source domain and the target domain can be accurately and effectively extracted according to the time sequence information of the video samples.

[0095] In one embodiment, the domain adaptation trainer performs multi-time scale convolution processing through a convolution layer; the domain adaptation trainer further includes a time node attention layer; and the video processing method further includes: through the time node attention layer, respectively assigning corresponding time node weights to the time nodes corresponding to the deep features according to the amount of information expressed by the deep features in each time node.

[0096] It can be understood that the network structure of the domain adaptation trainer includes a domain attention layer, and the domain attention layer further includes a convolution layer and a time node attention layer. The convolution layer is configured to respectively perform multi-time scale convolution processing on the deep features in the source domain and the target domain. The time node attention layer is configured to assign corresponding time node weights to the time nodes corresponding to the deep features.

[0097] For the domain adaptation task of the video, the video features in different time nodes have different importance, and by aligning the time nodes with larger amounts of information, the accuracy of domain adaptation can be effectively improved.

[0098] Specifically, the deep features of the video samples in the source domain and the target domain extracted by the to-be-trained recognition model can be video features with a lower time scale. Then, the computer device inputs the deep features of the video samples in the source domain and the target domain into the domain adaptation trainer, and through the convolution layer in the domain adaptation trainer, respectively performs multi-time scale convolution processing on the deep features in the source domain and the target domain according to the time sequence information corresponding to the video samples in the source domain and the target domain, to obtain the corresponding convolution results.

[0099] Meanwhile, the computer device also assigns corresponding time node weights to the time nodes corresponding to the deep features through the time node attention layer in the domain adaptation trainer according to the amount of information expressed by the deep features in each time node. Specifically, the time node weight is positively correlated with the amount of information expressed by the deep features in the corresponding time node, that is, the more the amount of information expressed by the deep features in each time node, the higher the time node weight assigned to the deep features in the corresponding time node.

[0100] Then, the computer device fuses the time node weight corresponding to the deep feature and the corresponding convolution result to obtain the multi-time scale video features in the source domain and the target domain respectively. The multi-time scale video features in the source domain and the target domain extracted by the domain adaptation trainer are high-time scale video features.

[0101] As shown in Figure 3 , it is a structural block diagram of the domain adaptation trainer in an embodiment. Referring to Figure 3 , the domain adaptation trainer includes a domain attention layer, a time scale attention layer and a time domain training module. The domain attention layer can be a domain attention empty residual layer, which further includes a time node attention layer and a plurality of convolution layers. The convolution layers can be a plurality of empty convolution layers, which are used to extract multi-time scale features from the input deep features. The time node attention layer is used to assign a corresponding time node weight to the deep features corresponding to the time nodes in the source domain and the target domain, and then perform multi-time scale convolution processing on the deep features through the plurality of empty convolution layers. Then, according to the time node weight corresponding to the deep features and the corresponding convolution result, the multi-time scale video features in the source domain and the target domain are obtained respectively. The time scale attention layer is used to determine the time scale weight corresponding to the multi-time scale video features. The time domain training module is used to group the video features in the source domain and the target domain according to the time node and the time scale weight, so as to separately perform domain adversarial training on the features in each group.

[0102] As shown in Figure 4 , it is a structural block diagram of the convolution layer in the domain adaptation trainer in an embodiment. The convolution layer in the domain adaptation trainer includes a time node attention layer and a plurality of empty convolution layers. The time node attention layer can further include a domain discriminator and a domain attention pooling layer. The domain discriminator is used to determine whether the input deep feature is from the video sample in the source domain or the video sample in the target domain. The cross entropy of the output result of the domain discriminator is used as a measure of the information amount expressed by the deep feature.

[0103] It can be understood that if the entropy of the output result of the domain discriminator is larger, it means that the domain discriminator is more difficult to distinguish the source domain of the input video sample deep feature, which means that the input deep feature contains more domain information, and the information amount expressed by the input deep feature is also more.

[0104] The domain attention pooling layer in the time node attention layer assigns a corresponding time node weight to the deep feature corresponding to the time node through the domain attention mechanism of measuring the entropy, so as to give a greater weight to the deep feature in the time node with a high information amount.

[0105] The multi-layered dilated convolution layer of the convolution layer in the domain trainer can be a convolution network, for example, a Conv convolution network. For example, the Conv convolution network can specifically include a dilated convolution layer with a dilated rate of 2 and a kernel size of 2, a ReLu activation function, and a 1x1 bottleneck convolution layer.

[0106] For example, the video features of the convolution layer in the input domain adaptation trainer are f s-1,t , f s-1,t+1 , f s-1,t+2 , f s-1 , t+3 , f s-1,t+4 The time node weights of the video features at each time node are determined by the time node attention layer, and the convolution results are added to the time node weights of the video features after the input video features are convolved by the multi-layered dilated convolution layer, thereby obtaining the multi-time scale video features f s,t .

[0107] In this embodiment, the domain adaptation trainer allocates corresponding time node weights to the time nodes corresponding to the depth features according to the amount of information expressed by the depth features at each time node, and then fuses the time node weights corresponding to the depth features with the corresponding convolution results, respectively obtaining the multi-time scale video features at the source domain and the target domain, so that the multi-time scale video features with higher time scales and rich in more time information can be more accurately extracted.

[0108] In one embodiment, before the video features at the source domain and the target domain are grouped and aligned according to the time nodes and time scale weights corresponding to the video features, the above video processing method further includes: determining the information entropy corresponding to the video features at each time scale at the source domain and the target domain by the time scale attention layer of the domain adaptation trainer; and respectively allocating corresponding time scale weights to the video features at each time scale according to the information entropy.

[0109] The information entropy refers to the probability of occurrence of information. The information entropy of the video feature can refer to the value of the amount of information expressed by the video feature. The information entropy represents the amount of information expressed by the corresponding video feature, that is, the greater the value of the information entropy, the more information the video feature expresses. Specifically, the information entropy can be cross-entropy, which can be used to measure the difference information between two probability distributions.

[0110] The time scale attention layer is included in the domain adaptation trainer. The computer device performs multi-time scale feature extraction on the deep features through the domain adaptation trainer, obtains video features of multiple time scales in the source domain and the target domain respectively, and further calculates information entropy corresponding to the video features of each time scale in the source domain and the target domain respectively through the time scale attention layer.

[0111] Specifically, the computer device can calculate the cross entropy between the probability that the current video feature belongs to the source domain and the probability that the current video feature belongs to the target domain through the domain adaptation trainer. The greater the cross entropy is, the more information the video feature represents, and vice versa. The computer device further assigns a corresponding time scale weight to the video feature according to the cross entropy, wherein the greater the cross entropy is, the higher the time scale weight assigned to the video feature, and vice versa.

[0112] In this embodiment, by determining the cross entropy of the video feature and taking the cross entropy as a measure of the amount of information represented by the video feature, different importance weights are assigned to video features of different time scales, thereby accurately measuring the value of video features of each time scale and accurately assigning matching importance weights to video features of each time scale.

[0113] In one embodiment, the step of grouping and aligning the video features in the source domain and the target domain according to the time nodes and time scale weights corresponding to the video features comprises: determining the video features to be aligned in the source domain and the target domain according to the time nodes and time scale weights corresponding to the video features through the domain adaptation trainer; and dividing the video features to be aligned into a group to obtain multiple groups of aligned video features.

[0114] It can be understood that grouping and aligning means grouping the video features in the source domain and the target domain respectively and aligning them in the source domain and the target domain, so that the video features in each group after alignment include video features of corresponding time scales in the source domain and the target domain.

[0115] When the computer device groups and aligns the video features in the source domain and the target domain through the domain adaptation trainer, the domain adaptation trainer first aligns the deep features of the video samples in the source domain and the target domain as a whole according to the time nodes of the video samples in the source domain and the time nodes of the video samples in the target domain.

[0116] The domain adaptation trainer further performs multi-time scale feature extraction on the deep features, and determines the time scale weights corresponding to the video features of each time scale in the source domain and the target domain respectively after obtaining the video features of multiple time scales in the source domain and the target domain respectively.

[0117] Then, the domain adaptation trainer further determines the video features to be aligned in the source domain and the target domain according to the time nodes and the time scale weights corresponding to the video features in the source domain and the target domain. Specifically, the computer device can determine the video features corresponding to the video frames or video segments of the same time scale that match the time nodes and the time scale weights in the source domain and the target domain as the video features to be aligned in the source domain and the target domain according to the same time scale. The matching of the time nodes can include at least one of the weight of the time node, the sequence of the time node, and the length of the time node.

[0118] In this embodiment, by grouping and aligning the video features in the source domain and the target domain according to the time nodes and the time scale weights, the domain adversarial training of the video features in each group in the source domain and the target domain can be more accurate.

[0119] In one embodiment, the step of determining the video features to be aligned in the source domain and the target domain according to the time nodes and the time scale weights corresponding to the video features by the domain adaptation trainer includes: determining the time node weights of the video features in the source domain and the target domain at the corresponding time nodes; and determining the video features with matching time node weights and time scale weights in different domains from the video features in the source domain and the target domain as the video features to be aligned in the source domain and the target domain.

[0120] The video features in different domains refer to the video features in the source domain and the video features in the target domain.

[0121] When the computer device groups and aligns the video features in the source domain and the target domain by the domain adaptation trainer, the computer device further assigns corresponding time node weights to the time nodes corresponding to the depth features according to the amount of information expressed by the depth features at each time node. After the domain adaptation trainer performs multi-time scale feature extraction on the depth features to obtain the multi-time scale video features in the source domain and the target domain, the domain adaptation trainer further determines the time scale weights corresponding to the video features of each time scale in the source domain and the target domain.

[0122] The domain adaptation trainer further determines the video features to be aligned in the source domain and the target domain according to the time nodes and the time scale weights corresponding to the video features in the source domain and the target domain. Specifically, the computer device can determine the video features corresponding to the video frames or video segments of the same time scale that match the time nodes and the time scale weights in the source domain and the target domain as the video features to be aligned in the source domain and the target domain according to the same time scale. The matching of the time nodes can include at least one of the weight of the time node, the sequence of the time node, and the length of the time node.

[0123] For example, the frame sequence of the video sample under the source domain is {s1, s2, s3, s4, s5,..., s20}, and the frame sequence of the video sample under the target domain is {t1, t2, t3, t4, t5,..., t20}. The time node of the video sample under the source domain includes 1-20, and the time node of the video sample under the target domain also includes 1-20. If the time node weight and the time scale weight of the video feature corresponding to the video segment with the frame sequence {s5-s10} in the video sample under the source domain match the time node weight and the time scale weight of the video feature corresponding to the video segment with the frame sequence {s15-s20} in the video sample under the target domain, the video feature corresponding to the video segment with the frame sequence {s5-s10} is determined as a group of video features to be aligned with the video feature corresponding to the video segment with the frame sequence {s15-s20}, and the video features corresponding to the two video segments are divided into a group.

[0124] In this embodiment, by grouping and aligning the video features under the source domain and the target domain according to the time node weight and the time scale weight, the video features with matching importance under the source domain and the target domain can be more accurately grouped and aligned, thereby effectively improving the matching degree of grouping and the accuracy of adversarial training.

[0125] In one embodiment, the steps of extracting the deep features of the video samples under the source domain and the target domain by the to-be-trained recognition model respectively include: extracting initial features of the video samples under the source domain and the target domain by an initial feature extractor in the to-be-trained recognition model; and extracting deep features of the video samples under the source domain and the target domain by a target feature extractor in the recognition model.

[0126] It can be understood that the to-be-trained recognition model includes an initial feature extractor and a target feature extractor. The initial feature extractor is configured to perform preliminary feature extraction on the video samples under the source domain and the target domain. The target feature extractor is configured to perform deep feature extraction on the video samples under the source domain and the target domain.

[0127] After the computer device obtains the video samples under the source domain and the target domain, the video samples under the source domain and the target domain are input to the to-be-trained recognition model. First, the initial feature extractor in the recognition model performs preliminary feature extraction on the video samples under the source domain and the target domain respectively. Specifically, the initial feature extractor can use shallow learning to detect key points in each image frame of the video sample to extract key point information, and then obtain initial features corresponding to each image frame according to the key point information, so as to extract the initial features of the video samples under the source domain and the target domain respectively.

[0128] Then the initial features of the video samples under the source domain and the target domain are input into a target feature extractor in the recognition model, and the initial features of the video samples under the source domain and the target domain are subjected to deep feature extraction by the target feature extractor. Specifically, the initial features are subjected to convolution, pooling, back propagation and other operations by the target feature extractor, and the deep feature expression of the initial features can be obtained, based on which the original video samples can be recognized and classified.

[0129] It can be understood that the initial feature extractor in the recognition model can be a trained feature extractor, and the target feature extractor in the recognition model can be a feature extractor to be trained. In the process of adversarial training of the video samples under the source domain and the target domain, the model parameters of the target feature extractor are constantly adjusted, so that the target feature extractor has the ability to extract field-independent video features that are aligned under the source domain and the target domain from the video to be processed under the target domain.

[0130] In one embodiment, as shown in FIG. 1, another video processing method is provided, which specifically includes the following steps: Figure 5

[0131] Step S502, initial features of video samples under the source domain and the target domain are extracted by an initial feature extractor in a recognition model to be trained.

[0132] Step S504, the initial features of the video samples under the source domain and the target domain are subjected to feature extraction by a target feature extractor in the recognition model, and deep features of the video samples under the source domain and the target domain are obtained.

[0133] Step S506, the deep features are subjected to multi-time scale feature extraction by a domain adaptation trainer, and multi-time scale video features under the source domain and the target domain are obtained.

[0134] Step S508, the video features under the source domain and the target domain are grouped and aligned according to the time nodes and time scale weights corresponding to the video features.

[0135] Step S510, an adversarial loss is determined according to the difference between the video features under the source domain and the target domain in the same group.

[0136] Step S512, a classifier of the recognition model is used to classify the video features of the video samples under the source domain, and a predicted class of the video samples under the source domain is obtained.

[0137] Step S514, a class loss is determined based on the difference between the predicted class of the video samples under the source domain and the corresponding sample label.

[0138] ​In step S516, the model parameters of the target feature extractor and the classifier in the recognition model are adjusted according to the adversarial loss and the category loss, and the adversarial training is continued, so that the recognition model reduces the difference between the video features in the source domain and the target domain in the same group in the process of iterative training, and the training is ended when the training stop condition is met.

[0139] It can be understood that the network structure of the recognition model to be trained includes the initial feature extractor, the target feature extractor, and the classifier. The classifier is used to classify the video samples according to the video features of the video samples.

[0140] The computer device performs multi-time scale feature extraction on the deep features in the source domain and the target domain through the domain adaptation trainer, obtains the multi-time scale video features in the source domain and the target domain, and aligns the video features in the source domain and the target domain in groups, and calculates the adversarial loss between the video features in the source domain and the target domain in the same group. The computer device further classifies the video samples in the source domain according to the video features of the video samples in the source domain through the classifier in the recognition model to be trained, and obtains the predicted categories of the video samples in the source domain.

[0141] In one of the embodiments, the computer device classifies the video samples in the source domain according to the deep features corresponding to the video samples in the source domain extracted by the target feature extractor through the classifier of the recognition model, thereby obtaining the predicted categories of the video samples in the source domain.

[0142] In another embodiment, the computer device can also classify the video samples in the source domain according to the multi-time scale features corresponding to the video samples in the source domain extracted by the domain adaptation trainer through the classifier of the recognition model, thereby obtaining the predicted categories of the video samples in the source domain.

[0143] Then, the computer device determines the category loss according to the difference between the predicted categories of the video samples in the source domain and the corresponding sample labels. Further, the model parameters of the target feature extractor in the recognition model are adjusted according to the adversarial loss and the category loss, and the model parameters of the classifier are adjusted according to the category loss, and the adversarial training is continued in the direction of reducing the difference between the video features in the source domain and the target domain in the same group and the direction of reducing the difference between the predicted categories of the video samples in the source domain and the corresponding sample labels. In order to make the recognition model reduce the difference between the video features in the source domain and the target domain in the same group in the process of iterative training, and end the training when the training stop condition is met, so that the trained recognition model can more accurately classify the videos in the target domain.

[0144] In an embodiment, the video processing method further comprises: determining a cross-entropy loss based on a difference between the predicted categories of the video samples in the target domain and the predicted categories of the video samples in the source domain.

[0145] The step of adjusting the model parameters of the identification model according to the adversarial loss and the category loss and continuing the adversarial training until the training is ended when the training stop condition is met comprises: adjusting the model parameters of the identification model according to the adversarial loss, the category loss and the cross-entropy loss and continuing the adversarial training, so that the identification model reduces the difference between the video features in the source domain and the target domain in the same group in the process of iterative training until the training is ended when the training stop condition is met.

[0146] The cross-entropy loss can be used to measure the similarity of the category probability distributions corresponding to the video samples in the source domain and the target domain, and can be further used to capture the difference between the classification prediction effect of the video samples in the source domain and the classification prediction effect of the video samples in the target domain.

[0147] After the computer device extracts the deep features corresponding to the video samples in the source domain and the target domain through the target feature extractor of the identification model, the computer device performs multi-time scale feature extraction on the deep features in the source domain and the target domain through the domain adaptation trainer to obtain the multi-time scale video features in the source domain and the target domain.

[0148] The computer device further classifies the video samples in the source domain according to the video features of the video samples in the source domain through the classifier in the identification model to be trained to obtain the predicted categories of the video samples in the source domain. The computer device classifies the video samples in the target domain according to the video features of the video samples in the target domain to obtain the predicted categories of the video samples in the target domain. The computer device then calculates the difference between the predicted categories of the video samples in the target domain and the predicted categories of the video samples in the source domain to obtain the corresponding cross-entropy loss.

[0149] Then, the computer device adjusts the model parameters of the target feature extractor in the identification model according to the adversarial loss, the category loss and the cross-entropy loss, adjusts the model parameters of the classifier in the identification model according to the category loss and the cross-entropy loss, and continues the adversarial training in the direction of reducing the difference between the video features in the source domain and the target domain in the same group and the direction of reducing the difference between the predicted categories of the video samples in the source domain and the corresponding sample labels. In order to make the identification model reduce the difference between the video features in the source domain and the target domain in the same group in the process of iterative training until the training is ended when the training stop condition is met.

[0150] In a specific embodiment, as shown in FIG. 1, it is a whole architecture diagram of the identification model in the training process in an embodiment. Figure 6 The computer device extracts the deep features corresponding to the video samples in the source domain and the target domain through the target feature extractor of the identification model, and performs multi-time scale feature extraction on the deep features in the source domain and the target domain through the domain adaptation trainer to obtain the multi-time scale video features in the source domain and the target domain.Figure 6 an overall architecture of the recognition model to be trained, including a fixed pre-trained feature extractor, a trainable feature extractor, a domain adaptation module, and a behavior recognition module.

[0151] The fixed pre-trained feature extractor, i.e., an initial feature extractor, can be a fixed trained backbone network in the recognition model, and can be specifically a feature extraction network constructed based on a ResNet101 (Residual Deep Convolutional Network), a ResNet50, an Inception-V3, an AlexNet, etc. In addition to using a 2D network, a 3D network such as an I3D network can also be used as a pre-trained feature extraction network for feature extraction.

[0152] The trainable feature extractor, i.e., a target feature extractor, is a fixed trainable feature extraction network in the recognition model, and is used to extract deep features of video samples in the source domain and the target domain. The target feature extractor can be constituted by a simple fully connected layer, or can be a feature extraction network constructed based on a DenseNet, a CNN (Convolutional Neural Networks), etc.

[0153] The behavior recognition module, i.e., a classifier, is a fixed classification network to be trained in the recognition model, and the classification network can be constituted by a pooling layer and a multilayer perceptron. For example, a simple pooling layer and a multilayer perceptron can be used, or other network structures such as a combination of a Temporal Relation Network and a multilayer perceptron, or a classifier constructed based on an ImageNet network, etc.

[0154] The domain adaptation module, i.e., a domain adaptation trainer, is a non-fixed domain adaptation trainer in the recognition model to be trained, and is used to transfer the knowledge learned from the video in the source domain, i.e., to align the distribution of the video features extracted by the feature extractor in the source domain and the target domain.

[0155] Specifically, the computer device can first perform preliminary single-frame feature extraction on video samples in the source and target domains using a pre-trained feature extractor. Then, a trainable feature extractor performs deep feature extraction on the initial single-frame features to obtain the corresponding deep features. These deep features from the video samples in the source and target domains are then input into the behavior recognition module and the domain adaptation module, respectively. The behavior recognition module classifies the behavior of objects in the video based on the video features corresponding to the video samples in the source and target domains. Simultaneously, the domain adaptation module transfers knowledge learned from the video samples in the source domain, aligning the distribution of multi-temporal-scale video features in the source and target domains. This enables the recognition model trained in the source domain to accurately recognize behaviors in videos in the target domain.

[0156] like Figure 7 The diagram shown illustrates the overall architecture of the recognition model during training in a specific embodiment. Referring to the diagram, the overall architecture of the recognition model to be trained includes a fixed pre-trained feature extractor, a trainable feature extractor F, a domain adaptation module, and a classification module C. Figure 7 The specific network structure of the domain adaptation module is shown in the image. (Refer to...) Figure 7 The domain adaptation module includes a domain discriminator, a gradient blocking layer, a domain attention hole residual layer, and a gradient reversal layer.

[0157] The training data includes a source domain dataset and a target domain dataset, which can be any dataset. The sample labels in the source and target domain datasets must be identical. The source domain dataset includes video samples from the source domain, and the target domain dataset includes video samples from the target domain.

[0158] For example, when the time scale is s, the video feature located at time node t is denoted as f. s,t Taking the features extracted by the feature extractor as the standard and considering it as time scale 1, the features extracted by the feature extractor can be represented as f. 1,t And f 1,t As input to the domain adaptation module.

[0159] The domain discriminator in the domain adaptation module is used to determine whether the input features come from source domain samples or target domain samples. The video features f... s,t The corresponding domain discriminator is denoted as D. s,t The higher the entropy of the domain discriminator's output, the more difficult it is for the domain discriminator to distinguish the source domain of the input video sample, indicating that the input video sample contains more domain information. Then, it passes through a domain attention-drilled residual layer H... s,tThe input deep features are subjected to multi-time scale feature extraction to obtain multi-time scale video features in the source domain and the target domain respectively. The domain attentional dilated residual layer H s,t The multi-time scale video features f s,t The expression of f

[0160] f s,t = H s,t (f s-1,t ,f s-1,t+d ,f s-1,t+2d ) (1);

[0161]

[0162] wherein s represents a time scale, t represents a time node, f s,t represents a video feature at the time node t at the time scale level s, E represents cross entropy, H s,t represents a domain attentional dilated residual layer, d represents a hole rate in the dilated convolution, D s,ti is a domain discriminator at the time scale s and the time node t, D s,ti (f s,t ) is an output result of the feature under the corresponding domain discriminator, i.e. a probability of belonging to the source domain. The activation function ReLU is included in the convolution layer Conv(fs,t1,fs,t2,fs,t3). By directly adding the output of the domain attentional pooling to the result of the convolution as a residual, the model can focus on the time nodes with large amounts of information and stabilize the model training. Thus, when extracting video features of different scales, the hole rate is gradually increased to quickly extract high-level features and reduce the network parameter quantity.

[0163] The part composed of the domain discriminator and the gradient blocking layer in the domain adaptation module is a time scale attention model Ns. The gradient blocking layer is used to block the gradient in the back propagation phase of the directional neural network training, so that the time scale attention model Ns can be trained independently of other parts of the network, thereby avoiding the influence on the domain adaptation trainer when training the time scale attention model Ns. The time scale attention model Ns calculates the time scale weight corresponding to the video feature of each time scale.

[0164] The specific expression can be: w’ s = E(N s (pooling([f s,1 ,f s,2 ,...,f s,ts ])) (3);

[0165] wherein w’ sThe original importance degree of the time scale s, wherein pooling represents a pooling operation on all time node features under the time scale s, and E represents a cross-entropy calculation.

[0166] The relative importance degree of the time scale s can be defined as:

[0167] wherein i represents a time scale, w' i is the original importance degree of the time scale i, and w s is the relative importance degree of the video feature of each time scale under the time scale s.

[0168] In order to avoid the negative effects brought by aligning the video features under different time nodes, the video features under different time nodes are grouped and aligned by the domain adaptation trainer for separate alignment.

[0169] The video features under the source domain and the target domain are aligned, and the expression can be:

[0170] wherein F is a trainable backbone feature extractor, D represents all domain discriminators, N represents a time scale attention model, H represents a domain attention hole residual layer, and L adv represents an adversarial loss between the video features under the source domain and the target domain.

[0171] The preliminarily aligned video features are grouped according to the time nodes and the time scales by the domain adaptation trainer, and separate domain adversarial training is performed within each group of features.

[0172] The expression of the adversarial loss function L adv can be:

[0173]

[0174] wherein D src represents the source domain dataset, D tar represents the target domain dataset. x represents the video data in the dataset, and y represents the sample label corresponding to the video data, for example, the behavior recognition classification label.

[0175] In order to train the recognition model in an end-to-end manner, a gradient reversal layer is added after the target feature extractor F to reverse the gradient during the forward propagation, so as to achieve the effect of the adversarial training in expression (5). At the same time of the domain adaptation adversarial training, the target feature extractor F needs to be trained together with the classifier, so that the video samples under the source domain can be correctly classified.

[0176] Specifically, the expression of the classification loss function L cls may be:

[0177]

[0178] The final training target can be:

[0179] wherein L cls is a classification loss function, B represents a pre-trained initial feature extractor, C represents a classifier, L ce is a cross-entropy loss function used to determine the cross-entropy loss between the predicted class of the video sample in the target domain and the predicted class of the video sample in the source domain.

[0180] During the training process, according to the adversarial loss, the category loss and the cross-entropy loss, the model parameters are adjusted in the direction of reducing the difference between the video features in the source domain and the target domain in the same group, and the direction of reducing the difference between the predicted class of the video sample in the source domain and the corresponding sample label, and the iterative adversarial training is continued, thereby more accurately aligning the video feature distributions in the source domain and the target domain, enabling the recognition model to have the ability to extract video features aligned in the source domain and the target domain, thereby effectively improving the accuracy of video classification.

[0181] In an embodiment, after the training is ended until the training stop condition is met, the above-mentioned video processing method further includes a video processing step, which specifically includes the following content: performing deep feature extraction on the to-be-processed video in the target domain by the trained recognition model to obtain target video features aligned in the source domain and the target domain; and performing classification and recognition processing on the to-be-processed video based on the target video features by the recognition model.

[0182] It can be understood that the trained recognition model has the ability to extract video features that are independent of the domain and aligned in the source domain and the target domain from the video in the target domain, and accurately classify the video in the target domain. The to-be-processed video is a video to be classified.

[0183] After the computer device obtains the trained recognition model, the recognition model can be used to classify and recognize the video in the target domain. Specifically, after the computer device obtains the to-be-processed video in the target domain, the to-be-processed video is input into the recognition model, and the recognition model first performs deep feature extraction on the to-be-processed video.

[0184] Since the trained recognition model can extract domain-independent video features that are aligned in both the source and target domains from the video in the target domain, by performing deep feature extraction on the video to be processed using the recognition model, the target video features that are domain-independent and aligned in both the source and target domains can be directly extracted from the video to be processed.

[0185] Then the recognition model can classify and recognize the video to be processed based on the characteristics of the target video, thereby accurately identifying the category of the object in the video to be processed.

[0186] In this embodiment, by using a trained recognition model to identify videos in the target domain, it is possible to accurately extract domain-independent target video features that are aligned in both the source and target domains from the videos to be processed. This enables precise classification and recognition of the videos to be processed in the target domain based on the target video features, thereby effectively improving the accuracy of video classification and recognition in the target domain.

[0187] In one embodiment, such as Figure 8 As shown, after training ends when the training stopping condition is met, the above video processing method further includes a video recognition processing step, which specifically includes the following:

[0188] Step S802: Using the trained recognition model, perform preliminary feature extraction on the video to be processed to obtain the initial features of the video to be processed.

[0189] Step S804: Based on the time nodes and time scale weights corresponding to the initial features, perform deep feature extraction on the initial features to obtain target video features aligned in the source and target domains.

[0190] Step S806: The video to be processed is classified and identified based on the characteristics of the target video using the recognition model.

[0191] Understandably, the trained recognition model includes an initial feature extractor, a target feature extractor, and a classifier. The target feature extractor is trained to extract domain-independent video features aligned between the source and target domains from videos within the target domain. The classifier is trained to accurately classify videos within the source domain.

[0192] Specifically, the computer device inputs the to-be-processed video into the recognition model, the recognition model first performs preliminary feature extraction on the to-be-processed video through an initial feature extractor to obtain initial features of the to-be-processed video. Then, the initial features are further subjected to deep feature extraction through a target feature extractor in the recognition model. Since the target feature extractor in the trained recognition model has the ability to extract video features that are irrelevant to the domain and aligned in the source domain and the target domain from videos in the target domain, the target video features that are irrelevant to the domain and aligned in the source domain and the target domain can be directly extracted from the to-be-processed video through deep feature extraction of the to-be-processed video by the recognition model.

[0193] Then, the recognition model can perform classification and recognition processing on the to-be-processed video according to the target video features. Since the classifier has the ability to accurately classify videos in the source domain after training, the class of the object in the to-be-processed video can be accurately recognized according to the target video features that are aligned in the source domain and the target domain.

[0194] In one embodiment, the step of performing classification and recognition processing on the to-be-processed video based on the target video features by the recognition model includes: performing behavior recognition on the object in the to-be-processed video based on the target video behavior features by the recognition model.

[0195] In the embodiment, the target video features are specifically target video behavior features. The classification and recognition task for video data is specifically a behavior recognition task. The trained recognition model has the ability to extract video features that are irrelevant to the domain and aligned in the source domain and the target domain from videos in the target domain, and accurately classify and recognize the behavior of the object in the videos in the target domain.

[0196] Specifically, after the computer device obtains the to-be-processed video in the target domain, the to-be-processed video is first subjected to deep feature extraction by the recognition model to extract target video features that are irrelevant to the domain and aligned in the source domain and the target domain from the to-be-processed video, and then the behavior of the object in the to-be-processed video is classified and recognized according to the target video features, so that the behavior class corresponding to the to-be-processed video can be accurately recognized.

[0197] In one specific test embodiment, the video processing method provided in the scheme embodiment of the present application is compared with the traditional related method for test. Specifically, the present scheme is compared with JAN (Joint Adaptation Networks, video domain adaptation network based on static pictures), DANN (Domain Adversarial Neural Network, domain adversarial neural network), and TA 3The test results are shown in Table 1.

[0198] The test results are shown in Table 1.

[0199]

[0200] Table 1

[0201] In the UCF-HMDB dataset, U→H indicates that the UCF dataset is used as the source domain and the HMDB dataset is used as the target domain, and H→U is the opposite. In the UCF-Olympic dataset, U→O indicates that the UCF dataset is used as the source domain and the Olympic dataset is used as the target domain, and O→U is the opposite. The Kinetics-Gameplay dataset does not distinguish between the source domain and the target domain.

[0202] From the test results of various processing methods shown in Table 1, it can be seen that the recognition model trained by the present scheme can more accurately classify and recognize videos in the target domain compared to the traditional video domain adaptation network model.

[0203] In one application scenario, the video processing method described above can be applied to video behavior recognition. Specifically, the computer device extracts the deep features of the video samples in the source domain and the target domain through the to-be-trained recognition model, performs multi-time scale feature extraction on the deep features through the domain adaptation trainer to obtain the multi-time scale video features in the source domain and the target domain, respectively, groups and aligns the video features in the source domain and the target domain according to the corresponding time nodes and time scale weights of the video features, adjusts the model parameters of the recognition model according to the class loss between the predicted class of the video sample in the source domain and the corresponding sample label, and continues the adversarial training until the training stopping condition is met, thereby ending the training and obtaining the trained recognition model.

[0204] Then, the computer device performs real-time video behavior recognition on the object in the video in the target domain by using the trained recognition model.

[0205] More specifically, the video to be processed in the target domain can be a monitoring video. For example, the video in the source domain can be a video of an indoor scene, and the video in the target domain can be a video of an outdoor scene, and vice versa. After the computer device acquires the collected video in the target domain, the computer device performs deep feature extraction on the monitoring video in the target domain by using the trained recognition model to obtain target video features aligned in the source domain and the target domain. Then, the computer device performs classification and recognition on the behavior of the object in the monitoring video according to the target video features by using the recognition model to obtain a behavior recognition result corresponding to the monitoring video. The computer device can further perform video behavior monitoring on the monitoring video according to the behavior recognition result.

[0206] In one application scenario, the above-mentioned video processing method can be applied to a video search scenario of a target category. Specifically, the computer device performs adversarial training on the to-be-trained recognition model by using the video samples in the source domain and the target domain, and obtains the trained recognition model.

[0207] Then, the computer device can obtain candidate videos in the target domain that meet the condition according to the category keyword corresponding to the target category, and then perform real-time video behavior recognition on the object in the candidate video in the target domain by using the trained recognition model. Specifically, the computer device performs deep feature extraction on the candidate video in the target domain by using the trained recognition model to obtain target video features aligned in the source domain and the target domain. Then, the computer device performs classification and recognition on the object in the candidate video according to the target video features by using the recognition model to obtain a corresponding recognition result. Then, the computer device extracts a video matching the target category from the recognition result from the candidate video as a video search result.

[0208] It should be understood that, although Figure 2 , Figure 5 and Figure 8 in the flowcharts are displayed in sequence according to the arrows, these steps are not necessarily executed in sequence according to the arrows. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, Figure 2 , Figure 5 and Figure 8 may include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential, but can be executed in rotation or alternation with other steps or steps or stages in other steps.

[0209] In one embodiment, as shown in Figure 9 A video processing apparatus 900 is provided, which can be a part of a computer device in the form of software modules or hardware modules, or a combination of both, and specifically includes a feature extraction module 902, a domain adaptation training module 904, a loss determination module 906, and a parameter adjustment module 908, wherein:

[0210] The feature extraction module 902 is configured to extract deep features of video samples in a source domain and a target domain respectively by a to-be-trained recognition model; the video samples in the source domain carry sample labels.

[0211] The domain adaptation training module 904 is configured to perform multi-time scale feature extraction on the deep features by a domain adaptation trainer to obtain multi-time scale video features in the source domain and the target domain respectively; and group and align the video features in the source domain and the target domain according to corresponding time nodes and time scale weights; the time scale weight is positively correlated with the amount of information expressed by the corresponding video feature.

[0212] The loss determination module 906 is configured to determine an adversarial loss according to the difference between the video features in the source domain and the target domain in the same group; and determine a category loss based on the difference between a predicted category of the video samples in the source domain and the corresponding sample labels; the predicted category is obtained by classifying the video features of the video samples in the source domain.

[0213] The parameter adjustment module 908 is configured to adjust the model parameters of the recognition model according to the adversarial loss and the category loss, and continue adversarial training until a training stop condition is met.

[0214] In one embodiment, the domain adaptation training module 904 is further configured to perform multi-time scale convolution processing on the deep features by the domain adaptation trainer to obtain convolution results corresponding to the deep features; and obtain the multi-time scale video features in the source domain and the target domain respectively according to the time node weights corresponding to the deep features and the corresponding convolution results.

[0215] In one embodiment, the domain adaptation trainer performs multi-time scale convolution processing through a convolution layer; the domain adaptation trainer further includes a time node attention layer; and the domain adaptation training module 904 is further configured to assign corresponding time node weights to the time nodes corresponding to the deep features according to the amount of information expressed by the deep features in each time node through the time node attention layer; the time node weight is positively correlated with the amount of information expressed by the deep features in the corresponding time node.

[0216] In an embodiment, the domain adaptation training module 904 is further configured to determine, by a time-scale attention layer of the domain adaptation trainer, information entropy corresponding to video features of each time scale under the source domain and the target domain; the information entropy represents an amount of information expressed by the corresponding video features; and the video features of each time scale are respectively assigned with corresponding time-scale weights according to the information entropy.

[0217] In an embodiment, the domain adaptation training module 904 is further configured to determine, by the domain adaptation trainer, video features to be aligned under the source domain and the target domain according to time nodes corresponding to the video features and the time-scale weights; and divide the video features to be aligned into a group to obtain a plurality of groups of aligned video features; the video features in each group include video features of the same time scale under the source domain and the target domain.

[0218] In an embodiment, the domain adaptation training module 904 is further configured to determine time node weights of the video features under the source domain and the target domain at corresponding time nodes; and determine, from the video features under the source domain and the target domain, video features with matching time node weights and time-scale weights under different domains as the video features to be aligned under the source domain and the target domain.

[0219] In an embodiment, the feature extraction module 902 is further configured to extract initial features of the video samples under the source domain and the target domain respectively by an initial feature extractor in the to-be-trained recognition model; and extract deep features of the video samples under the source domain and the target domain respectively by a target feature extractor in the recognition model.

[0220] In an embodiment, the video processing apparatus further includes a classification module configured to perform classification based on the video features of the video samples under the source domain by a classifier in the recognition model to obtain predicted categories of the video samples under the source domain; and the parameter adjustment module 908 is further configured to adjust model parameters of the target feature extractor and the classifier in the recognition model according to the adversarial loss and the category loss and continue the adversarial training, so that the recognition model reduces differences between the video features under the source domain and the target domain in the same group in the process of iterative training until the training is ended when a training stop condition is met.

[0221] In an embodiment, the loss determination module 906 is further configured to determine a cross-entropy loss based on differences between the predicted categories of the video samples under the target domain and the predicted categories of the video samples under the source domain; and the parameter adjustment module 908 is further configured to adjust model parameters of the recognition model according to the adversarial loss, the category loss and the cross-entropy loss and continue the adversarial training, so that the recognition model reduces differences between the video features under the source domain and the target domain in the same group in the process of iterative training until the training is ended when a training stop condition is met.

[0222] In one embodiment, after the training ends until the training stop condition is met, the video processing apparatus further comprises a video recognition module configured to perform deep feature extraction on the to-be-processed video in the target domain by the trained recognition model to obtain target video features aligned in the source domain and the target domain; and perform classification and recognition processing on the to-be-processed video based on the target video features by the recognition model.

[0223] In one embodiment, the video recognition module is further configured to perform preliminary feature extraction on the to-be-processed video by the trained recognition model to obtain initial features of the to-be-processed video; and perform deep feature extraction on the initial features based on time nodes corresponding to the initial features and time scale weights to obtain target video features aligned in the source domain and the target domain.

[0224] In one embodiment, the target video features are target video behavior features; and the video recognition module is further configured to perform behavior recognition on an object in the to-be-processed video based on the target video behavior features by the recognition model.

[0225] The specific limitations of the video processing apparatus can refer to the limitations of the video processing method described above, which will not be repeated here. Each module in the video processing apparatus described above can be implemented by software, hardware, and combinations thereof, in whole or in part. Each module described above can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory in the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0226] In one embodiment, a computer device is provided, which can be a terminal or a server, and its internal structure diagram can be as shown in Figure 10 The computer device includes a processor, a memory, and a network interface connected by a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the computer device is configured to communicate with external terminals through network connection. The computer program is executed by the processor to implement a video processing method.

[0227] Those skilled in the art can understand that Figure 10 the structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0228] In an embodiment, a computer device is also provided, including a memory and a processor, the memory storing a computer program, and the processor implementing the steps in the above-mentioned method embodiments when executing the computer program.

[0229] In an embodiment, a computer readable storage medium is provided, storing a computer program, the computer program being executed by a processor to implement the steps in the above-mentioned method embodiments.

[0230] In an embodiment, a computer program product or computer program is provided, including computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to make the computer device perform the steps in the above-mentioned method embodiments.

[0231] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0232] The technical features of the above embodiments can be combined in any manner. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, however, as long as the combinations of the technical features do not contradict, they should be considered within the scope of the present application.

[0233] The above-described embodiments are merely illustrative of several embodiments of the present application, which are described in more detail and in a specific and detailed manner, but should not be construed as limiting the scope of the patent. It should be noted that for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present application, and these are all within the scope of the present application. Therefore, the scope of protection of the patent of the present application should be subject to the appended claims.

Claims

1. A method of video processing, the method comprising: The method comprises: extracting deep features of video samples in a source domain and a target domain respectively by an identification model to be trained; the video samples in the source domain carry sample labels; extracting multi-time scale features of the deep features by a domain adaptation trainer to obtain video features in the source domain and the target domain respectively; grouping and aligning the video features in the source domain and the target domain according to time nodes and time scale weights corresponding to the video features; the time scale weights are positively correlated with information amounts expressed by the corresponding video features; determining an adversarial loss according to differences between the video features in the source domain and the target domain in the same group; determining a category loss based on differences between predicted categories of the video samples in the source domain and corresponding sample labels; the predicted categories are obtained by classifying the video features of the video samples in the source domain; adjusting model parameters of the identification model according to the adversarial loss and the category loss and continuing adversarial training until a training stop condition is met.

2. The method of claim 1, wherein, The method of extracting multi-time scale features of the deep features by the domain adaptation trainer to obtain video features in the source domain and the target domain respectively comprises: performing multi-time scale convolution processing on the deep features by the domain adaptation trainer to obtain convolution results corresponding to the deep features; obtaining the video features in the source domain and the target domain respectively according to time node weights corresponding to the deep features and the corresponding convolution results.

3. The method of claim 2, wherein, The domain adaptation trainer performs multi-time scale convolution processing through a convolution layer; the domain adaptation trainer further comprises a time node attention layer. The method further comprises: allocating corresponding time node weights to the time nodes corresponding to the deep features respectively according to information amounts expressed by the deep features in each time node through the time node attention layer; the time node weights are positively correlated with the information amounts expressed by the deep features in the corresponding time nodes.

4. The method of claim 1, wherein, Before the grouping and aligning of the video features in the source domain and the target domain according to the time nodes and the time scale weights corresponding to the video features, the method further comprises: determining information entropy corresponding to the video features in each time scale in the source domain and the target domain through a time scale attention layer of the domain adaptation trainer; the information entropy represents information amounts expressed by the corresponding video features; allocating corresponding time scale weights to the video features in each time scale respectively according to the information entropy.

5. The method of claim 1, wherein, The grouping and aligning of the video features in the source domain and the target domain according to the time nodes and the time scale weights corresponding to the video features comprises: determining video features to be aligned in the source domain and the target domain according to the time nodes and the time scale weights corresponding to the video features through the domain adaptation trainer; dividing the video features to be aligned into a group to obtain multiple groups of aligned video features; the video features in each group comprise video features in the source domain and the target domain of the same time scale.

6. The method of claim 5, wherein, The determination of the video features to be aligned in the source domain and the target domain according to the time nodes and the time scale weights corresponding to the video features through the domain adaptation trainer comprises: determining time node weights of video features under source and target domains at corresponding time nodes; determining video features with matching time node weights and time scale weights under different domains from the video features under the source and target domains as video features under the source and target domains to be aligned.

7. The method of claim 1, wherein, The deep features of the video samples under the source and target domains are extracted by the to-be-trained recognition model, including: The initial features of the video samples under the source and target domains are extracted by an initial feature extractor in the to-be-trained recognition model; The initial features of the video samples under the source and target domains are extracted by a target feature extractor in the recognition model to obtain the deep features of the video samples under the source and target domains.

8. The method of claim 7, wherein, The method further includes: The video features of the video samples under the source domain are classified by a classifier of the recognition model to obtain predicted categories of the video samples under the source domain; The model parameters of the recognition model are adjusted according to the adversarial loss and the category loss, and the adversarial training is continued until the training is ended when the training stop condition is met, including: The model parameters of the target feature extractor and the classifier in the recognition model are adjusted according to the adversarial loss and the category loss, and the adversarial training is continued, so that the recognition model reduces the difference between the video features under the source and target domains in the same group in the process of iterative training, and the training is ended when the training stop condition is met.

9. The method of claim 1, wherein, The method further includes: A cross-entropy loss is determined based on the difference between the predicted categories of the video samples under the target domain and the predicted categories of the video samples under the source domain; The model parameters of the recognition model are adjusted according to the adversarial loss, the category loss and the cross-entropy loss, and the adversarial training is continued, so that the recognition model reduces the difference between the video features under the source and target domains in the same group in the process of iterative training, and the training is ended when the training stop condition is met. After the training is ended when the training stop condition is met, the method further includes:

10. The method according to any one of claims 1 to 9, characterized in that, Deep feature extraction is performed on the to-be-processed video under the target domain by the trained recognition model to obtain target video features aligned under the source and target domains; The to-be-processed video is classified and recognized by the recognition model based on the target video features. The deep feature extraction is performed on the to-be-processed video by the trained recognition model to obtain the target video features aligned under the source and target domains, including:

11. The method of claim 10, wherein, The initial features of the to-be-processed video are extracted by the trained recognition model to obtain the initial features of the to-be-processed video; The initial features are deep feature extracted based on the time node and time scale weights corresponding to the initial features to obtain the target video features aligned under the source and target domains. The target video features are target video behavior features; 12. The method of claim 10, wherein, ​ The target video feature is used to perform classification and recognition on the to-be-processed video by the recognition model, including: The target video behavior feature is used to perform behavior recognition on the object in the to-be-processed video by the recognition model.

13. A video processing apparatus, comprising: The device comprises: A feature extraction module is configured to extract deep features of video samples in a source domain and a target domain respectively by a to-be-trained recognition model, wherein the video samples in the source domain carry sample labels; A domain adaptation training module is configured to perform multi-time scale feature extraction on the deep features by a domain adaptation trainer to obtain multi-time scale video features in the source domain and the target domain respectively, and group and align the video features in the source domain and the target domain according to corresponding time nodes and time scale weights of the video features, wherein the time scale weight is positively correlated with an information amount expressed by the corresponding video feature; A loss determination module is configured to determine an adversarial loss according to a difference between the video features in the source domain and the target domain in the same group, and determine a category loss based on a difference between a predicted category of the video samples in the source domain and corresponding sample labels, wherein the predicted category is obtained by classification based on the video features of the video samples in the source domain; A parameter adjustment module is configured to adjust model parameters of the recognition model according to the adversarial loss and the category loss, and continue adversarial training until a training stop condition is met.

14. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to implement the steps of the method in any one of claims 1 to 12.

15. A computer readable storage medium storing a computer program, wherein the computer program comprises program instructions configured to cause a processor to perform the method according to any one of claims 1 to 14. The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 12.

Images