Video recognition model training method, device, equipment and storage medium
Through the video recognition model training method of shared feature encoding network, the video recognition model is trained using training data sets of different modalities, which solves the problem of insufficient adaptability of existing models to data and classification tasks in different modalities, and achieves more efficient and accurate multimodal classification capability.
Patent Information
- Application Number
- CN202210578205.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-25
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2042-05-25
AI Technical Summary
Existing video recognition models have poor adaptability to different modal data and classification tasks, making it difficult to show high efficiency and accuracy in multiple classification tasks.
A video recognition model training method is adopted, by sharing the first recognition model and the second recognition model of the same feature encoding network, training data sets of different modes are used for training, and the parameters of the feature encoding network and classification network are adjusted to realize Multimodal training improves the universality and robustness of the model.
The video recognition model's recognition ability and robustness of data on different modalities is improved, the adaptability and universality of the model is enhanced, the number of model parameters and calculation amounts are reduced, and the accuracy and efficiency of classification tasks are improved.
Smart Images

Figure CN117197702B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to a training method, device, equipment and storage medium for a video recognition model. Background Art
[0002] By training the machine learning model, a video recognition model for classifying video data can be obtained, which helps to complete the classification task of video data in batches and reduce labor costs.
[0003] In the related art, an initial video recognition model is trained using video training samples, and a loss function is determined using the classification prediction results generated by the video recognition model and the label data of the training samples; the parameters in the initial video recognition model are adjusted according to the loss function; and when the loss function converges, a trained video recognition model is obtained so that the trained video recognition model can be used to perform the classification task of video data.
[0004] However, the video recognition model trained in the above manner has poor adaptability to different modal data and different classification tasks. Summary of the invention
[0005] The present application provides a video recognition model training method, apparatus, device and storage medium, which can increase the number of classification tasks performed by the video recognition model and improve the universality of the video recognition model. The technical solution is as follows:
[0006] According to one aspect of an embodiment of the present application, a training method for a video recognition model is provided, wherein the video recognition model includes: a first recognition model and a second recognition model, wherein the first recognition model and the second recognition model share the same feature encoding network, wherein the first recognition model includes the feature encoding network and a first classification network, wherein the second recognition model includes the feature encoding network and a second classification network, wherein the first classification network and the second classification network are used to perform two different classification tasks; the method includes:
[0007] The first recognition model is trained using a first data set, and parameters of the feature encoding network and the first classification network included in the first recognition model are adjusted to obtain a preliminarily trained feature encoding network and a preliminarily trained first classification network; wherein the first data set includes training data of a first modality;
[0008] Based on the second classification network and the preliminarily trained feature encoding network, construct an initialized second recognition model;
[0009] The second classification network in the initialized second recognition model is trained using a second data set, the parameters of the feature encoding network after the preliminary training are fixed, and the parameters of the second classification network are adjusted to obtain the second classification network after the preliminary training; wherein the second data set includes training data of a second modality, and the second modality and the first modality are two different modalities;
[0010] Based on the preliminarily trained feature encoding network and the preliminarily trained second classification network, constructing a preliminarily trained second recognition model;
[0011] Using the second data set to train the second recognition model after the initial training, adjusting the parameters of the feature encoding network after the initial training and the second classification network after the initial training, to obtain the feature encoding network after the secondary training and the second classification network after the secondary training;
[0012] Constructing a video recognition model to be trained based on the feature encoding network after the secondary training, the first classification network after the preliminary training, and the second classification network after the secondary training;
[0013] The video recognition model to be trained is trained using a third data set to obtain a trained video recognition model; wherein the third data set includes at least one of the following: training data of the first modality and training data of the second modality.
[0014] According to one aspect of an embodiment of the present application, a training device for a video recognition model is provided, wherein the video recognition model includes: a first recognition model and a second recognition model, wherein the first recognition model and the second recognition model share the same feature encoding network, wherein the first recognition model includes the feature encoding network and a first classification network, wherein the second recognition model includes the feature encoding network and a second classification network, wherein the first classification network and the second classification network are used to perform two different classification tasks; the device includes:
[0015] A first training module, used to train the first recognition model using a first data set, adjust the parameters of the feature encoding network and the first classification network included in the first recognition model, and obtain a preliminarily trained feature encoding network and a preliminarily trained first classification network; wherein the first data set includes training data of a first modality;
[0016] A model construction module, used to construct an initialized second recognition model based on the second classification network and the preliminarily trained feature encoding network;
[0017] A second training module, used to train the second classification network in the initialized second recognition model using a second data set, fix the parameters of the feature encoding network after the preliminary training, and adjust the parameters of the second classification network to obtain the second classification network after the preliminary training; wherein the second data set includes training data of a second modality, and the second modality and the first modality are two different modalities;
[0018] The model construction module is further used to construct a preliminarily trained second recognition model based on the preliminarily trained feature encoding network and the preliminarily trained second classification network;
[0019] The second training module is further used to train the second recognition model after the initial training using the second data set, adjust the parameters of the feature encoding network after the initial training and the second classification network after the initial training, and obtain the feature encoding network after the secondary training and the second classification network after the secondary training;
[0020] The model construction module is further used to construct a video recognition model to be trained based on the feature encoding network after the secondary training, the first classification network after the preliminary training, and the second classification network after the secondary training;
[0021] The third training module is used to train the video recognition model to be trained using a third data set to obtain a trained video recognition model; wherein the third data set includes at least one of the following: training data of the first modality and training data of the second modality.
[0022] According to one aspect of an embodiment of the present application, a computer device is provided, which includes a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the above-mentioned video recognition model training method.
[0023] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored. The computer program is loaded and executed by a processor to implement the above-mentioned video recognition model training method.
[0024] According to one aspect of an embodiment of the present application, a computer program product is provided, which includes a computer program stored in a computer-readable storage medium, and a processor reads and executes the computer program from the computer-readable storage medium to implement the above-mentioned video recognition model training method.
[0025] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:
[0026] The first recognition model and the second recognition model are trained by using the first training set and the second training set respectively including training data of different modalities, and the first recognition model and the second recognition model share the same feature encoding network. On the one hand, the feature encoding network has been trained with training data of different modalities. Since the noise faced by training data and classification tasks in different modalities may have the same characteristics, it helps to enhance the feature encoding network's ability to recognize noise in multimedia data of different modalities and improve the robustness of the trained video recognition model.
[0027] On the other hand, the above method is used to implement a multimodal training process for the video recognition model. The trained video recognition model is able to classify a certain video and obtain a first classification result and a second classification result of the video. Based on a single video recognition model, classification tasks of different content dimensions can be performed, which helps to increase the scope of use of the video recognition model and improve the universality of the video recognition model.
[0028] At the same time, since the first classification network and the second classification network share the same feature encoding network in the trained video recognition model, it helps to reduce the number of parameters in the video recognition model and reduce the amount of computation in the model inference process. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Figure 1 It is a schematic diagram of a solution implementation environment provided by an exemplary embodiment of the present application;
[0030] Figure 2 is a flowchart of a method for training a video recognition model provided by an exemplary embodiment of the present application;
[0031] Figure 3 is a schematic diagram of a method for capturing multiple input images provided by an exemplary embodiment of the present application;
[0032] Figure 4 is a schematic diagram of extracting feature information of training samples provided by an exemplary embodiment of the present application;
[0033] Figure 5 is a schematic diagram of a training process of a video recognition model provided by an exemplary embodiment of the present application;
[0034] Figure 6 is a schematic diagram of displaying recommendation information based on video recognition provided by an exemplary embodiment of the present application;
[0035] Figure 7 is a block diagram of a training device for a video recognition model provided by an exemplary embodiment of the present application;
[0036] Figure 8It is a structural block diagram of a computer device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION
[0037] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.
[0038] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.
[0039] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes natural language processing technology, machine learning / deep learning and other major directions.
[0040] Computer vision technology (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision such as using cameras and computers to replace human eyes to identify and measure targets, and further perform graphic processing so that computer processing becomes an image that is more suitable for human eye observation or transmission to instrument detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, virtual reality, augmented reality, simultaneous positioning and map construction, and also includes common biometric recognition technologies such as face recognition and fingerprint recognition.
[0041] Scene recognition refers to the automatic recognition of application scenarios in videos or pictures by machines. The most widely used dataset in the field of scene recognition is the places2 dataset, which contains more than 10 million pictures and more than 400 different scene categories.
[0042] Action recognition refers to identifying the actions that appear in the video, usually the actions of people in the video data. Since action recognition not only needs to analyze the content of each video frame in the video data, but also needs to dig clues from the timing information of continuous video frames, a model based on video modality is usually used to identify actions. Machine Learning (ML) is a multi-field cross-disciplinary subject involving multiple disciplines such as probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specializes in how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all fields of artificial intelligence. Machine learning and deep learning usually include technologies such as artificial neural networks, belief networks, reinforcement learning, transfer learning, and inductive learning. In this application, video data and image data can be extracted and further classified by using the relevant technologies of machine learning.
[0043] With the research and progress of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common video recognition and classification, recommendation tasks based on video recognition and classification, and data management tasks based on video recognition and classification. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.
[0044] Divided space-time attention: refers to a data processing mechanism introduced to alleviate the resource consumption of self-attention calculation in machine learning models. In time attention, each image block is only connected to the image blocks extracted from the corresponding positions of the rest of the input image. In space attention, an image block is only connected to other image blocks extracted from the same input image. The divided space-time attention mechanism can effectively reduce the amount of computation of the machine learning model in processing classification tasks and achieve a more accurate classification effect.
[0045] The solutions provided in the embodiments of the present application involve technologies such as artificial intelligence computer vision and machine learning, which are specifically illustrated by the following embodiments.
[0046] Figure 1 1 is a schematic diagram of a solution implementation environment provided by an exemplary embodiment of the present application. The solution implementation environment can be implemented as a computer system, such as a video recognition system. The solution implementation environment can include: a terminal device 10 and a server 20.
[0047] The terminal device 10 may be an electronic device such as a PC (Personal Computer), a tablet computer, a mobile phone, a multimedia player, a wearable device, a smart home appliance, a vehicle-mounted terminal, etc. The terminal device 10 may be installed with a client running a target application, and the target application may be a video player application, a social application supporting video data playback and / or recommendation, and other applications supporting video data playback; the target application may also be expanded to mean a service function in an operating system running on the terminal device, such as a media library supporting storage and display of multimedia data, etc.
[0048] The server 20 can provide background services for the target application running on the terminal device 10. For example, the server 20 can be the background server of the target application. The server 20 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud computing, cloud functions, cloud storage, network services, cloud communications, domain name services, security services, and big data and artificial intelligence platforms. The server 20 has at least the functions of data transmission, calculation, storage, etc., and is used to transmit data with the client on the terminal device 10 in real time.
[0049] In one example, the server 20 transmits certain video data and recommendation information corresponding to the video data to the terminal device 10. The recommendation information may be advertising information, and the recommendation information may be displayed in a display window corresponding to the video data. The recommendation information may also be related to other video data, for example, the recommendation information may be a cover image of other video data related to the video data, an image frame being played, etc.
[0050] Optionally, the recommendation information is obtained from a recommendation information database by classifying the video data using a video recognition model to obtain a first classification result and a second classification result of the video data.
[0051] In another example, the terminal device 10 quickly provides the user with multimedia data to be found through the video recognition model, and can also recommend a classification method for multimedia data in the multimedia database to the user based on the video recognition model.
[0052] For example, a video recognition model is running on the terminal device 10, and the video recognition model is used to perform a classification task on the multimedia data in the multimedia database to obtain a first type and a second type of the multimedia data. The terminal device 10 can classify the multimedia data in the multimedia database according to the first type and the second type of the multimedia data, and display the multimedia data set obtained after classification to the user for the user to select.
[0053] For another example, the terminal device 10 may provide the user with a search information input box so that the terminal device 10 can quickly obtain video data of the corresponding category based on the search information input by the user (the search information may be the first type and the second type of target video data that the user wants to find).
[0054] The training process of the video recognition model used in the above example can be completed by a computer device, which can be a terminal device 10, a server 20, or another device other than the terminal device 10 and the server 20. The computer device can be an electronic device with computing and storage functions, such as a personal computer, a tablet computer, or an intelligent robot, and is used to train and generate a video recognition model based on training data.
[0055] Figure 2 This is a flowchart of a video recognition model training method provided by an exemplary embodiment of the present application. Exemplarily, the execution subject of the method may be a computer device, and the video recognition model training method is introduced below using a computer device as the execution subject.
[0056] In an embodiment of the present application, the video recognition model includes: a first recognition model and a second recognition model, the first recognition model and the second recognition model share the same feature encoding network, the first recognition model includes a feature encoding network and a first classification network, the second recognition model includes a feature encoding network and a second classification network, and the first classification network and the second classification network are used to perform two different classification tasks.
[0057] The method may include the following steps (210-270):
[0058] Step 210, using the first data set to train the first recognition model, adjusting the parameters of the feature encoding network and the first classification network included in the first recognition model, to obtain the initially trained feature encoding network and the initially trained first classification network; wherein the first data set includes the first modality training data.
[0059] The first data set is used to train the first recognition model. In some embodiments, the first data set includes at least one first modality training data and at least one classification label corresponding to the first modality training data; wherein the first modality training data refers to data used to train the first recognition model, and the classification label is used to indicate the type information of the first modality training data.
[0060] In some embodiments, the training data of the first modality is multimedia data. For example, the training data of the first modality may be video data having more than one image frame. For another example, the first training sample may be a single image.
[0061] The modality is used to distinguish training data obtained through different acquisition methods, with different forms of expression or different ways of representing information. In some embodiments, the first modality is related to the form of expression of the training data. For example, when the training data is video data, the first modality is the video modality. For another example, when the training data is image data, the first modality is the image modality. In some embodiments, the first modality is related to the acquisition method of the training data. For example, the training data obtained by recording is video data, and the first modality is the video modality.
[0062] In some embodiments, all training data (first data) included in the first data set are of the first modality, that is, when training the first recognition model, training data of the same modality are used, which helps to reduce the input of irrelevant noise in the training data, so that the first recognition model, especially the first classification network in the first recognition model, can better classify and learn the common features in the training data of the first modality. At the same time, it also helps to improve the convergence speed of the first recognition model during the training process, so as to shorten the time consumed to complete the training of the first recognition model.
[0063] In some embodiments, the classification label is related to the content information of the training data, wherein the training data is used to train the video recognition model. The following takes the training data as the first modality training data as an example to illustrate the selection method of the classification label.
[0064] In one example, the classification label of the training data of the first modality is determined according to the scene content dimension in the training data of the first modality. In this case, the classification label of the training data of the first modality includes but is not limited to at least one of the following: natural scenes and social scenes. In some embodiments, the scene content can be further subdivided. For example, natural scenes include but are not limited to at least one of the following: ocean, sky, and land (such as forests, deserts, and mountains). For another example, social scenes include but are not limited to at least one of the following: work and study (such as conference rooms, offices, libraries, and laboratories) and home (such as bedrooms, kitchens, and living rooms).
[0065] In another example, the classification label of the training data of the first modality is determined according to the action content dimension in the training data of the first modality. In this case, the classification label of the training data of the first modality includes but is not limited to at least one of the following: interactive actions, single-person actions. Interactive actions can also be subdivided into: interacting with objects (such as playing musical instruments and equipment movements) and interacting with other organisms (such as watering plants, playing with animals, and collective chorus).
[0066] In another example, the classification label of the training data of the first modality is determined according to the objects in the training data of the first modality. For example, the classification label of the training sample of the first modality is the presence of a specific object, etc. Further, in the case where a certain first modality is a video modality, if the number of image frames in which a specific object appears in the training data of the first modality is greater than n frames, the classification label corresponding to the first training data is determined as the video attribute corresponding to the specific object. In this case, it is possible to classify the attributes of a complete video (especially a short video). For example, if the number of image frames in which a specific vacuum cleaner appears in the training data of a certain first modality is greater than n frames, the classification label corresponding to the training data of the first modality can be determined as a promotional product (vacuum cleaner) video.
[0067] The first recognition model refers to a machine learning model that determines the category of the input data. In some embodiments, the first recognition model is established based on the classic CNN (Convolutional Neural Networks). In other embodiments, the first recognition model is established based on a self-attention mechanism.
[0068] In some embodiments, the first recognition model feature encoding network is composed of a feature encoding network and a first classification network; wherein the feature encoding network is used to perform feature extraction processing on input data (such as training data, video data to be classified) input into the first recognition model. The first classification network is used to perform a classification task on the input data of the first recognition model to obtain a classification result of the input data.
[0069] In some embodiments, the feature encoding network includes more than one encoding layer. Different encoding layers are used to extract features of different dimensions of input information. The feature encoding network extracts features from the input data to obtain feature information of the input data; the feature encoding network passes the feature information to the first classification network. The first classification network determines the type of the input data based on the feature information.
[0070] During the process of training the first recognition model, the model recognition device obtains a first data set, and trains the first recognition model through the training samples included in the first data set to obtain a converged first recognition model, which includes a feature encoding network after preliminary training and a first classification network after preliminary training.
[0071] Step 220: construct an initialized second recognition model based on the second classification network and the preliminarily trained feature encoding network.
[0072] The initialized second recognition model includes a preliminarily trained feature encoding network and a second classification network.
[0073] The second classification network refers to a classification network used to perform the second classification task. The feature coding network after preliminary training refers to the feature coding network in the first classification network obtained by training the first data set in step 210, that is, the parameters in the feature coding network after preliminary training can assist the first classification network after preliminary training to perform the first classification task.
[0074] In some embodiments, the computer device constructs an initialized second recognition model by accessing the second classification network after the initialized feature coding network. For example, the model device constructs the second classification network by adding a data transmission interface of the feature coding network so that the feature coding network can transmit feature information to the second classification network.
[0075] In other embodiments, the model construction device establishes an initial video recognition model, wherein the initial video recognition model includes: a feature encoding network, a first classification network, and a second classification network; wherein the feature encoding network and the first classification network constitute a first recognition model, and the feature encoding network and the second classification network constitute a second recognition model.
[0076] In one example, the feature coding network provides feature information of input data to the first classification network through data interface 1, and the feature coding network provides feature information of input data to the second classification network through data interface 2; before performing step 210, the computer device closes data interface 2, so that the second classification network cannot receive the feature information provided by the feature coding network; after completing step 210, the computer device reopens data interface 2 so that the second classification network can re-receive the feature information provided by the feature coding network.
[0077] The computer device constructs an initialized second recognition model by using the preliminarily trained feature encoding network and the second classification network, so that the second recognition model and the first recognition model share the same feature encoding network. Compared with the first recognition model and the second recognition model each having an independent feature encoding network, the first recognition model and the second recognition model use the same feature encoding network, which can linearly reduce the number of parameters used in the video recognition model (specifically, reduce the parameters of a feature encoding network used), reduce the learning of additional hyperparameters, and help reduce the amount of calculation required by the video recognition model to classify video data.
[0078] Step 230, using the second data set to train the second classification network in the initialized second recognition model, fixing the parameters of the feature encoding network after preliminary training, and adjusting the parameters of the second classification network to obtain the second classification network after preliminary training; wherein the second data set includes training data of the second modality, and the second modality and the first modality are two different modalities.
[0079] The second data set is used to train the second classification network. In some implementations, the second training network includes at least one second modality training data and a classification label corresponding to the second modality training data, the second modality training data refers to data used to train the second classification network, and the classification label is used to indicate the type information of the second modality training data.
[0080] In some embodiments, the training data of the second modality is multimedia data. For details about the content of the training data of the second modality, please refer to the above introduction to the training data of the first modality, which will not be elaborated here.
[0081] The first modality and the second modality are two different modalities. For example, the first modality is a video modality, and the second modality is an image modality. For another example, the first modality is an image modality, and the second modality is a video modality.
[0082] Below, examples are given for the first data set, the first classification task, the second data set, and the second classification task.
[0083] In one example, the first data set is a video data set, that is, the first data set includes training data in video modality (which can be called training video), and the first classification task is action classification; the second data set is an image data set, that is, the second data set includes training data in image modality (which can be called training images), and the second classification task is scene classification.
[0084] In this example, the computer device trains the feature encoding network and the action classification network (i.e., the first classification network) through the video data set (i.e., the first data set) to obtain the feature encoding network and the action classification network after the initial training. Subsequently, the computer device trains the scene classification network (i.e., the first classification network) through the image data set (i.e., the first data set) to obtain the scene classification network after the initial training. After completing the above training process, the computer device continues to execute from step 240.
[0085] In another example, the first data set is an image data set, that is, the first data set includes training data in image modality; the first classification task is scene classification; the second data set is a video data set, that is, the second data set includes training data in video modality, and the second classification task is action classification.
[0086] In this example, the computer device trains the feature encoding network and the scene classification network (i.e., the first classification network) through the image data set (i.e., the first data set) to obtain the feature encoding network and the scene classification network after the initial training. Subsequently, the computer device trains the action classification network (i.e., the first classification network) through the video data set (i.e., the first data set) to obtain the action classification network after the initial training. After completing the above training process, the computer device starts execution from step 240.
[0087] In another example, the first data set is a video data set, and the first classification task is video attribute classification (such as pet videos, beauty videos, promotional videos, etc.); the second data set is an image data set, that is, the training data included in the second data set is in image mode, and the second classification task is scene classification.
[0088] It should be noted that the specific contents corresponding to the first data set, the first classification task, the second data set, and the second classification task in the above embodiments are only used as examples, and do not represent a limitation on the specific contents of the first data set, the first classification task, the second data set, and the second classification task, nor can they be regarded as limiting the correspondence between the first data set and the first classification task, and the correspondence between the second data set and the second classification task. In the actual training process of the video recognition model, the first data set, the first classification task, the second data set, and the second classification task are determined according to actual needs, and this application does not limit them here.
[0089] Because the first modality and the second modality are two different modalities, the training data of the first modality used to train the first recognition model and the training data of the second modality used to train the second classification network belong to different modalities, that is, the first classification network after initial training and the second classification network after initial training can perform classification tasks for multimedia data of different modalities.
[0090] Furthermore, the first classification network after preliminary training can perform a first classification task for the data of the first modality, and the second classification network after preliminary training can perform a second classification task for the data of the second modality. Therefore, the video recognition model including the feature encoding network, the first classification network and the second classification network can complete multiple classification tasks for the same input data, which helps to improve the adaptability of the classification ability of the video recognition model and expand the scope of application of the video recognition model.
[0091] In some embodiments, the classification labels corresponding to the training data of the first modality and the classification labels corresponding to the training data of the second modality are divided into different content dimensions (that is, the first classification task and the second classification task are different). For example, the classification labels corresponding to the training data of the first modality are related to the action content in the training data of the first modality, and the classification labels corresponding to the training data of the second modality are related to the scene content in the training data of the second modality. For another example, the classification labels corresponding to the training data of the first modality are related to the action content in the training data of the first modality, and the classification labels corresponding to the training data of the second modality are related to the specific object content in the training data of the second modality. Please refer to the above introduction for the specific content of the classification labels, which will not be elaborated in this application.
[0092] In some embodiments, during the training of the second classification network, the computer device does not change the parameters of other network structures in the video recognition model (including the feature encoding network after initial training and the first classification network after initial training), and only adjusts the parameters in the second classification network until the second recognition model reaches convergence, thereby obtaining the feature encoding network after initial training. For details of this process, please refer to the following embodiments.
[0093] In this step, the computer device fixes the parameters of the feature coding network after preliminary training, and fine-tunes the video recognition model by adjusting the parameters in the first second classification network in a targeted manner, so that the second classification network after initial training can cooperate with the feature coding network after initial training to perform the second classification task more accurately. Compared with simultaneously adjusting the parameters in the feature coding network and the second classification network after initial training in the initialized second recognition model, the adverse effect on the accuracy of the first type determined by the first recognition model is reduced, which helps to reduce the adjustment of the parameters of the feature coding network after initial training in the subsequent training process, and improves the time spent on completing the training of the video recognition model.
[0094] Step 240, constructing a preliminarily trained second recognition model based on the preliminarily trained feature encoding network and the preliminarily trained second classification network.
[0095] In some embodiments, the preliminarily trained second recognition model is composed of a preliminarily trained feature encoding network and a preliminarily trained second classification network. After the model classification device trains the second classification network using the second data set to obtain the preliminarily trained second classification network, it can obtain the preliminarily trained second recognition model.
[0096] Step 250, using the second data set to train the second recognition model after the initial training, adjusting the parameters of the feature encoding network after the initial training and the second classification network after the initial training, to obtain the feature encoding network after the secondary training and the second classification network after the secondary training.
[0097] In some embodiments, after obtaining the preliminarily trained second classification network, the computer device trains the preliminarily trained second recognition model using a second training set to change parameters of the preliminarily trained feature encoding network and the preliminarily trained second classification network.
[0098] Since the preliminarily trained second classification network can perform classification of the training data of the second modality, after obtaining the preliminarily trained first classification network, the second recognition model after the preliminarily training is trained using the second training set, so that the preliminarily trained second classification network can continue to learn the features of the training data belonging to the second modality, which helps to improve the accuracy of the second classification network after the second training obtained after the training in performing the second classification task.
[0099] Step 260, constructing a video recognition model to be trained based on the feature encoding network after secondary training, the first classification network after preliminary training, and the second classification network after secondary training.
[0100] In some embodiments, the model to be trained and to be recognized includes a feature encoding network after secondary training, a second classification network after preliminary training, and a second classification network after secondary training.
[0101] Step 270, using a third data set to train the video recognition model to be trained to obtain a trained video recognition model; wherein the third data set includes at least one of the following: training data of the first modality, training data of the second modality.
[0102] The computer device trains the video recognition model to be trained using the third data set, and obtains a trained video recognition model after the video recognition model to be trained reaches convergence.
[0103] In some embodiments, the third data set is used to train the video recognition model to be trained. In order to improve the accuracy of the trained video recognition model in performing the first classification task and the second classification task, the third data set may include training data of the first modality and training data of the second modality.
[0104] In some embodiments, in the video recognition model to be trained, the first recognition model and the second recognition model use the same feature encoder after secondary training to extract features from the training data. In one example, the feature encoding network after secondary training extracts features from the training data to obtain feature information of the training data, and transmits the feature information to the first classification network after initial training and the second classification network after secondary training. The first classification network after initial training and the second classification network after secondary training respectively perform corresponding classification tasks based on the feature information.
[0105] Since the first classification network after the initial training and the second classification network after the secondary training can determine the first classification results and the second classification results corresponding to the training data respectively according to the same feature information, the number of times the feature encoding network after the secondary training extracts features for a certain training data is reduced in the process of training the video recognition model using the third data set, which helps to reduce the amount of computation in the training process of obtaining the completed video recognition model.
[0106] In any training cycle of step 270, the computer device can simultaneously adjust the parameters of the feature encoding network after secondary training, the second classification network after preliminary training, and the second classification network after secondary training.
[0107] In any training cycle of step 270, the computer device may also select any one or two networks from the feature encoding network after secondary training, the second classification network after preliminary training, and the second classification network after secondary training, and adjust the parameters of the selected network in the training cycle. For example, in a certain training cycle, the parameters of the feature encoding network after secondary training and the first classification network after preliminary training are adjusted, and in the next training cycle, the parameters of the feature encoding network after secondary training and the first classification network after secondary training are adjusted.
[0108] In some embodiments, the computer device may adjust parameters of the feature encoding network after secondary training and the first classification network after secondary training, the feature encoding network after secondary training and the second classification network after secondary training respectively according to the training ratio.
[0109] In some embodiments, the training ratio is pre-configured according to the importance of different classification tasks, for example, the number of training batches used to adjust the parameters of the first recognition model: the number of training batches used to adjust the parameters of the first recognition model = 1:2.
[0110] In other embodiments, the training ratio is dynamically set according to the accuracy of the classification task performed by the video recognition model that has not completed training. Please refer to the embodiments below for details.
[0111] In summary, the first recognition model and the second recognition model are trained by using the first training set and the second training set including training data of different modalities, and the first recognition model and the second recognition model share the same feature encoding network. On the one hand, the feature encoding network has been trained with training data of different modalities. Since the noise faced by training data and classification tasks in different modalities may have the same characteristics, it helps to enhance the feature encoding network's ability to recognize noise in multimedia data of different modalities and improve the robustness of the trained video recognition model.
[0112] On the other hand, the above method is used to implement a multimodal training process for the video recognition model. The trained video recognition model is able to classify a certain video and obtain a first classification result and a second classification result of the video. Based on a single video recognition model, classification tasks of different content dimensions can be performed, which helps to increase the scope of use of the video recognition model and improve the universality of the video recognition model.
[0113] At the same time, since the first classification network and the second classification network share the same feature encoding network in the trained video recognition model, it helps to reduce the number of parameters in the video recognition model and reduce the amount of computation in the model inference process.
[0114] As described above, the first modality and the second modality refer to two different modalities. In some embodiments, the first modality belongs to one of a video modality or an image modality, and the second modality belongs to another of the video modality or the image modality that is different from the first modality.
[0115] In some embodiments, the first modality is a video modality, and the first data set includes at least one training video; the second modality is an image modality, and the second data set includes at least one training image; the first data set is used to train the first recognition model, and the parameters of the feature encoding network and the first classification network contained in the first recognition model are adjusted to obtain a preliminarily trained feature encoding network and a preliminarily trained first classification network, including: for each training video, according to multiple interception time points, obtaining multiple image frames corresponding to the training video; using the multiple image frames corresponding to the training video, the first recognition model is trained, and the parameters of the feature encoding network and the first classification network contained in the first recognition model are adjusted to obtain a preliminarily trained feature encoding network and a preliminarily trained first classification network. The second classification network in the initialized second recognition model is trained using the second data set, the parameters of the feature encoding network after preliminary training are fixed, and the parameters of the second classification network are adjusted to obtain the second classification network after preliminary training, including: for each training image, the training image is transformed to obtain a plurality of transformed images corresponding to the training image; the second classification network is trained using the plurality of transformed images corresponding to the training image, the parameters of the feature encoding network after preliminary training are fixed, and the parameters of the second classification network are adjusted to obtain the second classification network after preliminary training.
[0116] In some embodiments, before each training video in the first data set is input into the feature encoding network of the video recognition model, the training video needs to be preprocessed. Preprocessing the training video includes: reducing or increasing the playback length of the training video so that the lengths of multiple training videos are consistent, which facilitates the subsequent determination of multiple interception time points, and using the multiple interception time points to process at least one training video separately.
[0117] In some embodiments, for different batches of training videos, the total number of corresponding intercepted time points is different. For example, the total number of intercepted time points in the first training batch is x, and the total number of intercepted time points in the second training batch is y, where x is not equal to y. This method enables the trained feature encoding network to adapt to the number of image frames, which helps to improve the adaptability of the video recognition model to video data of different lengths.
[0118] In some embodiments, transforming the training image to obtain multiple transformed images corresponding to the training image includes: performing multiple different transforms on the same training image to obtain multiple transformed images corresponding to the training image. In some embodiments, the transform includes but is not limited to at least one of the following: stretching, shrinking, local deformation, rotation, mirror symmetry, etc. of the training image.
[0119] By setting the first modality to be the video modality and the second modality to be the image modality, the computer device uses the video data set to train the first recognition model and uses the image training set to train the second classification network. Compared with image data, video data includes more data, so that the feature encoding network obtained after the initial training using the video training set can summarize the features of the video data as much as possible, so that in the process of using the image data set to train the second classification network, the parameters in the feature encoding network after the initial training can adapt to the feature extraction of the training data in the picture data set, so that the second classification network can converge as soon as possible.
[0120] The training process of the second classification network is introduced below through several embodiments.
[0121] In some embodiments, the second modality is an image modality, and the second data set includes at least one training image; the computer device uses the second data set to train the second classification network in the initialized second recognition model, fixes the parameters of the feature encoding network after preliminary training, and adjusts the parameters of the second classification network to obtain the second classification network after preliminary training, including: the computer device captures multiple different image contents from the same training image to obtain multiple input images; wherein there is an overlapping area between any two input images belonging to the same training image; the computer device uses the multiple input images as training samples of the initialized second recognition model; the computer device extracts feature information of the training samples through the feature encoding network after preliminary training; the computer device determines the predicted classification results of the training samples according to the feature information of the training samples through the second classification network; the computer device determines the training loss according to the predicted classification results and classification labels of the training samples, and adjusts the parameters of the second classification network based on the training loss to obtain the second classification network after preliminary training.
[0122] The input image is obtained by intercepting the training image, and the input image is used to be input into the initialized second recognition model to train the second classification network.
[0123] The input image includes part of the content information of the corresponding training image. The display sizes of the multiple input images intercepted from the same training image may not be completely the same.
[0124] Because the sizes of image frames in the video data are the same, the sizes of multiple input images captured from the same training image can be the same. For example, a 225*225 pixel image can be captured to obtain m 200*200 (180*180, 160*160, etc.) pixel input images.
[0125] In some embodiments, the presence of overlapping areas (overlapping content information) between multiple input images captured from the same training image can also be understood as the presence of partially identical image content between the two input images, but the input images are not completely identical. This method can simulate the correlation of content information between different image frames in video data. This enables the second classification network trained using the training data of the image modality in the second training set to perform training tasks in the video modality.
[0126] In some embodiments, the computer device extracts a selectable input images from the same training image, and selects b from the a selectable input images as input images; wherein a is a positive integer, b is a positive integer and b is less than or equal to a.
[0127] For example, the computer device extracts 5 selectable input images from different positions of the same training image, and randomly selects 3 images from the 5 selectable input images as input images. In this way, it is possible to simulate the scenario in which multiple image frames are obtained by using different interval lengths when the video recognition model performs classification tasks on the same video data.
[0128] The feature information of the training sample refers to the feature vector used to characterize the features of the training sample.
[0129] The second recognition model can be constructed based on the basic idea of models with spatiotemporal attention mechanisms such as Timesformer, Video-swin-transformer or X-ViT, or can be constructed based on LTSM (Long short-term memory) models.
[0130] In some embodiments, the second classification network of the second recognition model adopts a two-layer MLP (Multilayer Perceptron) structure. In one example, the second classification network of the second recognition model has at least two FC (fully connected) layers and an activation layer (such as a ReLU (Rectified Linear Unit) layer). In some embodiments, the activation layer is set between the two FC layers to reduce the amount of intermediate data generated during the training process.
[0131] The classification label of the training sample is related to the training image corresponding to the multiple input data included in the training sample. In some embodiments, the classification label of the training sample is the same as the classification label of the training image. For the introduction of the classification label, please refer to the above embodiment, which will not be repeated here.
[0132] After obtaining multiple input data, the model recognition device inputs the multiple input data as training samples into the second recognition model; and extracts the training sample feature information from the multiple input data through the preliminarily trained feature encoding network. The preliminarily trained feature encoding network passes the feature information to the second classification network, so that the second classification network performs the second classification task according to the feature information and obtains the classification prediction result corresponding to the training sample.
[0133] In this embodiment, image data is used to train the second classification network. On the one hand, when the second modality is an image modality, the second data set is easier to construct and obtain, and the amount of data in the second data set is smaller. On the other hand, multiple input images are obtained by intercepting a training image, which simulates the continuous change of content information in the video data. In addition, there are overlapping parts in the multiple input images, and there is a certain degree of correlation between the input images, which helps to reduce the interference information obtained by the second recognition model and helps to speed up the aggregation speed of the model.
[0134] The following is an introduction to the method of obtaining multiple input images from the same training image.
[0135] In some embodiments, the computer device obtains multiple input images by intercepting the same training image.
[0136] For example, the computer device performs a capture in the training image to obtain an intermediate image, and processes the intermediate image multiple times to obtain multiple input images.
[0137] For another example, a computer device performs multiple captures on a training image to obtain multiple intermediate images, and content information in different intermediate images is not completely the same, and corresponding input images are obtained through the multiple intermediate images.
[0138] In some embodiments, a computer device captures multiple different image contents from the same training image to obtain multiple input images, including: the computer device determines the size of an image capture frame; the computer device places the image capture frame at multiple different positions of the training image, and captures the image contents within the image capture frame respectively to obtain multiple input images.
[0139] The size of the image capture frame refers to the range of the image area selected in the training image. In some embodiments, the size of the image capture frame is equal to the size of the input image. In other embodiments, the size of the image capture frame is not equal to the size of the input image, for example, after the image capture block captures the intermediate image, the intermediate image is enlarged, reduced or stretched according to the proportion.
[0140] In some embodiments, before using the image capture frame to capture the training image, the computer device determines multiple capture positions in the training image for using the image capture frame for capture (that is, the positions where the image capture frame is placed in the training image). The coordinate information of the multiple capture positions does not overlap. In some embodiments, the multiple capture positions are evenly distributed in the training image, that is, the distances between the capture positions are equal, or the image areas where the capture positions are special points (such as: center points, edge points, any point inside the area) are evenly distributed in the training image, or there is a parallel or vertical relationship between two capture positions. The image capture frame is placed at multiple different positions in the training image, and there are partially overlapping areas between these positions.
[0141] Figure 3 The figure shows a method for intercepting multiple input images provided by an exemplary embodiment of the present application. The computer device uses the upper left corner, lower left corner, upper right corner, lower right corner, and center point (intersection of two diagonal lines) of the training image 310 as interception positions, and uses the image interception frame to intercept the above five positions in the training image to obtain five selectable input images. And among the five selectable input images, three input images (311, 312, and 313) are selected to form training samples.
[0142] This method can simulate the continuous changes of image frames in a section of content appearing in video data, and improve the ability of the second classification network trained using image data to perform the second classification task on video data.
[0143] In some embodiments, a computer device captures multiple different image contents from the same training image to obtain multiple input images, including: the computer device determines the size of an image capture frame and its placement in the training image; the computer device determines multiple different scaling ratios corresponding to the training image within a scaling range corresponding to the training image; the computer device scales the size of the training image according to the multiple scaling ratios to obtain multiple scaled training images; the computer device captures the image contents within the image capture frame from the multiple scaled training images to obtain multiple input images.
[0144] In some embodiments, the scaling range corresponding to the training image refers to the range in which the size of the training image is only adjusted. The scaling range can be any range between [0, 1], such as [0.5, 0.95], [0.40, 0.82], [0.65, 0.90], etc.
[0145] In some embodiments, the computer device determines the corresponding positions of the placement position of the image capture frame from the multiple scaled training images, and captures multiple input images at the corresponding positions. For example, the placement position of the image capture frame (the center point of the image capture frame) in the training image has a coordinate of (1, 1), and after the training image is scaled by 0.9 times, the scaled training image is obtained. At this time, in the scaled training image, the corresponding position of the placement position of the image capture frame is (0.9, 0.9).
[0146] In other embodiments, a computer device captures multiple different image contents from the same training image to obtain multiple input images, including: the computer device determines the size and scaling ratio of the image capture frame, changes the scaling ratio of the image capture frame according to the size of the image capture frame, changes the size of the image capture frame, and uses the image capture frame after the size change to capture the training image to obtain multiple intermediate images that need to be scaled. The computer device scales the multiple intermediate images that need to be scaled respectively to obtain multiple input images of the same size.
[0147] By scaling the training image and the size of the image capture frame, the change in the display size of the content in the image frame caused by the advancement and retreat of the lens in the video data is simulated.
[0148] In some embodiments, by performing scaling processing, there is an arrangement order in the captured input image, for example, multiple input images are sorted from large to small according to the scaling ratio, and the above multiple input images are respectively input into the second recognition model according to the arrangement order, so that the multiple input images have time attributes, thereby improving the second recognition model's processing ability for continuously changing video data.
[0149] In some embodiments, a computer device captures multiple different image contents from the same training image to obtain multiple input images, including: the computer device uses multiple different blur processing parameters to blur the training image respectively to obtain multiple processed training images; the computer device captures the image content within the image capture box from the multiple processed training images respectively to obtain multiple input images.
[0150] In some embodiments, the blurring process is called Gaussian blurring. By blurring the training image, blurring of a portion of the image frame caused by a change in the focus position of the lens in the video data is simulated.
[0151] In one embodiment, a computer device determines a plurality of different blur radii, and blurs a training image according to the blur radii to obtain a plurality of blurred training images, wherein the ranges and widths of blurred edges of the plurality of blurred training images are different.
[0152] By providing different modes of interception operations, simulation of related image frames in video data is achieved, which helps to improve the richness of training data, improve the adaptability of the second classification network to changes in image frames in different video data, and help improve the robustness of the model, so that the training effect of the second recognition model trained on image data in image modality can be transferred to video data in video modality and still maintain good results.
[0153] It should be noted that the method for intercepting the training image involved in the above-mentioned embodiments can be used alone, for example, in one training cycle, multiple training images are scaled to obtain multiple input images. They can also be used in combination, for example, in one training cycle, the image interception frame is placed at different positions for training image A, and training image A is intercepted multiple times to obtain multiple input images. Training data B is blurred multiple times, and multiple input images are intercepted from the same position of multiple blurred input images to obtain multiple input images.
[0154] After obtaining a plurality of input images, the computer device uses the plurality of input images as training samples of the second recognition model to train the second recognition model. The process is described below through several embodiments.
[0155] In some embodiments, the feature encoding network includes: a linear projection layer and a spatiotemporal encoding layer.
[0156] The computer device extracts feature information of the training sample through a feature coding network after preliminary training, including: for a target input image in the training sample, the computer device divides the target input image into blocks to obtain multiple image blocks corresponding to the same target input image; for a target image block in the multiple image blocks, the computer device performs convolution processing on the target image block through a linear projection layer to obtain content coding of the target image block; wherein the content coding of the target image block is used to characterize the image content of the target image block; the computer device determines the time coding of the target image block and the space coding of the target image block; wherein the time coding of the target image block is used to characterize the time step to which the target image block belongs, and the space coding of the target image block is used to characterize the position information of the target image block in the input image to which it belongs; the computer device combines the content coding of the target image block, the time coding of the target image block and the space coding of the target image block to obtain coding information of the target image block; the computer device inputs the coding information corresponding to the training sample into a spatiotemporal coding layer; wherein the coding information corresponding to the training sample includes: coding information of image blocks corresponding to multiple input images belonging to the training sample respectively; the computer device performs coding processing on the coding information corresponding to the training sample through the spatiotemporal coding layer to obtain feature information corresponding to the training sample.
[0157] In some embodiments, the linear projection layer is used to perform linear mapping on the image data and convert the image data into a vector form. The linear projection layer can be an FC layer or a CNN layer. The spatiotemporal coding layer is used to extract features according to the content codes corresponding to the multiple input images.
[0158] In some embodiments, the model recognition device will require multiple input images in the training sample to be processed in blocks, and input the processed image blocks into the feature encoding network after initial training.
[0159] The target input image refers to any one of the multiple input images in the training sample. In some embodiments, the computer device averagely divides the target data image into blocks to obtain multiple image blocks corresponding to the target input image. Optionally, there is no overlapping area between the multiple image blocks corresponding to the target input image. For example, the computer device divides the target input image into 9 equal parts to obtain 9 image blocks corresponding to the target data image.
[0160] After completing the division of multiple input images in the training sample into blocks and obtaining multiple image blocks corresponding to the multiple input images, the computer device inputs the multiple image blocks into the linear projection layer. In some embodiments, the process of inputting the multiple image blocks into the linear projection layer has an input order. For example, multiple image blocks belonging to the same input data are continuously input into the linear projection layer. For another example, for multiple image blocks corresponding to the same input image, the multiple image blocks are respectively input in the order of the corresponding positions of the image blocks in the input image.
[0161] The linear projection layer performs convolution processing on the multiple image blocks respectively to obtain content codes corresponding to the multiple image blocks. In some embodiments, the content code is related to the content information in the image block, and different image blocks corresponding to the same target input image may have different content codes.
[0162] After obtaining the content code corresponding to the target image block, the feature coding network determines the spatial code and the temporal code of the target image block.
[0163] The spatial coding is related to the position of the target image block in the target input image, and the temporal coding is related to the arrangement order of the target input image corresponding to the target object block in multiple input images in the training sample. In some embodiments, the temporal coding is called temporal coding.
[0164] In some embodiments, the spatial encodings corresponding to image blocks at the same position in different input images are the same. For example, the spatial encoding of the image block corresponding to position 1 (such as the upper left corner) of input image 1 is the same as the spatial encoding of the image block corresponding to position 1 (such as the upper left corner) of input image 2.
[0165] In some embodiments, image blocks corresponding to multiple input images of the same training sample use the same time encoding.
[0166] Since video data and image data differ in modality, for each image data in the training sample, instead of using multiple different time codes, a unified time code corresponding to a time step (e.g., t=0) is used. For example, for a certain training sample, the image blocks corresponding to multiple input images of the training sample all use time code 1 corresponding to time step 1.
[0167] In view of the modal differences encountered when migrating image data to the video modality for application, the image data and video data are differentiated so that the second recognition model can distinguish between image data and video data, which helps to prevent the video recognition model from being affected by the inherent morphological bias of image data (for example, the relative position and relative size of objects in image data remain unchanged, while the relative position of objects in video data will change). Previous methods either could not handle cross-modal tasks or could not effectively solve the data bias problem caused by image data when processing image and video modalities.
[0168] The feature coding network combines the content coding, spatial coding, and temporal coding of the target image block to obtain the coding information of the image block. In some embodiments, after the coding information corresponding to the multiple image blocks is obtained, the coding information corresponding to the multiple image blocks is regarded as the coding information of the training sample, and the feature coding network passes the coding information of the multiple image blocks to the spatiotemporal coding layer.
[0169] In some embodiments, the coding information corresponding to the training sample is encoded by the space-time coding layer to obtain feature information corresponding to the training sample, including: the space-time coding layer processes the coding information of multiple image blocks to obtain intermediate codes of multiple image blocks, combines the intermediate codes of multiple image blocks to obtain a combined code, and the space-time coding network processes the combined code to obtain feature information corresponding to the training sample.
[0170] Figure 4 It is a schematic diagram of extracting feature information of training samples provided by an exemplary embodiment of the present application.
[0171] The computer device uses M input images as training samples, and divides the M input images into blocks before inputting the M input images into the feature coding network. For any one of the M input images, the input image is divided into N equal parts to obtain N image blocks, and the N image blocks are input and transmitted to the linear projection layer to obtain the content codes corresponding to the N image blocks. According to the position of a certain image block in the input image, the spatial code of the image block is determined, and the fixed time code (the time code corresponding to t=1), the spatial code and the content code are composed of the coding information of the image block. The coding information of the M*N image blocks is input into the spatiotemporal coding layer to obtain the combined code 410, and the spatiotemporal coding layer is used to determine the feature information corresponding to the training sample according to the combined code.
[0172] The following describes the training process of the video recognition model to be trained through several embodiments.
[0173] Example 1:
[0174] In some embodiments, a computer device uses a third data set to train a video recognition model to be trained to obtain a trained video recognition model, including: the computer device alternately uses a first classification network and a second classification network as a target classification network to be adjusted; wherein, another classification network other than the target classification network in the first classification network and the second classification network is used as a supervised classification network; the computer device determines a training sample of the target classification network in the third data set; the computer device outputs a target classification result and a supervised classification result corresponding to the training sample of the target classification network through the video recognition model to be trained; wherein the target classification result refers to the classification result output by the target classification network, and the supervised classification result refers to the classification result output by the supervised classification network; the computer device determines a reference classification result corresponding to the training sample of the target classification network through a reference classification network; wherein the reference classification network refers to a classification network that has converged and performs the same classification task as the supervised classification network; the computer device determines the training loss of the target classification network based on the target classification result, the supervised classification result and the reference classification result; the computer device adjusts the parameters of the feature encoding network and the target classification network based on the training loss of the target classification network until a stop training condition is met, thereby obtaining a trained video recognition model.
[0175] In some embodiments, the computer device determines the training loss of the target classification network based on the target classification result, the supervised classification result and the reference classification result, including: the computer device determines the classification loss of the target classification network based on the target classification result and the classification label corresponding to the target classification result; wherein the classification loss is used to measure the accuracy of the target classification result output by the target classification network; the computer device determines the supervision loss of the target classification network based on the supervised classification result and the reference classification result; wherein the supervision loss is used to measure the consistency between the supervised classification result and the reference classification result; the computer device determines the training loss of the target classification network based on the classification loss and the supervision loss.
[0176] In Example 1, the computer device only adjusts parameters of one of the first recognition model or the second recognition model in a training batch.
[0177] In some embodiments, the target classification network refers to a classification network that has parameters adjusted in the current training batch. The supervised classification network refers to a classification network that has no parameters adjusted in the current training batch.
[0178] In some embodiments, the control classification network and the supervised classification network are used to perform the same classification task. For example, if the supervised classification network is used to perform a first classification task, the control classification network is also used to perform the first classification task. If the supervised classification network is used to perform a second classification task, the control classification network is also used to perform the second classification task.
[0179] In some embodiments, the control classification network is a first classification network after preliminary training, or a second classification network after preliminary training. The combination of the supervised classification network and the feature encoding network after initial training can determine the type result of the input data, and the type result has a certain accuracy. In some embodiments, the model recognition device forms a control recognition model by using the control classification network and the feature encoding network after initial training.
[0180] The training samples of the target classification network are related to the classification task that the target classification network is used to perform. For example, when the target separation network is a first classification network, the training data of the target classification network is training data whose label data is related to the first classification type.
[0181] In other embodiments, the training samples of the target classification model have the same modality as the training data used in the initial training of the target classification model. For example, if the target classification network is a second classification network and the second modality is an image modality, the training samples of the target classification network in the training process of this stage include image data.
[0182] In one example, when a training video is used to train a target classification network: multiple time points are determined; and image frames corresponding to the multiple time points in the training video are used as training data for the target classification model.
[0183] In another example, when the target classification network is trained using a training image: the same training image is intercepted to obtain multiple input images; the multiple input images are used as training data for the target classification image. Please refer to the above embodiment for the specific content of this process, which will not be repeated here. The computer device inputs the training data A of the target training sample into the video recognition model, obtains the target classification result of the training data A through the target classification network, and obtains the supervised classification result of the training data A through the supervised classification network. The computer device inputs the training data A of the target training sample into the control recognition model, and determines the control classification result of the training data A through the control classification network.
[0184] The classification label corresponding to the target classification result refers to the classification label of the training data corresponding to the target classification result. For example, if the classification label of the training data is (action type) "running", then the classification label corresponding to the target classification result is "running".
[0185] In some embodiments, the computer device calculates the classification loss of the target classification network through the target classification result and the label corresponding to the target classification result. The computer device calculates the supervision loss through the supervised classification result and the label corresponding to the comparison classification result, and weights the classification loss and supervision loss of the target classification network according to the weight ratio to obtain the training loss of the target classification network.
[0186] In some embodiments, the weight ratio is pre-set, for example, the weight ratio is set to classification loss of target classification network: supervision loss = 1:1, classification loss of target classification network: supervision loss = 3:2, etc.
[0187] Below, the training process of the video recognition model to be trained is illustrated by taking the first classification network performing an action classification task (hereinafter the first classification network is referred to as the action classification network) and the second classification network performing a scene classification task (hereinafter the second classification network is referred to as the action classification network).
[0188] Figure 5 A schematic diagram of a training process of a video recognition model to be trained provided by an exemplary embodiment of the present application is shown.
[0189] In a certain training batch, the computer device uses the scene classification network after secondary training in the video recognition model 510 to be trained as the target classification network, and the target training data is image data. The computer device uses the action classification network after initial training as the control classification network, and the action classification network after secondary training as the supervision network. By inputting the training sample into the video recognition model 510 to be trained, the target classification result (scene type) and the supervision classification result (action type) to be trained are obtained. And the control classification result (action type) of the training sample is determined by the feature encoding network after initial training (which can be called a fixed action model (fixed action model) at this time) and the action classification network after initial training. The supervision loss is determined by the supervision classification result and the control classification result, and the classification loss of the scene classification network is determined by the target classification result and the label classification corresponding to the target classification result. The parameters in the feature encoding network after secondary training and the action classification network after secondary training are adjusted by the classification loss of the supervision loss and the scene classification network.
[0190] The above method helps to reduce the need for major changes to the parameters of the video recognition model to be trained, helps to speed up the training of the video recognition model to be trained, and also helps to maintain the accuracy of the supervised classification network in performing the corresponding classification task.
[0191] Example 2:
[0192] In some embodiments, a computer device uses a third data set to train a video recognition model to be trained to obtain a trained video recognition model, including: the computer device determines multiple training samples of the video recognition model to be trained from the third data set according to the sample selection ratio of the first modality and the second modality; the computer device outputs a first classification result and a second classification result corresponding to the training samples through the video recognition model to be trained; wherein the first classification result refers to the classification result output by the first classification network, and the second classification result refers to the classification result output by the second classification network; the computer device determines the model training loss based on the first classification result and the second classification result; the computer device adjusts the parameters of each network in the video recognition model to be trained according to the model training loss until the training stop condition is met, thereby obtaining a trained video recognition model.
[0193] In some embodiments, the sample selection ratio of the first modality and the second modality can be preconfigured, for example, set according to the accuracy requirements of the first classification task and the accuracy requirements of the second classification task.
[0194] Each network in the video recognition model to be trained includes: a feature encoding network after secondary training, a first classification network after initial training, and a second classification network after secondary training.
[0195] In some embodiments, the computer device determines the model training loss based on the first classification result and the second classification result, including: the computer device determines the first classification loss based on the first classification result and the classification label corresponding to the first classification result, and the first classification loss is used to measure the accuracy of the first classification result output by the first classification network; the computer device determines the second classification loss based on the second classification result and the classification label corresponding to the second classification result, and the second classification loss is used to measure the accuracy of the second classification result output by the second classification network; the computer device determines the first constraint loss based on the first classification result and the reference classification result corresponding to the first classification result, and the first constraint loss is used to measure the consistency between the first classification result and the reference classification result corresponding to the first classification result; the computer device determines the second constraint loss based on the reference classification result corresponding to the second classification result and the second classification result, and the second constraint loss is used to measure the consistency between the second classification result and the reference classification result corresponding to the second classification result; the computer device performs weighted summation of the first classification loss, the second classification loss, the first constraint loss and the second constraint loss according to the weighted weights to obtain the model training loss; wherein the weighted weights are dynamically adjusted according to the loss proportion of each classification task during the model training process.
[0196] The calculation method of the reference classification result corresponding to the first classification result and the reference classification result corresponding to the first classification result is the same as the calculation method of the control classification result in Example 1. Please refer to Example 1 for details and will not be repeated here.
[0197] In some embodiments, during the training process of the video model to be trained, the uncertainty loss (Uncertainty Loss) is changed to determine the weights between the first classification loss, the second classification loss, the first constraint loss and the second constraint loss, reducing the weight corresponding to the loss with a larger value, and increasing the weight corresponding to the loss with a smaller value.
[0198] By training the video recognition model to be trained, it helps to improve the accuracy of the video recognition model in performing the first classification task and the second classification task. The trained video recognition model shares the parameters in the feature encoding network in different classification tasks, achieving potential synergy and helping to obtain competitive classification results in multimedia data classification tasks in both image and video modalities.
[0199] For a video recognition model that can only perform a single classification task, two complete models are needed to analyze the video data to perform the first classification task (such as action classification) and the second classification task (such as scene classification). Assuming that each model processes 5,000 50-minute long videos every day, 400 containers are required (a container refers to a collection of computer resources that runs a video recognition model). The video recognition model obtained through this solution only needs half the number of containers to process the same amount of data by sharing the parameters of the feature encoding network.
[0200] In addition, since the training process of the video recognition model integrates the training process of multiple tasks and utilizes the potential synergy between different tasks, the recognition effect is further improved. In the experimental data obtained by the video device model trained by this method for the classification experiment of video data, the scene recognition effect based on video data is 3% higher than the accuracy of the original image version of the model, and the action recognition accuracy is 0.8% higher.
[0201] The video recognition model obtained after training by the above-mentioned video recognition model training method is a model of a general architecture. For a video data input, the trained video recognition model only needs to extract feature information once to simultaneously predict the first type and the second type, and the amount of calculation in the process of classifying the video data is relatively small. The first type and the second type can belong to different modalities, that is, one video recognition model can be used to complete the classification task in the image modality and the classification task in the video modality.
[0202] In one embodiment, a classification task is performed on video data through a trained video recognition model, including: determining multiple image frames in the video data; dividing the multiple image frames into blocks to obtain image blocks corresponding to the multiple image frames, inputting the image blocks corresponding to the multiple image frames into the video recognition model, linearly mapping the multiple image blocks through the linear mapping layer of the feature coding network to obtain content codes corresponding to the multiple image blocks, and for a target image block among the multiple image blocks, determining the time code and space code of the target image block; wherein the time code of the target image block is related to the time point when the image frame to which the target image block belongs appears in the video data, and the time code of the target image block is related to the position of the target image block in the image frame to which it belongs; combining the time code, space code and content code of the target image block to obtain coding information of the target image block; inputting the coding information corresponding to the video image into the spatial coding network, obtaining feature information corresponding to the video image through the spatial coding network, using a first classification network to determine a first classification result of the video data according to the feature information, and using a second classification network to determine a second classification result of the video data according to the feature information.
[0203] In some embodiments, determining multiple image frames in video data includes: determining multiple interval durations; selecting multiple image frames in the video data according to the interval durations; wherein the interval duration refers to the interval between time points corresponding to two adjacent selected image frames in the video data. The multiple interval durations can be equal. For example, for video data with a playback duration of 20 seconds, starting from the first image frame, an image frame is determined every 2 seconds, and a total of 10 image frames are obtained. For the specific content of the above embodiment, please refer to the content of the video recognition model training method embodiment, which will not be repeated here.
[0204] In one example, the trained video recognition model can be applied to the delivery of recommended information (such as advertisements). In some embodiments, the video recognition model can be used to classify video data, obtain first type results and second type results corresponding to the video data, and determine the recommended information corresponding to the video data based on the first type results and the second type results.
[0205] For example, the video recognition model determines the scene classification and action classification of the video data, and the server can select the corresponding recommendation information in the recommendation information library according to the scene classification and action classification of the video data. When the client of the terminal device needs to play the video data, the client requests the video data from the server, and the server sends the video data and the recommendation information corresponding to the video data to the terminal device. The client displays the recommendation information during the process of playing the video data.
[0206] Figure 6 It is a schematic diagram showing an exemplary embodiment of the present application for displaying recommendation information based on video recognition.
[0207] The video recognition model determines that the scene type of the video clip is an ancient courtyard 610, and the action type is playing the zither 620. The server can determine the target recommendation information in the recommendation information library based on the above information, such as a liquor advertisement 630 corresponding to the customer placed in the ancient scene (and also including recommendation information for minors to avoid drinking). When the client requests the video data, the target recommendation information is sent to the client.
[0208] In one example, the trained video recognition model can be applied to screen video data. For example, a multimedia data storage space includes multiple video data, making it difficult for users to directly find the video data they need to find. In this case, the video recognition model can be used to respectively determine the first classification type and the second classification type corresponding to the multiple video data. For example, the first classification type is a video attribute type, and the second classification type is related to a specific object. The video data in the multimedia data storage space is classified by the video recognition model, and the first classification result and the second classification result corresponding to each video data are determined and stored. In the case where the user needs to query a certain video data, the first classification result and the second classification result to which the video data belongs can be provided. The multimedia data storage space queries the stored video data according to the first classification result and the second classification result provided by the user, and preferentially displays the candidate video data that meets the above two classification results to the user.
[0209] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.
[0210] Figure 7 A block diagram of a video recognition model training device provided by an exemplary embodiment of the present application is shown. The device 700 may include: a first training module 710 , a model construction module 720 , a second training module 730 , and a third training module 740 .
[0211] The first training module 710 is used to train the first recognition model using a first data set, adjust the parameters of the feature encoding network and the first classification network included in the first recognition model, and obtain the feature encoding network after preliminary training and the first classification network after preliminary training; wherein the first data set includes training data of the first modality.
[0212] The model construction module 720 is used to construct an initialized second recognition model based on the second classification network and the preliminarily trained feature encoding network.
[0213] The second training module 730 is used to train the second classification network in the initialized second recognition model using a second data set, fix the parameters of the feature encoding network after the preliminary training, and adjust the parameters of the second classification network to obtain the second classification network after the preliminary training; wherein the second data set includes training data of a second modality, and the second modality and the first modality are two different modalities.
[0214] The model construction module 720 is also used to construct a preliminarily trained second recognition model based on the preliminarily trained feature encoding network and the preliminarily trained second classification network.
[0215] The second training module 730 is also used to use the second data set to train the second recognition model after the initial training, adjust the parameters of the feature encoding network after the initial training and the second classification network after the initial training, and obtain the feature encoding network after the secondary training and the second classification network after the secondary training.
[0216] The model construction module 720 is also used to construct a video recognition model to be trained based on the feature encoding network after the secondary training, the first classification network after the preliminary training and the second classification network after the secondary training.
[0217] The third training module 740 is used to train the video recognition model to be trained using a third data set to obtain a trained video recognition model; wherein the third data set includes at least one of the following: training data of the first modality and training data of the second modality.
[0218] In some embodiments, the first modality is a video modality, and the first data set includes at least one training video; the second modality is an image modality, and the second data set includes at least one training image;
[0219] The first training module is used to obtain, for each training video, a plurality of image frames corresponding to the training video according to a plurality of interception time points; train the first recognition model using the plurality of image frames corresponding to the training video, adjust the parameters of the feature encoding network and the first classification network included in the first recognition model, and obtain the initially trained feature encoding network and the initially trained first classification network;
[0220] The second training module is used to transform each training image to obtain multiple transformed images corresponding to the training image; use the multiple transformed images corresponding to the training image to train the second classification network, fix the parameters of the feature encoding network after the preliminary training, and adjust the parameters of the second classification network to obtain the second classification network after the preliminary training.
[0221] In some embodiments, the second modality is an image modality, and the second data set includes at least one training image; the second training module 730 includes: an image processing submodule ( Figure 7(not shown), for capturing a plurality of different image contents from the same training image to obtain a plurality of input images; wherein any two input images belonging to the same training image have some identical image contents; a preprocessing submodule ( Figure 7 (not shown), used to use the multiple input images as training samples of the initialized second recognition model; feature extraction submodule ( Figure 7 (not shown), used to extract the feature information of the training sample through the feature encoding network after the preliminary training; a classification determination submodule, used to determine the predicted classification result of the training sample according to the feature information of the training sample through the second classification network; a parameter adjustment submodule ( Figure 7 (not shown) is used to determine the training loss according to the predicted classification results and classification labels of the training samples, and adjust the parameters of the second classification network based on the training loss to obtain the second classification network after the preliminary training.
[0222] In some embodiments, the image processing submodule is used to determine the size of an image capture frame; place the image capture frame at a plurality of different positions of the training image, and capture the image content within the image capture frame respectively to obtain the plurality of input images.
[0223] In some embodiments, the image processing submodule is used to determine the size of the image capture frame and its placement in the training image; determine multiple different scaling ratios corresponding to the training image within the scaling range corresponding to the training image; scale the size of the training image according to the multiple different scaling ratios to obtain multiple scaled training images; and capture the image content in the image capture frame from the multiple scaled training images to obtain the multiple input images.
[0224] In some embodiments, the image processing submodule is used to use a variety of different blur processing parameters to blur the training images respectively to obtain multiple processed training images; and to capture the image content within the image capture box from the multiple processed training images respectively to obtain the multiple input images.
[0225] In some embodiments, the feature encoding network includes: a linear projection layer and a spatiotemporal encoding layer; the feature extraction submodule is used to divide the target input image in the training sample into blocks to obtain multiple image blocks corresponding to the same target input image; for a target image block in the multiple image blocks, convolution processing is performed on the target image block through the linear projection layer to obtain a content encoding of the target image block; wherein the content encoding of the target image block is used to characterize the image content of the target image block; determine the time encoding of the target image block, and the spatial encoding of the target image block; wherein the time encoding of the target image block is used to characterize The time step to which the target image block belongs, the spatial encoding of the target image block is used to characterize the position information of the target image block in the input image to which it belongs; the content encoding of the target image block, the time encoding of the target image block and the spatial encoding of the target image block are combined to obtain the encoding information of the target image block; the encoding information corresponding to the training sample is input into the spatiotemporal encoding layer; wherein the encoding information corresponding to the training sample includes: the encoding information of the image blocks respectively corresponding to the multiple input images belonging to the training sample; the encoding information corresponding to the training sample is encoded through the spatiotemporal encoding layer to obtain the feature information corresponding to the training sample.
[0226] In some embodiments, the image blocks corresponding to the multiple input images of the same training sample use the same time encoding.
[0227] In some embodiments, the parameter adjustment submodule includes: a network selection unit, which is used to alternately use the first classification network and the second classification network as the target classification network to be adjusted; wherein, the other classification network other than the target classification network in the first classification network and the second classification network is used as a supervised classification network; a sample determination unit, which is used to determine the training samples of the target classification network in the third data set; a classification determination unit, which is used to output the target classification results and supervised classification results corresponding to the training samples of the target classification network through the video recognition model to be trained; wherein, the target classification result refers to the classification result output by the target classification network, and the supervised classification result The classification result refers to the classification result output by the supervised classification network; the comparison determination unit is used to determine the reference classification result corresponding to the training sample of the target classification network through the comparison classification network; wherein the comparison classification network refers to a classification network that has converged and performs the same classification task as the supervised classification network; the loss determination unit is used to determine the training loss of the target classification network according to the target classification result, the supervised classification result and the reference classification result; the parameter adjustment unit is used to adjust the parameters of the feature encoding network and the target classification network according to the training loss of the target classification network until the training stop condition is met, thereby obtaining the trained video recognition model.
[0228] In some embodiments, the loss determination unit is used to determine the classification loss of the target classification network based on the target classification result and the classification label corresponding to the target classification result; wherein the classification loss is used to measure the accuracy of the target classification result output by the target classification network; determine the supervised loss of the target classification network based on the supervised classification result and the reference classification result; wherein the supervised loss is used to measure the consistency between the supervised classification result and the reference classification result; determine the training loss of the target classification network based on the classification loss and the supervised loss.
[0229] In some embodiments, the parameter adjustment submodule includes: a sample determination unit, which is used to determine multiple training samples of the video recognition model to be trained from the third data set according to the sample selection ratio of the first modality and the second modality; a classification determination unit, which is used to output the first classification result and the second classification result corresponding to the training sample through the video recognition model to be trained; wherein the first classification result refers to the classification result output by the first classification network, and the second classification result refers to the classification result output by the second classification network; a loss determination unit, which is used to determine the model training loss according to the first classification result and the second classification result; a parameter adjustment unit, which is used to adjust the parameters of each network in the video recognition model to be trained according to the model training loss until the training stop condition is met, and the trained video recognition model is obtained.
[0230] In some embodiments, the parameter adjustment unit is used to determine a first classification loss based on the first classification result and the classification label corresponding to the first classification result, and the first classification loss is used to measure the accuracy of the first classification result output by the first classification network; determine a second classification loss based on the second classification result and the classification label corresponding to the second classification result, and the second classification loss is used to measure the accuracy of the second classification result output by the second classification network; determine a first constraint loss based on the first classification result and the reference classification result corresponding to the first classification result, and the first constraint loss is used to measure the consistency between the first classification result and the reference classification result corresponding to the first classification result; determine a second constraint loss based on the second classification result and the reference classification result corresponding to the second classification result, and the second constraint loss is used to measure the consistency between the second classification result and the reference classification result corresponding to the second classification result; and perform weighted summation of the first classification loss, the second classification loss, the first constraint loss and the second constraint loss according to weighted weights to obtain the model training loss; wherein the weighted weights are dynamically adjusted according to the loss proportion of each of the classification tasks during the model training process.
[0231] It should be noted that the device provided in the above embodiment, when realizing its function, only uses the division of the above functional modules as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the content structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device provided in the above embodiment belongs to the same concept as the method embodiment. The specific implementation process is detailed in the method embodiment, which will not be repeated here. Please refer to the description of the method side embodiment for the beneficial effects of the device provided in the above embodiment, which will not be repeated here.
[0232] Figure 8 A structural block diagram of a computer device provided by an exemplary embodiment of the present application is shown.
[0233] Typically, the computer device 800 includes a processor 801 and a memory 802 .
[0234] The processor 801 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like. The processor 801 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). The processor 801 may also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 801 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 801 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.
[0235] The memory 802 may include one or more computer-readable storage media, which may be tangible and non-transitory. The memory 802 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 802 stores a computer program, which is loaded and executed by the processor 801 to implement the training method of the video recognition model provided in the above-mentioned method embodiments.
[0236] Those skilled in the art will understand that Figure 8 The structure shown in the figure does not constitute a limitation on the computer device 800, and the computer device 800 may include more or less components than those shown in the figure, or combine some components, or adopt a different arrangement of components.
[0237] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. The computer program is loaded and executed by a processor to implement the training method of the video recognition model provided by the above-mentioned method embodiments.
[0238] The computer readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer readable instructions, data structures, program modules or other data. Computer storage media include RAM (Random Access Memory), ROM (Read-Only Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory or other solid-state storage technology, DVD (Digital Video Disc) or other optical storage, cassettes, magnetic tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will know that the computer storage medium is not limited to the above-mentioned ones.
[0239] An embodiment of the present application also provides a computer program product, which includes a computer program, and the computer program is stored in a computer-readable storage medium. A processor reads and executes the computer program from the computer-readable storage medium to implement the training method of the video recognition model provided in the above-mentioned method embodiments.
[0240] It should be understood that the "plurality" mentioned in this article refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.
[0241] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions. For example, the target voice data involved in this application are all obtained with full authorization.
[0242] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent switching, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.
Claims
1. A video recognition model training method, characterized in that: The video recognition model includes: a first recognition model and a second recognition model, the first recognition model and the second recognition model share the same feature encoding network, the first recognition model includes the feature encoding network and a first classification network, the second recognition model includes the feature encoding network and a second classification network, the first classification network and the second classification network are used to perform two different classification tasks; the method includes: The first recognition model is trained using a first data set, and parameters of the feature encoding network and the first classification network included in the first recognition model are adjusted to obtain a preliminarily trained feature encoding network and a preliminarily trained first classification network; wherein the first data set includes training data of a first modality; Based on the second classification network and the preliminarily trained feature encoding network, construct an initialized second recognition model; The second classification network in the initialized second recognition model is trained using a second data set, the parameters of the feature encoding network after the preliminary training are fixed, and the parameters of the second classification network are adjusted to obtain the second classification network after the preliminary training; wherein the second data set includes training data of a second modality, and the second modality and the first modality are two different modalities; Based on the preliminarily trained feature encoding network and the preliminarily trained second classification network, constructing a preliminarily trained second recognition model; Using the second data set to train the second recognition model after the initial training, adjusting the parameters of the feature encoding network after the initial training and the second classification network after the initial training, to obtain the feature encoding network after the secondary training and the second classification network after the secondary training; Constructing a video recognition model to be trained based on the feature encoding network after the secondary training, the first classification network after the preliminary training, and the second classification network after the secondary training; The video recognition model to be trained is trained using a third data set to obtain a trained video recognition model; wherein the third data set includes at least one of the following: training data of the first modality and training data of the second modality.
2. The method according to claim 1, characterized in that The first modality is a video modality, and the first data set includes at least one training video; the second modality is an image modality, and the second data set includes at least one training image; The first recognition model is trained using the first data set, and the parameters of the feature encoding network and the first classification network included in the first recognition model are adjusted to obtain the preliminarily trained feature encoding network and the preliminarily trained first classification network, including: For each training video, obtaining multiple image frames corresponding to the training video according to multiple interception time points; Using a plurality of image frames corresponding to the training video, training the first recognition model, adjusting the parameters of the feature encoding network and the first classification network included in the first recognition model, and obtaining the initially trained feature encoding network and the initially trained first classification network; The second classification network in the initialized second recognition model is trained using the second data set, the parameters of the feature encoding network after the preliminary training are fixed, and the parameters of the second classification network are adjusted to obtain the second classification network after the preliminary training, including: For each training image, transform the training image to obtain a plurality of transformed images corresponding to the training image; The second classification network is trained using a plurality of transformed images corresponding to the training image, the parameters of the feature encoding network after the preliminary training are fixed, and the parameters of the second classification network are adjusted to obtain the second classification network after the preliminary training.
3. The method according to claim 1, characterized in that The second modality is an image modality, and the second data set includes at least one training image; The second classification network in the initialized second recognition model is trained using the second data set, the parameters of the feature encoding network after the preliminary training are fixed, and the parameters of the second classification network are adjusted to obtain the second classification network after the preliminary training, including: Extracting a plurality of different image contents from the same training image to obtain a plurality of input images; wherein any two input images belonging to the same training image have some identical image contents; Using the multiple input images as training samples of the initialized second recognition model; Extracting feature information of the training samples through the preliminarily trained feature encoding network; Determining a predicted classification result of the training sample according to the feature information of the training sample through the second classification network; According to the predicted classification results and classification labels of the training samples, the training loss is determined, and the parameters of the second classification network are adjusted based on the training loss to obtain the second classification network after the preliminary training.
4. The method according to claim 3, characterized in that The method of extracting a plurality of different image contents from the same training image to obtain a plurality of input images includes: Determine the size of the image capture frame; The image capture frame is placed at a plurality of different positions of the training image, and the image contents within the image capture frame are captured respectively to obtain the plurality of input images.
5. The method according to claim 3, characterized in that: The method of extracting a plurality of different image contents from the same training image to obtain a plurality of input images includes: Determine the size of the image capture frame and its placement in the training image; Determining a plurality of different scaling ratios corresponding to the training image within a scaling range corresponding to the training image; Rescaling the size of the training image according to the multiple different scaling ratios to obtain multiple scaled training images; Image contents within the image capture frames are respectively captured from the multiple scaled training images to obtain the multiple input images.
6. The method according to claim 3, characterized in that The method of extracting a plurality of different image contents from the same training image to obtain a plurality of input images includes: Using a plurality of different fuzzy processing parameters to perform fuzzy processing on the training image respectively to obtain a plurality of processed training images; Image contents within image capture frames are respectively captured from the multiple processed training images to obtain the multiple input images.
7. The method according to claim 3, characterized in that The feature encoding network includes: a linear projection layer and a spatiotemporal encoding layer; The extracting feature information of the training sample through the feature encoding network after the preliminary training includes: For a target input image in the training sample, dividing the target input image into blocks to obtain a plurality of image blocks corresponding to the same target input image; For a target image block among the multiple image blocks, convolution processing is performed on the target image block through the linear projection layer to obtain a content code of the target image block; wherein the content code of the target image block is used to represent the image content of the target image block; Determine a time code of the target image block and a space code of the target image block; wherein the time code of the target image block is used to represent the time step to which the target image block belongs, and the space code of the target image block is used to represent the position information of the target image block in the input image to which it belongs; Combining the content coding of the target image block, the time coding of the target image block and the space coding of the target image block to obtain coding information of the target image block; Inputting the coding information corresponding to the training sample into the spatiotemporal coding layer; wherein the coding information corresponding to the training sample includes: coding information of the image blocks respectively corresponding to the multiple input images belonging to the training sample; The coding information corresponding to the training sample is coded through the spatiotemporal coding layer to obtain feature information corresponding to the training sample.
8. The method according to claim 7, characterized in that The image blocks respectively corresponding to the multiple input images of the same training sample use the same time encoding.
9. The method according to claim 1, characterized in that: The step of training the video recognition model to be trained by using the third data set to obtain a trained video recognition model includes: Taking turns to use the first classification network and the second classification network as target classification networks to be adjusted; wherein another classification network other than the target classification network in the first classification network and the second classification network is used as a supervised classification network; Determining training samples of the target classification network in the third data set; Outputting target classification results and supervised classification results corresponding to the training samples of the target classification network through the video recognition model to be trained; wherein the target classification result refers to the classification result output by the target classification network, and the supervised classification result refers to the classification result output by the supervised classification network; Determine a reference classification result corresponding to the training sample of the target classification network by comparing the classification network; wherein the comparison classification network refers to a classification network that has converged and performs the same classification task as the supervised classification network; Determining the training loss of the target classification network according to the target classification result, the supervised classification result and the reference classification result; According to the training loss of the target classification network, the parameters of the feature encoding network and the target classification network are adjusted until the training stop condition is met, thereby obtaining the trained video recognition model.
10. The method according to claim 9, characterized in that The step of determining the training loss of the target classification network according to the target classification result, the supervised classification result and the reference classification result includes: Determine the classification loss of the target classification network according to the target classification result and the classification label corresponding to the target classification result; wherein the classification loss is used to measure the accuracy of the target classification result output by the target classification network; Determining the supervised loss of the target classification network according to the supervised classification result and the reference classification result; wherein the supervised loss is used to measure the consistency between the supervised classification result and the reference classification result; The training loss of the target classification network is determined according to the classification loss and the supervision loss.
11. The method according to claim 1, characterized in that: The step of training the video recognition model to be trained by using the third data set to obtain a trained video recognition model includes: Determining a plurality of training samples of the video recognition model to be trained from the third data set according to a sample selection ratio of the first modality and the second modality; Outputting a first classification result and a second classification result corresponding to the training sample through the video recognition model to be trained; wherein the first classification result refers to the classification result output by the first classification network, and the second classification result refers to the classification result output by the second classification network; Determine a model training loss according to the first classification result and the second classification result; The parameters of each network in the video recognition model to be trained are adjusted according to the model training loss until the training stop condition is met, thereby obtaining the trained video recognition model.
12. The method according to claim 11, characterized in that The determining the model training loss according to the first classification result and the second classification result includes: Determine a first classification loss according to the first classification result and a classification label corresponding to the first classification result, where the first classification loss is used to measure the accuracy of the first classification result output by the first classification network; Determine a second classification loss according to the second classification result and the classification label corresponding to the second classification result, where the second classification loss is used to measure the accuracy of the second classification result output by the second classification network; determining a first constraint loss according to the first classification result and a reference classification result corresponding to the first classification result, wherein the first constraint loss is used to measure the consistency between the first classification result and the reference classification result corresponding to the first classification result; Determining a second constraint loss according to the second classification result and a reference classification result corresponding to the second classification result, where the second constraint loss is used to measure the consistency between the second classification result and the reference classification result corresponding to the second classification result; According to the weighted weights, the first classification loss, the second classification loss, the first constraint loss and the second constraint loss are weightedly summed to obtain the model training loss; wherein the weighted weights are dynamically adjusted according to the loss proportion of each classification task during the model training process.
13. A training device for a video recognition model, characterized in that: The video recognition model includes: a first recognition model and a second recognition model, the first recognition model and the second recognition model share the same feature encoding network, the first recognition model includes the feature encoding network and a first classification network, the second recognition model includes the feature encoding network and a second classification network, the first classification network and the second classification network are used to perform two different classification tasks; the device includes: A first training module, used to train the first recognition model using a first data set, adjust the parameters of the feature encoding network and the first classification network included in the first recognition model, and obtain a preliminarily trained feature encoding network and a preliminarily trained first classification network; wherein the first data set includes training data of a first modality; A model construction module, used to construct an initialized second recognition model based on the second classification network and the preliminarily trained feature encoding network; A second training module, used to train the second classification network in the initialized second recognition model using a second data set, fix the parameters of the feature encoding network after the preliminary training, and adjust the parameters of the second classification network to obtain the second classification network after the preliminary training; wherein the second data set includes training data of a second modality, and the second modality and the first modality are two different modalities; The model construction module is further used to construct a preliminarily trained second recognition model based on the preliminarily trained feature encoding network and the preliminarily trained second classification network; The second training module is further used to train the second recognition model after the initial training using the second data set, adjust the parameters of the feature encoding network after the initial training and the second classification network after the initial training, and obtain the feature encoding network after the secondary training and the second classification network after the secondary training; The model construction module is further used to construct a video recognition model to be trained based on the feature encoding network after the secondary training, the first classification network after the preliminary training, and the second classification network after the secondary training; The third training module is used to train the video recognition model to be trained using a third data set to obtain a trained video recognition model; wherein the third data set includes at least one of the following: training data of the first modality and training data of the second modality.
14. A computer device, characterized in that: The computer device includes a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the training method of the video recognition model as described in any one of claims 1 to 12.
15. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, which is loaded and executed by a processor to implement the training method of the video recognition model as described in any one of claims 1 to 12.
16. A computer program product, characterized in that The computer program product includes a computer program, which is stored in a computer-readable storage medium. The processor reads and executes the computer program from the computer-readable storage medium to implement the training method of the video recognition model as described in any one of claims 1 to 12.
