Action recognition model training method and apparatus, electronic device, and storage medium
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GUANGDONG OPPO MOBILE TELECOMMUNICATIONS CORP LTD
- Filing Date
- 2022-03-17
- Publication Date
- 2026-06-02
Smart Images
Figure CN116824418B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of neural network technology, and in particular to a method, apparatus, electronic device and storage medium for training an action recognition model. Background Technology
[0002] Action recognition models can be used to identify actions in videos and determine the action category to which the action belongs. Currently, these action recognition models are mainly trained using high-resolution videos. However, in some scenarios, such as to avoid human visual recognition of the video and reduce the risk of improper dissemination, low-resolution videos are generally stored. In such cases, if the action recognition model is used to identify these low-resolution videos, the recognition accuracy will often be poor, resulting in low recognition precision. Summary of the Invention
[0003] This application provides a method, apparatus, electronic device, and storage medium for training an action recognition model, which can improve the recognition accuracy of the action recognition model for low-resolution videos. The technical solution is as follows:
[0004] On the one hand, a method for training an action recognition model is provided, the method comprising:
[0005] Multiple sample video clips are acquired, and each sample video clip is labeled with an action tag, which is a tag corresponding to an action in the sample video clip;
[0006] Based on the plurality of sample video segments, a comparison label is determined for every two sample video segments. The comparison label is used to represent the comparison relationship between the two sample video segments. At least one of the two sample video segments has a resolution lower than the baseline resolution.
[0007] The action recognition model is trained by training a model based on every two sample video segments and their corresponding action labels and comparison labels.
[0008] On the other hand, an action recognition model training device is provided, the device comprising:
[0009] The first acquisition module is used to acquire multiple sample video segments, wherein the sample video segments are labeled with action tags, and the action tags are the tags corresponding to the actions in the sample video segments.
[0010] The first determining module is used to determine a comparison label corresponding to every two sample video segments based on the plurality of sample video segments. The comparison label is used to indicate the comparison relationship between the two sample video segments, and at least one of the two sample video segments has a resolution lower than the reference resolution.
[0011] The training module is used to train the model based on every two sample video segments and their corresponding action labels and contrast labels to obtain the action recognition model.
[0012] On the other hand, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one piece of program code, which is loaded and executed by the processor to implement the action recognition model training method described above.
[0013] On the other hand, a computer-readable storage medium is provided, wherein at least one piece of program code is stored in the computer-readable storage medium, the at least one piece of program code being loaded and executed by a processor to implement the action recognition model training method described above.
[0014] On the other hand, a computer program product is provided, which stores at least one piece of program code, which is loaded and executed by a processor to implement the action recognition model training method described above.
[0015] The beneficial effects of the technical solutions provided in this application are:
[0016] This application provides a method for training an action recognition model. The method trains the model by determining the contrast labels corresponding to every two sample video segments. Since the contrast labels can represent the contrast relationship between the two sample video segments, and at least one of the sample video segments has a resolution lower than the baseline resolution, the model can learn the resolution features of the low-resolution video segments based on the contrast labels during training. At the same time, it can also learn the action features in the low-resolution video based on the action labels of the sample video segments. Therefore, the trained action recognition model can accurately identify actions in low-resolution videos, thereby improving the accuracy of action recognition in low-resolution videos.
[0017] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this disclosure. Attached Figure Description
[0018] Figure 1 This is a schematic diagram of the implementation environment of an action recognition model training method provided in an embodiment of this application;
[0019] Figure 2 This is a flowchart of an action recognition model training method provided in an embodiment of this application;
[0020] Figure 3 This is a flowchart of an action recognition model training method provided in an embodiment of this application;
[0021] Figure 4 This is a schematic diagram of a constructed time-domain comparison label provided in an embodiment of this application;
[0022] Figure 5 This is a schematic diagram of a constructed spatial comparison label provided in an embodiment of this application;
[0023] Figure 6 This is a schematic diagram of a neural network architecture provided in an embodiment of this application;
[0024] Figure 7 This is a schematic diagram illustrating model training based on temporal and spatial contrast labels, provided in an embodiment of this application.
[0025] Figure 8 This is a schematic diagram illustrating the training of a model for recognizing actions in low-resolution videos based on self-supervised learning, as provided in an embodiment of this application.
[0026] Figure 9 This is a flowchart of an action recognition method provided in an embodiment of this application;
[0027] Figure 10 This is a schematic diagram of the structure of an action recognition model training device provided in an embodiment of this application;
[0028] Figure 11 This is a structural block diagram of a terminal provided in an embodiment of this application;
[0029] Figure 12 This is a structural block diagram of a server provided in an embodiment of this application. Detailed Implementation
[0030] To make the technical solution and advantages of this application clearer, the embodiments of this application will be described in further detail below.
[0031] The terms "first," "second," "third," and "fourth," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.
[0032] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the sample video clips, motion tags, temporal comparison tags, and spatial comparison tags involved in this application were all obtained with full authorization.
[0033] Figure 1 This is a schematic diagram of the implementation environment of an action recognition model training method provided in an embodiment of this application. See also... Figure 1 The implementation environment includes an electronic device, which can be provided as a terminal 101, a server 102, or both a terminal 101 and a server 102. The terminal 101 and the server 102 can be connected via a wireless or wired network. In this embodiment, the electronic device is not specifically limited.
[0034] If the electronic device is provided as terminal 101, then terminal 101 performs model training to obtain an action recognition model. This action recognition model can be deployed in terminal 101, so that terminal 101 can use the action recognition model to identify actions in low-resolution or even very low-resolution videos and determine the action label of the action, that is, the action category to which the action belongs.
[0035] If the electronic device is provided as server 102, then server 102 performs model training to obtain an action recognition model. This action recognition model can be deployed to terminal 101. Accordingly, the implementation environment also includes terminal 101, which can be connected to server 102 via a wireless or wired network. Terminal 101 can use the action recognition model to recognize actions in low-resolution or even very low-resolution videos.
[0036] If the electronic device provides a terminal 101 and a server 102, the server 102 performs model training to obtain an action recognition model, and then the action recognition model is deployed to the terminal 101. In this way, the terminal 101 can use the action recognition model to recognize actions in low-resolution or even very low-resolution videos.
[0037] In this embodiment, the method can be applied to video storage scenarios. In such scenarios, to avoid direct visual recognition of the video and reduce the risk of improper dissemination, low-resolution or even extremely low-resolution videos are typically uploaded to a cloud server. Before uploading to the cloud server, an action recognition model can be deployed on a terminal. This terminal connects to the cloud server via a wireless or wired network. The terminal acquires low-resolution or even extremely low-resolution videos, then uses the action recognition model to identify the actions in these videos, determine the action tags, and then sends these videos to the cloud server. The cloud server can then classify and store the videos based on the action tags. The video can be related to a target object, which can be a person, an object, or a vehicle, etc., without specific limitations.
[0038] Terminal 101 is at least one of a mobile phone, tablet computer, and PC (Personal Computer) device. Server 102 can be at least one of a single server, a server cluster consisting of multiple servers, a cloud server, a cloud computing platform, and a virtualization center.
[0039] Figure 2 This is a flowchart of an action recognition model training method provided in an embodiment of this application. See also... Figure 2 The method includes:
[0040] Step 201: Obtain multiple sample video clips, each labeled with an action tag. The action tag is the tag corresponding to the action in the sample video clip.
[0041] Step 202: Based on multiple sample video segments, determine the comparison label corresponding to each pair of sample video segments. The comparison label is used to represent the comparison relationship between the two sample video segments. At least one of the sample video segments has a resolution lower than the baseline resolution.
[0042] Step 203: Based on every two sample video segments and their corresponding action labels and contrast labels, train the model to obtain the action recognition model.
[0043] In one possible implementation, based on multiple sample video segments, a comparison label is determined for every two sample video segments, including:
[0044] For every two sample video segments, determine at least one of the temporal comparison label and the spatial comparison label corresponding to the two sample video segments. The temporal comparison label is used to represent the temporal comparison relationship between the two sample video segments, and the spatial comparison label is used to represent the resolution comparison relationship between the two sample video segments.
[0045] At least one of the temporal comparison label and the spatial comparison label is determined as the comparison label corresponding to the two sample video segments.
[0046] In another possible implementation, determining the temporal comparison labels corresponding to the two sample video segments includes:
[0047] If two sample video segments come from the same sample video, the temporal comparison label is determined to be a positive temporal comparison label, and the resolution of the sample video is lower than the baseline resolution;
[0048] If two sample video segments come from different sample videos, the temporal comparison label is determined to be the temporal comparison negative label.
[0049] In another possible implementation, spatial contrast labels corresponding to two sample video segments are determined, including:
[0050] If two sample video clips come from sample videos with the same content but different resolutions, the spatial contrast label is determined to be a positive spatial contrast label.
[0051] If two sample video clips come from sample videos with different content or sample videos with the same resolution, the spatial contrast label is determined to be a negative spatial contrast label.
[0052] In another possible implementation, the model is trained based on every two sample video segments and their corresponding action labels and contrast labels to obtain an action recognition model, including:
[0053] Based on each pair of sample video segments and their corresponding contrast labels, the contrast loss parameter is determined. The contrast loss parameter is used to represent the loss parameter based on the contrast labels during the model training process.
[0054] Based on every two sample video segments and their corresponding action labels, the classification loss parameters are determined. The classification loss parameters are used to represent the loss parameters based on action labels during the model training process.
[0055] The action recognition model is obtained by training the model based on the contrastive loss parameters and the classification loss parameters.
[0056] In another possible implementation, the comparison labels include at least one of temporal comparison labels and spatial comparison labels;
[0057] Based on every two sample video segments and their corresponding contrast labels, the contrast loss parameters are determined, including:
[0058] Based on at least one of the temporal comparison label and spatial comparison label corresponding to each pair of sample video segments, feature extraction is performed on the two sample video segments respectively to obtain at least one of the first temporal feature vector pair and the first spatial feature vector pair.
[0059] Based on at least one of the first temporal feature vector pair and the first spatial feature vector pair, determine at least one of the temporal loss parameter and the spatial loss parameter;
[0060] The contrast loss parameter is determined based on at least one of the temporal loss parameter and the spatial loss parameter.
[0061] In another possible implementation, based on at least one of the temporal comparison label and spatial comparison label corresponding to each pair of sample video segments, feature extraction is performed on the two sample video segments respectively to obtain at least one of the first temporal feature vector pair and the first spatial feature vector pair, including:
[0062] Based on at least one of the temporal comparison label and spatial comparison label corresponding to each pair of sample video segments, feature extraction is performed on the two sample video segments respectively to obtain at least one of the second temporal feature vector pair and the second spatial feature vector pair.
[0063] Dimensionality reduction is performed on at least one of the second time-domain feature vector pair and the second spatial feature vector pair to obtain at least one of the first time-domain feature vector pair and the first spatial feature vector pair.
[0064] In another possible implementation, the comparison labels include: temporal comparison labels and spatial comparison labels;
[0065] Based on at least one of the temporal loss parameters and spatial loss parameters, determine the contrast loss parameters, including:
[0066] Obtain the pre-set temporal comparison weights and spatial comparison weights;
[0067] Based on the temporal contrast weight and the spatial contrast weight, the temporal loss parameter and the spatial loss parameter are weighted and summed to obtain the contrast loss parameter.
[0068] In another possible implementation, the process of determining the temporal loss parameters based on the first temporal feature vector pair includes:
[0069] The first vector distance is determined based on the two time-domain feature vectors in the first time-domain feature vector pair;
[0070] Determine the time-domain comparison label value based on the time-domain comparison label;
[0071] Obtain a pre-set first edge distance parameter, which represents the maximum distance between two time-domain feature vectors in the first time-domain feature vector pair;
[0072] The temporal loss parameters are determined based on the first vector distance, the temporal comparison label value, and the first edge distance parameter.
[0073] In another possible implementation, the model is trained based on the contrastive loss parameter and the classification loss parameter to obtain the action recognition model, including:
[0074] The total loss parameter is determined based on the contrast loss parameter and the classification loss parameter;
[0075] Based on the total loss parameter, the model is trained to obtain the action recognition model.
[0076] In another possible implementation, the method also includes:
[0077] Acquire the target video to be identified, where the resolution of the target video is lower than the baseline resolution;
[0078] Extract target video segments from the target video;
[0079] The target video clip is input into the action recognition model to obtain the action labels corresponding to the actions in the target video.
[0080] This application provides a method for training an action recognition model. The method trains the model by determining the contrast labels corresponding to every two sample video segments. Since the contrast labels can represent the contrast relationship between the two sample video segments, and at least one of the sample video segments has a resolution lower than the baseline resolution, the model can learn the resolution features of the low-resolution video segments based on the contrast labels during training. At the same time, it can also learn the action features in the low-resolution video based on the action labels of the sample video segments. Therefore, the trained action recognition model can accurately identify actions in low-resolution videos, thereby improving the accuracy of action recognition in low-resolution videos.
[0081] Figure 3 This is a flowchart of an action recognition model training method provided in an embodiment of this application, executed by an electronic device. See also... Figure 3 The method includes:
[0082] Step 301: The electronic device acquires multiple sample video clips.
[0083] For each sample video segment, an action label is provided, which is the label corresponding to the action in the sample video segment.
[0084] In this step, the electronic device can first acquire multiple sample videos, which are videos with a resolution higher than the reference resolution. Then, it performs at least one of temporal transformation and spatial transformation on the multiple sample videos to obtain multiple sample video segments. Some of these sample video segments have a resolution higher than the reference resolution, while others have a resolution lower than the reference resolution.
[0085] When converting sample videos, electronic devices can extract a number of consecutive video frames of a preset number of frames from the sample video according to action tags, that is, extract a number of consecutive video frames of the same action from the sample video to obtain a sample video segment. Alternatively, they can extract a number of consecutive video frames of a preset number of frames from the sample video according to resolution, that is, extract a number of preset video frames of the same action at different resolutions from the sample video to obtain a sample video segment.
[0086] The reference resolution can be set and changed as needed. In this embodiment, the reference resolution includes, but is not limited to, 12*16, 24*32, etc.
[0087] Step 302: For every two sample video segments, the electronic device determines at least one of the temporal contrast label and spatial contrast label corresponding to the two sample video segments.
[0088] Temporal contrast labels are used to indicate the temporal correlation between two sample video clips, while spatial contrast labels are used to indicate the resolution correlation between two sample video clips, where at least one of the sample video clips has a resolution lower than the baseline resolution.
[0089] In this step, the electronic device can first obtain a sample video segment with a resolution lower than the baseline resolution from multiple sample video segments. Then, it can traverse the remaining multiple sample video segments and combine each of the remaining multiple sample video segments with the original sample video segment in turn. When the remaining multiple sample video segments have been traversed, a new sample video segment with a resolution lower than the baseline resolution is obtained from the multiple sample video segments. Then, it can traverse the remaining multiple sample video segments except for the sample video segments with a resolution lower than the baseline resolution, and continue to combine them in the above way until every two sample video segments are combined.
[0090] The process by which an electronic device determines the temporal comparison label corresponding to the two sample video segments can be as follows: if the two sample video segments come from the same sample video, the temporal comparison label is determined to be a positive temporal comparison label, and the resolution of the sample video is lower than the baseline resolution; if the two sample video segments come from different sample videos, the temporal comparison label is determined to be a negative temporal comparison label.
[0091] In this implementation, different video segments from the same sample video, i.e., different stages of the same sample video, should tend to be represented in a compact space after being learned by the neural network. Based on this, if two video segments are video segments from different times within the same sample video, it means that these two video segments are different stages of the same action, and these two video segments constitute positive labels for temporal contrast. If these two video segments come from different sample videos, it means that these two video segments are video segments of different actions, and these two video segments constitute negative labels for temporal contrast.
[0092] The process by which an electronic device determines the spatial contrast label corresponding to the two sample video segments can be as follows: if the two sample video segments come from sample videos with the same content but different resolutions, the spatial contrast label is determined to be a positive spatial contrast label; if the two sample video segments come from sample videos with different content or sample videos with the same resolution, the spatial contrast label is determined to be a negative spatial contrast label.
[0093] In this implementation, sample videos of different resolutions, after being learned by a neural network, should tend to converge in a compact space. Based on this, if two sample video clips originate from sample videos with the same content but different resolutions, it means that these two sample video clips are video clips of the same action at different resolutions, that is, a high-resolution video clip and a low-resolution video clip of the same action. In this case, these two sample video clips constitute a spatial contrast positive label.
[0094] If the two sample video clips come from sample videos with different content, it means that the two sample videos are low-resolution videos with different actions, or that the two sample videos are a low-resolution video and a high-resolution video with different actions. If the two sample video clips come from sample videos with the same resolution, since at least one of the two sample video clips is a low-resolution video clip, it means that the two sample video clips come from low-resolution sample videos, and the two sample video clips constitute spatial contrast negative labels.
[0095] See Figure 4 , Figure 4 The two sample video clips in the lower center constitute positive temporal contrast labels, while the two sample video clips in the upper right and lower left constitute negative temporal contrast labels. The two sample video clips in the upper right and lower right also constitute negative temporal contrast labels. See also... Figure 5 , Figure 5 The two sample video clips in the lower middle constitute positive spatial contrast labels, while the two sample video clips in the upper right and lower left constitute negative spatial contrast labels.
[0096] Step 303: The electronic device determines at least one of the temporal comparison label and the spatial comparison label as the comparison label corresponding to the two sample video segments.
[0097] This comparison label is used to indicate the contrast relationship between the two sample video segments. The comparison label can be a temporal comparison label, a spatial comparison label, or a combination of both; there are no specific limitations.
[0098] Step 304: The electronic device determines the contrast loss parameter based on every two sample video segments and their corresponding contrast labels.
[0099] The contrastive loss parameter is used to represent the loss parameter during model training based on the contrastive label.
[0100] This step can be achieved through the following steps (1) to (3), including:
[0101] (1) The electronic device performs feature extraction on the two sample video segments based on at least one of the temporal comparison label and spatial comparison label corresponding to each pair of sample video segments, and obtains at least one of the first temporal feature vector pair and the first spatial feature vector pair.
[0102] The electronic device performs feature extraction on the two sample video segments based on at least one of the temporal comparison label and spatial comparison label corresponding to the two sample video segments, and obtains at least one of the second temporal feature vector pair and the second spatial feature vector pair; and performs dimensionality reduction processing on at least one of the second temporal feature vector pair and the second spatial feature vector pair to obtain at least one of the first temporal feature vector pair and the first spatial feature vector pair.
[0103] In this step, the electronic device obtains the first temporal feature vector pair by: based on the temporal comparison labels corresponding to the two sample video segments, the electronic device extracts temporal features from the two sample video segments respectively to obtain the second temporal feature vector pair, and then maps the two first-dimensional temporal feature vectors in the second temporal feature vector pair to two first-preset-dimensional temporal feature vectors to obtain the first temporal feature vector pair.
[0104] In this implementation, the electronic device can obtain a first temporal feature vector pair through a neural network. The neural network includes a feature extraction module and a self-supervised learning module. The electronic device can input the two sample video segments into the neural network based on the temporal comparison labels corresponding to the two sample video segments. The feature extraction module in the neural network extracts temporal features to obtain a second temporal feature vector pair. Then, the feature extraction module inputs the second temporal feature vector pair into the self-supervised learning module. The self-supervised learning module is a multi-layer fully connected layer that can map the temporal feature vectors into a compact space, that is, map the two first-dimensional temporal feature vectors into two first-preset-dimensional temporal feature vectors to obtain the first temporal feature vector pair.
[0105] The process by which the electronic device obtains the first spatial feature vector pair can be as follows: Based on the spatial comparison labels corresponding to the two sample video segments, the electronic device extracts spatial features from the two sample video segments respectively to obtain the second spatial feature vector pair. Then, the two second-dimensional spatial feature vectors in the second spatial feature vector pair are mapped to two second-preset-dimensional spatial feature vectors to obtain the first spatial feature vector pair.
[0106] In this implementation, the electronic device can also input the two sample video segments into a neural network based on the spatial comparison labels corresponding to the two sample video segments. The feature extraction module in the neural network extracts spatial features to obtain a second spatial feature vector pair. Then, the feature extraction module inputs the second spatial feature vector pair into a self-supervised learning module. The self-supervised learning module can map the spatial feature vectors into a compact space, that is, map the two second-dimensional spatial feature vectors into two second-preset-dimensional spatial feature vectors to obtain a first spatial feature vector pair.
[0107] The number of fully connected layers can be set and changed as needed; for example, the number of fully connected layers can be 2. The first preset dimension is smaller than the first dimension, and the second preset dimension is smaller than the second dimension. The first preset dimension and the second preset dimension can be the same or different, and both can be set and changed as needed. For example, the first preset dimension and the second preset dimension can be the same, both being 128 dimensions.
[0108] See Figure 6 ,from Figure 6As can be seen, the feature extraction module includes three branches. The first branch corresponds to sample video segments with a resolution lower than the baseline resolution. The second branch corresponds to another sample video segment that forms a temporal comparison label with this sample video segment. The third branch corresponds to another sample video segment that forms a spatial comparison label with this sample video segment. These three branches have the same structure and parameters. After the feature extraction module extracts the temporal and spatial features, it inputs the second temporal feature vector pair corresponding to the two sample video segments that form the temporal comparison label into the temporal learning module of the self-supervised learning module. Through two fully connected layers of this temporal learning module, the second temporal feature vector pair is mapped into a compact space to obtain the first temporal feature vector pair. Similarly, the second spatial feature vector pair corresponding to the two sample video segments that form the spatial comparison label is input into the spatial learning module of the self-supervised learning module. Through two fully connected layers of this spatial learning module, the second spatial feature vector pair is mapped into a compact space to obtain the first spatial feature vector pair.
[0109] In the embodiments of this application, it can be achieved through This represents the modeling of sample video segments with resolutions lower than the baseline resolution by the feature extraction module, through... Let represent the modeling of sample video segments with a resolution higher than the baseline resolution by the feature extraction module, and let s(:) represent the self-supervised learning module. Then, the final learned representation g(V, X; θ) in the embedding space can be expressed as: x = g(V, X; θ) = s(f(V, X; θ)). Here, i represents the action label, j represents the order of the sample video segment among the multiple sample video segments corresponding to the action label (where j is a positive integer), k represents the stage of the sample video segment in its corresponding sample video, and θ represents the parameters of the fully connected layer.
[0110] See Figure 7 ,from Figure 7 As can be seen from (a): when the time-domain learning module s in the self-supervised learning module is... t (:) Input time-domain comparison of the second time-domain feature vector corresponding to the positive label. At that time, the first temporal feature vector pair corresponding to the positive label in the temporal domain is obtained. and Where k1 and k2 represent the different stages of two sample video segments within their corresponding sample videos. Figure 7 As can be seen from (c): when the time-domain learning module s in the self-supervised learning module is... t (:) Input time-domain comparison of the second time-domain feature vector corresponding to the negative label. At that time, the first temporal feature vector pair corresponding to the temporal comparison negative label is obtained. and Here, i1 and i2 represent different action labels. From Figure 7 As can be seen in (b): when the spatial learning module s in the self-supervised learning module is... s (:) Input space comparison with the second space feature vector pair corresponding to the positive label. At that time, the first spatial feature vector pair corresponding to the positive label in spatial comparison is obtained. and from Figure 7 As can be seen from (d): when the spatial learning module s in the self-supervised learning module is... s (:) Input space comparison with the second space feature vector pair corresponding to the negative label. At that time, the first spatial feature vector pair corresponding to the spatial comparison negative label is obtained. and
[0111] In this embodiment, the neural network structure of the feature extraction module can be any fully convolutional module, including but not limited to ResNet (Deep Residual Network), VGG (Visual Geometry Group, Deep Convolutional Neural Network), and MobileNet (Lightweight Network). The self-supervised learning method can also be adjusted, including but not limited to the method of using two fully connected layers to map to a compact space involved in this embodiment.
[0112] (2) The electronic device determines at least one of the time domain loss parameters and the spatial loss parameters based on at least one of the first time domain feature vector pair and the first spatial feature vector pair.
[0113] In this step, the process of determining the time-domain loss parameters of the electronic device can be achieved through the following steps (A-1) to (A-4), including:
[0114] (A-1) The electronic device determines the first vector distance based on the two time-domain feature vectors in the first time-domain feature vector pair.
[0115] The electronic device determines the vector difference between two time-domain feature vectors in the first time-domain feature vector pair, determines the norm corresponding to this vector difference, and obtains the first vector distance, which can be represented as: ||x p -x q ||, where x p and x q They can represent two temporal feature vectors for temporal comparison of positive labels, or they can represent two temporal feature vectors for temporal comparison of negative labels.
[0116] (A-2) Electronic devices determine time-domain comparison tag values based on time-domain comparison tags.
[0117] If the time-domain contrast label is positive, the time-domain contrast label value is 1; if the time-domain contrast label is negative, the time-domain contrast label value is 0. This time-domain contrast label value can be represented as: y′ (p，q) .
[0118] (A-3) The electronic device acquires a pre-set first edge distance parameter.
[0119] The first edge distance parameter is used to represent the maximum distance between the two time-domain feature vectors in the first time-domain feature vector pair. See also... Figure 7 (c), Figure 7 The diameter of the circle shown in (c) is the first edge distance parameter.
[0120] (A-4) The electronic device determines the temporal loss parameter based on the first vector distance, the temporal comparison label value, and the first edge distance parameter.
[0121] Based on the first vector distance, the temporal domain contrast label value, and the first edge distance parameter, the temporal domain loss parameter of the electronic device can be expressed by the following formula:
[0122]
[0123] Among them, L t Let represent the temporal loss parameter, T represent the subset size of the temporal contrast labels, m1 be the first edge distance parameter, and max(0, m1-||x||). p -x q ||) represents taking the values 0 and m1-||x p -x q The maximum value in ||.
[0124] In this step, the process of determining the space loss parameters by the electronic device can be achieved through the following steps (B-1) to (B-4):
[0125] (B-1) The electronic device determines the second vector distance based on two spatial feature vectors in the first spatial feature vector pair.
[0126] (B-2) Electronic devices determine spatial comparison tag values based on spatial comparison tags.
[0127] (B-3) The electronic device acquires a pre-set second edge distance parameter.
[0128] The second edge distance parameter is used to represent the maximum distance between two spatial feature vectors in the first spatial feature vector pair. See also... Figure 7 (d), Figure 7 The diameter of the circle shown in (d) is the second edge distance parameter.
[0129] (B-4) The electronic device determines the spatial loss parameter based on the second vector distance, the spatial comparison label value, and the second edge distance parameter.
[0130] The space loss parameter can be expressed as:
[0131]
[0132] Among them, L s Let S represent the spatial loss parameter, S represent the subset size of the spatial contrast labels, m2 represent the second edge distance parameter, and ||x p′ -x q′ || represents the distance between the second vectors, x p′ and x q′ It can represent two spatial feature vectors for spatial contrast positive labels, or it can represent two spatial feature vectors for spatial contrast negative labels, y′ (p′，q′) This indicates a spatial comparison label value.
[0133] The process of determining the spatial loss parameters for electronic devices is similar to the process of determining the temporal loss parameters described above, and will not be repeated here.
[0134] (3) The electronic device determines the contrast loss parameter based on at least one of the temporal loss parameter and the spatial loss parameter.
[0135] If the comparison label only includes a temporal comparison label, the electronic device determines the temporal loss parameter as the comparison loss parameter. If the comparison label only includes a spatial comparison label, the electronic device determines the spatial loss parameter as the comparison loss parameter. If the comparison label includes both temporal and spatial comparison labels, the electronic device determines the comparison loss parameter based on the temporal and spatial loss parameters. This process can be as follows: the electronic device obtains pre-set temporal and spatial comparison weights; based on the temporal and spatial comparison weights, it performs a weighted summation of the temporal and spatial loss parameters to obtain the comparison loss parameter.
[0136] For example, the contrast loss parameter can be expressed as: L = μ1L t +μ2L s , where L represents the contrast loss parameter, and μ1 and μ2 represent the temporal contrast weight and spatial contrast weight, respectively.
[0137] In the embodiments of this application, m1, m2, μ1 and μ2 are all preset hyperparameters.
[0138] It should be noted that the model training process aims to minimize the embedding space distance between positive labels and maximize the embedding space distance between negative labels. The positive labels are either temporal contrast positive labels or spatial contrast positive labels, and the negative labels are either temporal contrast negative labels or spatial contrast negative labels.
[0139] Step 305: The electronic device determines the classification loss parameters based on every two sample video segments and their corresponding action labels.
[0140] The classification loss parameter is used to represent the loss parameters based on the action label during model training.
[0141] In this step, the neural network also includes a classification module, which can also use fully connected layers to classify actions in the sample video clips. The electronic device can determine the loss between the action labels based on the action labels determined by the classification module for the two sample video clips and the actual action labels labeled for the two sample video clips, thus obtaining the classification loss parameters.
[0142] In this context, the classification loss parameter for electronic devices can be represented by the cross-entropy function.
[0143] Step 306: The electronic device trains the model based on the contrast loss parameter and the classification loss parameter to obtain the action recognition model.
[0144] The electronic device determines the total loss parameter based on the contrast loss parameter and the classification loss parameter, and then trains the model based on the total loss parameter to obtain the action recognition model.
[0145] The electronic device can directly determine the sum of the contrast loss parameter and the classification loss parameter to obtain the total loss parameter, or it can obtain the weights of the contrast loss parameter and the classification loss parameter separately, and then perform a weighted sum based on their respective weights to obtain the total loss parameter. No specific limitation is made on this.
[0146] During the model training process based on the total loss parameter, when the number of iterations reaches the preset number of iterations or the convergence condition is met, an action recognition model is obtained. This action recognition model is used to recognize actions in low-resolution or even very low-resolution videos and is a model that can be trained end-to-end.
[0147] See Figure 8 ,from Figure 8 As can be seen, the electronic device first acquires high-resolution sample videos from the training subset, performs at least one of temporal and spatial transformations on the sample videos to obtain transformed sample video segments, then extracts temporal and / or spatial features from the transformed sample video segments through a feature extraction module, and then performs self-supervised learning through a self-supervised learning module to determine the contrastive loss parameters. Furthermore, a classification module classifies the sample video segments to determine the classification loss parameters. Finally, the model is trained based on the contrastive and classification loss parameters to obtain the action recognition model.
[0148] In this embodiment, a temporal contrast label is introduced through self-supervised learning. This label enables the neural network to determine whether two sample video segments originate from different intervals of the same low-resolution or even extremely low-resolution sample video, effectively learning identifiable features under low-resolution or extremely low-resolution conditions. Furthermore, a spatial contrast label is also introduced through self-supervised learning. This label enables the neural network to determine whether a low-resolution or extremely low-resolution video segment in two sample video segments is a downsampled version of its corresponding high-resolution video segment. This spatial embedding learning also allows the neural network to effectively learn low-resolution data representations.
[0149] This method introduces self-supervised learning into the process of action recognition in low-resolution or even very low-resolution videos. By learning two types of contrast labels, temporal contrast labels and spatial contrast labels, the features of low-resolution or even very low-resolution videos can be effectively represented, which helps to identify actions in low-resolution or even very low-resolution videos and thus improves the recognition accuracy in low-resolution or even very low-resolution situations.
[0150] Furthermore, related technologies require optical flow data to recognize actions in videos, but acquiring optical flow data is time-consuming and labor-intensive, hindering the implementation of the model. The method provided in this application, however, does not require optical flow data as input and still achieves high recognition accuracy.
[0151] In addition, electronic devices can introduce more contrast labels to learn the features of low-resolution or even very low-resolution videos, thereby further improving recognition accuracy.
[0152] This application provides a method for training an action recognition model. The method trains the model by determining the contrast labels corresponding to every two sample video segments. Since the contrast labels can represent the contrast relationship between the two sample video segments, and at least one of the sample video segments has a resolution lower than the baseline resolution, the model can learn the resolution features of the low-resolution video segments based on the contrast labels during training. At the same time, it can also learn the action features in the low-resolution video based on the action labels of the sample video segments. Therefore, the trained action recognition model can accurately identify actions in low-resolution videos, thereby improving the accuracy of action recognition in low-resolution videos.
[0153] For ease of distinction, the electronic device used to train the action recognition model is referred to as the first electronic device, and the electronic device in this embodiment is referred to as the second electronic device. The first electronic device and the second electronic device can be the same electronic device or different electronic devices. If the first electronic device is provided as a terminal, the second electronic device can be the same as the first electronic device and also provided as the terminal. If the first electronic device is provided as a server, the second electronic device can be provided as a terminal, and the terminal is connected to the server via a wireless or wired network. If the first electronic device is provided as both a terminal and a server, the second electronic device can be provided as the terminal within the first electronic device. In this embodiment, no specific limitations are made.
[0154] Figure 9 This is a flowchart of an action recognition method provided in an embodiment of this application, executed by a second electronic device. See also... Figure 9 The method includes:
[0155] Step 901: The second electronic device acquires the target video to be identified.
[0156] The target video has a lower resolution than the baseline resolution.
[0157] Step 902: The second electronic device extracts the target video segment from the target video.
[0158] The resolution of the target video clip is also lower than the baseline resolution.
[0159] The second electronic device can randomly extract a preset number of consecutive video frames from the target video to obtain the target video segment. Alternatively, it can divide the target video into multiple video segments in chronological order based on the preset number of frames and then designate any one of these video segments as the target video segment, or designate the video segment located in the middle as the target video segment. Of course, the second electronic device can also extract the target video segment from the target video using other methods, without specific limitations.
[0160] Step 903: The second electronic device inputs the target video segment into the action recognition model to obtain the action tags corresponding to the actions in the target video.
[0161] The second electronic device inputs the target video clip into the action recognition model, identifies the actions in the target video clip through the action recognition model, determines the action tag corresponding to the action, and uses the action tag as the action tag corresponding to the action in the target video.
[0162] It should be noted that the second electronic device can also directly input the target video into the action recognition model after acquiring the target video. The action recognition model will then extract the target video segments and identify the actions within those segments to obtain action tags. Accordingly, steps 902 and 903 can be replaced with: the second electronic device inputs the target video into the action recognition model to obtain the action tags corresponding to the actions in the target video.
[0163] This application provides an action recognition method. The method uses an action recognition model to identify actions in low-resolution videos. Since the model learns the resolution features of low-resolution video segments based on contrast labels and the action features of low-resolution video segments based on action labels during training, the model can accurately identify actions in low-resolution videos, thereby improving the accuracy of action recognition in low-resolution videos.
[0164] Figure 10 This is a schematic diagram of the structure of an action recognition model training device provided in an embodiment of this application. See also... Figure 10 The device includes:
[0165] The first acquisition module 1001 is used to acquire multiple sample video segments, which are labeled with action tags. The action tags are the tags corresponding to the actions in the sample video segments.
[0166] The first determining module 1002 is used to determine the comparison label corresponding to every two sample video segments based on multiple sample video segments. The comparison label is used to indicate the comparison relationship between the two sample video segments. At least one of the two sample video segments has a resolution lower than the reference resolution.
[0167] Training module 1003 is used to train the model based on every two sample video segments and their corresponding action labels and contrast labels to obtain the action recognition model.
[0168] In one possible implementation, the first determining module 1002 is used to determine, for every two sample video segments, at least one of a temporal comparison label and a spatial comparison label corresponding to the two sample video segments. The temporal comparison label is used to represent the temporal comparison relationship between the two sample video segments, and the spatial comparison label is used to represent the resolution comparison relationship between the two sample video segments. At least one of the temporal comparison label and the spatial comparison label is determined as the comparison label corresponding to the two sample video segments.
[0169] In another possible implementation, the device also includes:
[0170] The second determining module is used to determine the temporal comparison label as a positive temporal comparison label if the two sample video segments come from the same sample video, and the resolution of the sample video is lower than the baseline resolution; and to determine the temporal comparison label as a negative temporal comparison label if the two sample video segments come from different sample videos.
[0171] In another possible implementation, the device also includes:
[0172] The third determination module is used to determine the spatial comparison label as a positive spatial comparison label if two sample video clips come from sample videos with the same content but different resolutions; and to determine the spatial comparison label as a negative spatial comparison label if two sample video clips come from sample videos with different content or sample videos with the same resolution.
[0173] In another possible implementation, the training module 1003 is used to determine a contrast loss parameter based on every two sample video segments and their corresponding contrast labels. The contrast loss parameter represents the loss parameter based on the contrast labels during model training. The classification loss parameter is used to determine a classification loss parameter based on every two sample video segments and their corresponding action labels. The classification loss parameter represents the loss parameter based on the action labels during model training. The model is trained based on the contrast loss parameter and the classification loss parameter to obtain the action recognition model.
[0174] In another possible implementation, the comparison labels include at least one of temporal comparison labels and spatial comparison labels;
[0175] The training module 1003 is used to extract features from two sample video segments based on at least one of the temporal contrast label and spatial contrast label corresponding to each pair of sample video segments, to obtain at least one of the first temporal feature vector pair and the first spatial feature vector pair; to determine at least one of the temporal loss parameter and the spatial loss parameter based on at least one of the first temporal feature vector pair and the first spatial feature vector pair; and to determine the contrast loss parameter based on at least one of the temporal loss parameter and the spatial loss parameter.
[0176] In another possible implementation, the training module 1003 is used to extract features from two sample video segments based on at least one of the temporal contrast label and spatial contrast label corresponding to each pair of sample video segments, to obtain at least one of the second temporal feature vector pair and the second spatial feature vector pair; and to perform dimensionality reduction processing on at least one of the second temporal feature vector pair and the second spatial feature vector pair to obtain at least one of the first temporal feature vector pair and the first spatial feature vector pair.
[0177] In another possible implementation, the comparison labels include: temporal comparison labels and spatial comparison labels;
[0178] Training module 1003 is used to obtain pre-set temporal contrast weights and spatial contrast weights; based on the temporal contrast weights and spatial contrast weights, the temporal loss parameters and spatial loss parameters are weighted and summed to obtain the contrast loss parameters.
[0179] In another possible implementation, the device also includes:
[0180] The fourth determining module is used to determine the vector distance based on the two time-domain feature vectors in the first time-domain feature vector pair; determine the time-domain comparison label value based on the time-domain comparison label; obtain a pre-set first edge distance parameter, which is used to represent the range of the time-domain feature vector; and determine the time-domain loss parameter based on the vector distance, the time-domain comparison label value and the first edge distance parameter.
[0181] In another possible implementation, the training module 1003 is used to determine the total loss parameter based on the contrastive loss parameter and the classification loss parameter; and to train the model based on the total loss parameter to obtain the action recognition model.
[0182] In another possible implementation, the device also includes:
[0183] The second acquisition module is used to acquire the target video to be identified, the resolution of which is lower than the reference resolution;
[0184] The extraction module is used to extract target video segments from the target video;
[0185] The recognition module is used to input the target video clip into the action recognition model to obtain the action labels corresponding to the actions in the target video.
[0186] This application provides an action recognition model training device. The device trains the model by determining the contrast labels corresponding to every two sample video segments. Since the contrast labels can represent the contrast relationship between the two sample video segments, and at least one of the sample video segments has a resolution lower than the baseline resolution, the model can learn the resolution features of the low-resolution video segments based on the contrast labels during training. At the same time, it can also learn the action features in the low-resolution video based on the action labels of the sample video segments. Therefore, the trained action recognition model can accurately identify actions in low-resolution videos, thereby improving the accuracy of action recognition in low-resolution videos.
[0187] If the electronic device is provided as a terminal, please refer to... Figure 11This diagram illustrates a structural block diagram of a terminal 1100 according to an exemplary embodiment of this application. The terminal 1100 can be a smartphone, tablet computer, or other device with the function of controlling other devices. The terminal 1100 in this application may include one or more of the following components: a processor 1110 and a memory 1120.
[0188] Processor 1110 may include one or more processing cores. Processor 1110 connects to various parts within terminal 1100 using various interfaces and lines, and performs various functions and processes data of terminal 1100 by running or executing program code, programs, code sets, or program code sets stored in memory 1120, and by calling data stored in memory 1120. Optionally, processor 1110 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). Processor 1110 may integrate one or more of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), Neural-network Processing Unit (NPU), and modem. Specifically, the CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display on the screen; the NPU is used to implement Artificial Intelligence (AI) functions; and the modem is used for wireless communication. It is understandable that the aforementioned modem may not be integrated into the processor 1110, but may be implemented using a separate chip.
[0189] The memory 1120 may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory 1120 may include a non-transitory computer-readable storage medium. The memory 1120 may be used to store program code, programs, code, code sets, or program code sets. The memory 1120 may include a program storage area and a data storage area, wherein the program storage area may store program code for implementing an operating system, program code for at least one function (such as touch function, sound playback function, image playback function, etc.), program code for implementing the various method embodiments described above, etc.; the data storage area may store data created according to the use of the terminal 1100 (such as audio data, phone book, etc.).
[0190] In addition, those skilled in the art will understand that the structure of the terminal 1100 shown in the above figures does not constitute a limitation on the terminal 1100. The terminal 1100 may include more or fewer components than shown, or combine certain components, or have different component arrangements. For example, the terminal 1100 may also include a microphone, speaker, radio frequency circuit, input unit, sensor, audio circuit, Wireless Fidelity (Wi-Fi) module, power supply, Bluetooth module, etc., which will not be described in detail here.
[0191] If the electronic device is provided as a server, see [link to relevant documentation]. Figure 12 , Figure 12 This is a structural block diagram of a server provided in an embodiment of this application. The server 1200 can vary significantly due to different configurations or performance. It may include a processor (central processing unit, CPU) 1201 and a memory 1202. The memory 1202 stores at least one line of program code, which is loaded and executed by the processor 1201 to implement the methods provided in the above-described method embodiments. Of course, the server 1200 may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server 1200 may also include other components for implementing device functions, which will not be elaborated here.
[0192] If the electronic device is provided as a terminal and a server, the block diagrams of the terminal and server can be found separately. Figure 11 and Figure 12 .
[0193] In an exemplary embodiment, a computer-readable medium is also provided, which stores at least one piece of program code that is loaded and executed by a processor to implement the action recognition model training method in the above embodiments.
[0194] In an exemplary embodiment, a computer program product is also provided, which stores at least one piece of program code, which is loaded and executed by a processor to implement the action recognition model training method in the above embodiments.
[0195] In some embodiments, the computer program involved in the present application embodiments may be deployed and executed on a computer device, or executed on multiple computer devices located in one location, or executed on multiple computer devices distributed in multiple locations and interconnected through a communication network. Multiple computer devices distributed in multiple locations and interconnected through a communication network may constitute a blockchain system.
[0196] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.
[0197] The above description is only for the purpose of enabling those skilled in the art to understand the technical solution of this application, and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.
Claims
1. A method for training an action recognition model, characterized in that, The method includes: Multiple sample video clips are acquired, and each sample video clip is labeled with an action tag, which is a tag corresponding to an action in the sample video clip; Based on the plurality of sample video segments, a comparison label is determined for every two sample video segments. One of the two sample video segments has a resolution lower than the baseline resolution, and the other has a resolution higher than the baseline resolution. The comparison label includes a temporal comparison label and a spatial comparison label. The temporal comparison label includes a positive temporal comparison label and a negative temporal comparison label. The positive temporal comparison label is used to indicate that the two sample video segments are different stages of the same action, and the negative temporal comparison label is used to indicate that the two sample video segments are video segments of different actions. The spatial comparison label includes a positive spatial comparison label and a negative spatial comparison label. The positive spatial comparison label is used to indicate that the two sample video segments are video segments of the same action at different resolutions, and the negative spatial comparison label is used to indicate that the two sample video segments are video segments of different actions at different resolutions. Based on the temporal and spatial contrast labels corresponding to each pair of sample video segments, feature extraction is performed on the two sample video segments respectively to obtain a first temporal feature vector pair and a first spatial feature vector pair; based on the first temporal and spatial feature vector pairs, temporal loss parameters and spatial loss parameters are determined; based on the temporal and spatial loss parameters, contrast loss parameters are determined, and the contrast loss parameters are used to represent the loss parameters based on the contrast labels during model training. Based on every two sample video segments and their corresponding action labels, a classification loss parameter is determined, which is used to represent the loss parameter based on the action labels during model training. Based on the contrast loss parameters and the classification loss parameters, the model is trained to obtain the action recognition model.
2. The method according to claim 1, characterized in that, The step of determining the comparison label corresponding to every two sample video segments based on the multiple sample video segments includes: For every two sample video segments, determine the temporal comparison label and spatial comparison label corresponding to the two sample video segments; The temporal comparison label and spatial comparison label are determined as the comparison labels corresponding to the two sample video segments.
3. The method according to claim 2, characterized in that, Determining the temporal comparison labels corresponding to the two sample video segments includes: If the two sample video segments originate from the same sample video, the temporal comparison label is determined to be a positive temporal comparison label, and the resolution of the sample video is lower than the baseline resolution; If the two sample video segments come from different sample videos, the temporal comparison label is determined to be a temporal comparison negative label.
4. The method according to claim 2, characterized in that, Determining the spatial contrast labels corresponding to the two sample video segments includes: If the two sample video segments originate from sample videos with the same content but different resolutions, the spatial comparison label is determined to be a positive spatial comparison label. If the two sample video segments originate from sample videos with different content or sample videos with the same resolution, the spatial contrast label is determined to be a negative spatial contrast label.
5. The method according to claim 1, characterized in that, The step of extracting features from the two sample video segments based on the temporal and spatial comparison labels corresponding to each pair of sample video segments to obtain a first temporal feature vector pair and a first spatial feature vector pair includes: Based on the temporal and spatial comparison labels corresponding to each pair of sample video segments, feature extraction is performed on the two sample video segments respectively to obtain a second temporal feature vector pair and a second spatial feature vector pair. The second temporal feature vector pair and the second spatial feature vector pair are subjected to dimensionality reduction processing to obtain the first temporal feature vector pair and the first spatial feature vector pair.
6. The method according to claim 1, characterized in that, The step of determining the contrast loss parameter based on the temporal loss parameter and the spatial loss parameter includes: Obtain the pre-set temporal comparison weights and spatial comparison weights; Based on the temporal contrast weight and spatial contrast weight, the temporal loss parameter and the spatial loss parameter are weighted and summed to obtain the contrast loss parameter.
7. The method according to claim 1, characterized in that, The process of determining the temporal loss parameters based on the first temporal feature vector pair includes: The first vector distance is determined based on the two time-domain feature vectors in the first time-domain feature vector pair; Based on the time-domain comparison labels, determine the time-domain comparison label values; Obtain a pre-set first edge distance parameter, which represents the maximum distance between two time-domain feature vectors in the first time-domain feature vector pair; The temporal loss parameter is determined based on the first vector distance, the temporal comparison label value, and the first edge distance parameter.
8. The method according to claim 1, characterized in that, The step of training the model based on the contrast loss parameter and the classification loss parameter to obtain the action recognition model includes: Based on the contrast loss parameter and the classification loss parameter, the total loss parameter is determined; Based on the total loss parameters, the model is trained to obtain the action recognition model.
9. The method according to claim 1, characterized in that, The method further includes: Obtain the target video to be identified, wherein the resolution of the target video is lower than the reference resolution; Extract the target video segment from the target video; The target video segment is input into the action recognition model to obtain the action tags corresponding to the actions in the target video.
10. A training device for an action recognition model, characterized in that, The device includes: The first acquisition module is used to acquire multiple sample video segments, wherein the sample video segments are labeled with action tags, and the action tags are the tags corresponding to the actions in the sample video segments. The first determining module is used to determine a comparison label corresponding to every two sample video segments based on the plurality of sample video segments. At least one of the two sample video segments has a resolution lower than the reference resolution. The comparison label includes a temporal comparison label and a spatial comparison label. The temporal comparison label includes a positive temporal comparison label and a negative temporal comparison label. The positive temporal comparison label is used to indicate that the two sample video segments are different stages of the same action. The negative temporal comparison label is used to indicate that the two sample video segments are video segments of different actions. The spatial comparison label includes a positive spatial comparison label and a negative spatial comparison label. The positive spatial comparison label is used to indicate that the two sample video segments are video segments of the same action at different resolutions. The negative spatial comparison label is used to indicate that the two sample video segments are video segments of different actions and both have a resolution lower than the reference resolution, or are video segments of different actions with one resolution higher than the reference resolution and the other lower than the reference resolution. The training module is used to extract features from two sample video segments based on their corresponding temporal and spatial contrast labels, obtaining a first temporal feature vector pair and a first spatial feature vector pair; determine temporal and spatial loss parameters based on the first temporal and spatial feature vector pairs; determine contrast loss parameters based on the temporal and spatial loss parameters, whereby the contrast loss parameters represent the loss parameters during model training based on the contrast labels; determine classification loss parameters based on two sample video segments and their corresponding action labels, whereby the classification loss parameters represent the loss parameters during model training based on the action labels; and perform model training based on the contrast loss parameters and the classification loss parameters to obtain the action recognition model.
11. The apparatus according to claim 10, characterized in that, The first determining module is used to determine, for every two sample video segments, the temporal comparison label and the spatial comparison label corresponding to the two sample video segments; and to determine the temporal comparison label and the spatial comparison label as the comparison label corresponding to the two sample video segments.
12. The apparatus according to claim 11, characterized in that, The device further includes: The second determining module is used to determine the temporal comparison label as a positive temporal comparison label if the two sample video segments originate from the same sample video, and the resolution of the sample video is lower than the reference resolution; and to determine the temporal comparison label as a negative temporal comparison label if the two sample video segments originate from different sample videos.
13. The apparatus according to claim 11, characterized in that, The device further includes: The third determining module is used to determine the spatial comparison label as a positive spatial comparison label if the two sample video segments originate from sample videos with the same content but different resolutions; and to determine the spatial comparison label as a negative spatial comparison label if the two sample video segments originate from sample videos with different content or sample videos with the same resolution.
14. The apparatus according to claim 10, characterized in that, The training module is used to extract features from the two sample video segments based on the temporal and spatial contrast labels corresponding to each pair of sample video segments, to obtain a second temporal feature vector pair and a second spatial feature vector pair; and to perform dimensionality reduction processing on the second temporal feature vector pair and the second spatial feature vector pair to obtain a first temporal feature vector pair and a first spatial feature vector pair.
15. The apparatus according to claim 10, characterized in that, The training module is used to obtain pre-set temporal contrast weights and spatial contrast weights; based on the temporal contrast weights and spatial contrast weights, the temporal loss parameter and the spatial loss parameter are weighted and summed to obtain the contrast loss parameter.
16. The apparatus according to claim 10, characterized in that, The device further includes: The fourth determining module is used to determine a first vector distance based on two time-domain feature vectors in the first time-domain feature vector pair; determine a time-domain comparison label value based on the time-domain comparison label; obtain a pre-set first edge distance parameter, which represents the maximum distance between two time-domain feature vectors in the first time-domain feature vector pair; and determine the time-domain loss parameter based on the first vector distance, the time-domain comparison label value, and the first edge distance parameter.
17. The apparatus according to claim 10, characterized in that, The training module is used to determine the total loss parameter based on the contrast loss parameter and the classification loss parameter; and to train the model based on the total loss parameter to obtain the action recognition model.
18. The apparatus according to claim 10, characterized in that, The device further includes: The second acquisition module is used to acquire the target video to be identified, wherein the resolution of the target video is lower than the reference resolution; The extraction module is used to extract target video segments from the target video; The recognition module is used to input the target video segment into the action recognition model to obtain the action tags corresponding to the actions in the target video.
19. An electronic device, characterized in that, The electronic device includes a processor and a memory, wherein the memory stores at least one piece of program code, which is loaded and executed by the processor to implement the action recognition model training method as described in any one of claims 1 to 9.
20. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one piece of program code, which is loaded and executed by a processor to implement the action recognition model training method as described in any one of claims 1 to 9.
21. A computer program product, characterized in that, The computer program product stores at least one piece of program code, which is loaded and executed by a processor to implement the action recognition model training method as described in any one of claims 1 to 9.