Method and apparatus for processing video image data

Abstract

The application discloses a method and device for processing video image data. A specific embodiment of the method comprises: performing the following training operation until a trained feature extraction network is obtained: selecting a video image from the video image data, and extracting feature information of the video image through an initial feature extraction network; determining layer feature information corresponding to each layer in a preset layer set corresponding to the feature information; for each layer in the preset layer set except a target layer, predicting category information to which the video image belongs under the layer, wherein the target layer represents a layer to which category information represented by a label of the video image belongs; and updating the initial feature extraction network according to a preset loss function, wherein the loss function is used to represent minimization of distances between the layer feature information and the category information corresponding to each layer under which the video image is. The application improves the comprehensiveness of the feature information extracted by the feature extraction network.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a method and apparatus for processing video image data. Background Technology

[0002] Currently, most deep learning-based feature extraction networks rely on large-scale video data for pre-training. However, in pre-training datasets, individual training data often only label video data corresponding to a single layer of information, making it difficult to widely apply the trained video image features across different scenarios. Furthermore, due to the complexity of video image content, which typically contains information from different layers such as objects, scenes, interactions, and actions, if the pre-training dataset only contains label information from one layer while completely discarding information from other layers, the extracted video image features will not provide a complete description of the video and may easily lead to overfitting of the video image feature training to a single aspect of the information. Summary of the Invention

[0003] This application provides a method and apparatus for processing video image data.

[0004] In a first aspect, embodiments of this application provide a method for processing video image data, comprising: performing the following training operations on a feature extraction network based on the video image data until a trained feature extraction network is obtained: selecting a video image from the video image data and extracting feature information of the video image through an initial feature extraction network; determining the layer feature information corresponding to each layer in a preset layer set; for each layer in the preset layer set except for the target layer, predicting the category information to which the video image belongs at that layer, wherein the target layer represents the layer to which the category information represented by the label of the video image belongs; and updating the initial feature extraction network according to a preset loss function, wherein the loss function is used to characterize minimizing the distance between the layer feature information and the category information corresponding to each layer of the video image.

[0005] In some embodiments, determining the feature information corresponding to the layer feature information of each layer in the preset layer set includes: mapping the feature information to the feature space based on a feature mapping layer including an attention mechanism to obtain the layer feature information corresponding to each layer in the preset layer set.

[0006] In some embodiments, predicting the category information of a video image at each layer in the preset layer set other than the target layer includes: predicting the category of a video image at each layer in the preset layer set other than the target layer using the classification model corresponding to that layer, and mapping the category to the feature space to obtain the category information of the video image at that layer, wherein the classification model is used to characterize the correspondence between the video image and the category to which the video image belongs at that layer.

[0007] In some embodiments, updating the initial feature extraction network according to a preset loss function includes: updating the initial feature extraction network according to a first loss function and a second loss function, wherein the first loss function represents minimizing the distance between the layer feature information corresponding to the target layer and the category information corresponding to the label, and maximizing the distance between the layer feature information corresponding to the target layer and the category information under the target layer other than the category information corresponding to the label, and the second loss function represents minimizing the distance between the layer feature information and the category information corresponding to each layer in the preset layer set other than the target layer.

[0008] In some embodiments, the method further includes: taking the acquired video to be processed as input, performing feature extraction through a trained feature extraction network to obtain feature information corresponding to the video to be processed.

[0009] In some embodiments, the above method further includes: performing information recognition based on feature information to determine the target object in the video to be processed.

[0010] Secondly, embodiments of this application provide a video image data processing apparatus, comprising: performing the following training operations on a feature extraction network based on the video image data until a trained feature extraction network is obtained: an extraction unit configured to select video images from the video image data and extract feature information of the video images through an initial feature extraction network; a determination unit configured to determine the layer feature information corresponding to each layer in a preset layer set; a prediction unit configured to predict the category information to which the video image belongs at each layer in the preset layer set except for a target layer, wherein the target layer represents the layer to which the category information represented by the label of the video image belongs; and an update unit configured to update the initial feature extraction network according to a preset loss function, wherein the loss function is used to characterize minimizing the distance between the layer feature information and the category information corresponding to each layer of the video image.

[0011] In some embodiments, the determining unit is further configured to: map feature information to a feature space based on a feature mapping layer including an attention mechanism, to obtain layer feature information corresponding to each layer in a preset layer set.

[0012] In some embodiments, the prediction unit is further configured to: for each layer in the preset layer set other than the target layer, predict the category to which the video image belongs under that layer using the classification model corresponding to that layer, and map the category to the feature space to obtain the category information of the video image under that layer, wherein the classification model is used to characterize the correspondence between the video image and the category to which the video image belongs under that layer.

[0013] In some embodiments, the updating unit is further configured to: update the initial feature extraction network according to a first loss function and a second loss function, wherein the first loss function represents minimizing the distance between the layer feature information corresponding to the target layer and the category information corresponding to the label, and maximizing the distance between the layer feature information corresponding to the target layer and the category information under the target layer other than the category information corresponding to the label, and the second loss function represents minimizing the distance between the layer feature information and the category information corresponding to each layer in the preset layer set other than the target layer.

[0014] In some embodiments, the above apparatus further includes: a feature extraction unit, configured to take the acquired video to be processed as input, and perform feature extraction through a trained feature extraction network to obtain feature information corresponding to the video to be processed.

[0015] In some embodiments, the above apparatus further includes a processing unit configured to perform information recognition based on feature information to determine a target object in the video to be processed.

[0016] Thirdly, embodiments of this application provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, it implements the method as described in any implementation of the first aspect.

[0017] Fourthly, embodiments of this application provide an electronic device, including: one or more processors; and a storage device storing one or more programs thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first aspect.

[0018] The video image data processing method and apparatus provided in this application embodiment perform the following training operations until a trained feature extraction network is obtained: selecting video images from video image data and extracting feature information of the video images through an initial feature extraction network; determining the layer feature information corresponding to each layer in a preset layer set; predicting the category information of the video image at each layer except the target layer in the preset layer set, wherein the target layer represents the layer to which the category information represented by the label of the video image belongs; updating the initial feature extraction network according to a preset loss function, wherein the loss function is used to characterize minimizing the distance between the layer feature information and the category information corresponding to each layer of the video image, thereby providing a video image data processing method and improving the comprehensiveness of the feature information extracted by the feature extraction network. Attached Figure Description

[0019] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0020] Figure 1 This is an exemplary system architecture diagram in which one embodiment of this application can be applied;

[0021] Figure 2 This is a flowchart of an embodiment of the video image data processing method according to this application;

[0022] Figure 3 This is a schematic diagram illustrating an application scenario of the video image data processing method according to this embodiment;

[0023] Figure 4 This is a flowchart of yet another embodiment of the video image data processing method according to this application;

[0024] Figure 5 This is a schematic diagram of the network structure for executing video image data processing methods;

[0025] Figure 6 This is a structural diagram of an embodiment of a video image data processing apparatus according to this application;

[0026] Figure 7 This is a schematic diagram of the structure of a computer system suitable for implementing the embodiments of this application. Detailed Implementation

[0027] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0028] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0029] Figure 1 An exemplary architecture 100 is shown that can be applied to the video image data processing method and apparatus of this application.

[0030] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. The communication connections between terminal devices 101, 102, and 103 form a network topology. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0031] Terminal devices 101, 102, and 103 can be hardware or software that supports network connectivity for data interaction and processing. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices that support network connectivity, information acquisition, interaction, display, and processing functions, including but not limited to smartphones, tablets, e-book readers, laptops, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the aforementioned electronic devices. They can be implemented as, for example, multiple software programs or software modules to provide distributed services, or as a single software program or software module. No specific limitations are imposed here.

[0032] Server 105 can be a server that provides various services, such as a backend server that acquires video image data captured or sent by terminal devices 101, 102, and 103, extracts features, and processes the obtained feature information. As an example, server 105 can be a cloud server.

[0033] It should be noted that a server can be either hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software programs or software modules (such as software programs or software modules used to provide distributed services), or as a single software program or software module. No specific limitations are made here.

[0034] It should also be noted that the video image data processing method provided in the embodiments of this application can be executed by a server, by a terminal device, or by a combination of both. Accordingly, the various parts (e.g., various units) of the video image data processing apparatus can be all located in the server, all located in the terminal device, or located separately in the server and the terminal device.

[0035] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included. When the electronic devices on which the video image data processing method runs do not need to transmit data with other electronic devices, the system architecture may only include the electronic devices (e.g., servers or terminal devices) on which the video image data processing method runs.

[0036] Continue to refer to Figure 2 The flowchart 200 illustrates an embodiment of a video image data processing method, in which a feature extraction network is trained based on the video image data until a trained feature extraction network is obtained:

[0037] Step 201: Select video images from the video image data and extract feature information of the video images through the initial feature extraction network.

[0038] In this embodiment, the execution entity of the video image data processing method (e.g.) Figure 1 The server in the process can select video images from the video image data and extract the feature information of the video images through the initial feature extraction network.

[0039] Video image data serves as the training sample set for training the initial feature extraction network. Each video image in the data is labeled with information representing the category to which it belongs at a certain level. Video images contain rich information, including but not limited to aspects such as action, time, interaction, motion, objects, and scenes. Each level of a video image can be further subdivided into multiple categories. For example, at the object level, the categories can be based on various classifications of physical objects, such as tables, chairs, air conditioners, and sofas.

[0040] The initial feature extraction network can be any type of neural network with feature extraction capabilities, including but not limited to two-dimensional convolutional neural networks and three-dimensional convolutional neural networks.

[0041] As an example, the aforementioned execution entity can use a 3D Local Global Propagation Network as the feature extraction network, take a 16-frame video segment as input, and use the output of the last convolutional layer of the 3D Local Global Propagation Network as the feature information of the video image.

[0042] Step 202: Determine the layer feature information corresponding to each layer in the preset layer set.

[0043] In this embodiment, the aforementioned execution entity can determine the feature information corresponding to the layer feature information of each layer in the preset layer set.

[0044] In this embodiment, although the video image is labeled with tag information at a single layer, it is highly likely that the video image also includes information at other layers. The preset layer set can include various layers involved in the video image.

[0045] As an example, the aforementioned execution entity can decompose the feature information into layer feature information corresponding to each layer in a preset layer set.

[0046] In some optional implementations of this embodiment, the execution entity can map feature information to a feature space based on a feature mapping layer including an attention mechanism to obtain layer feature information corresponding to each layer in a preset layer set.

[0047] Specifically, in general, the information of each layer in a video image is only associated with a portion of the image, especially regions containing objects or actions. For example, the feature information corresponding to a table is only associated with the area where the table is located in the video image. Therefore, directly using a single global feature, such as the output of a global pooling layer in a neural network, as the layer feature information for a single layer, introduces additional noise due to the inclusion of irrelevant regions. To find the most salient spatiotemporal region corresponding to a certain layer, a self-attention mechanism is used, along with a linear mapping layer, as the feature mapping function P for each layer. The self-attention mechanism assigns a positive weight to each spatiotemporal location of the feature information extracted by the 3D convolutional neural network; these weights can be considered as the probability that the location should be noticed by a certain layer. Finally, a weighted average is performed using these weights to obtain a feature vector, which is then linearly mapped into the feature space.

[0048] Specifically, a three-dimensional convolutional neural network extracts features from video v, resulting in a feature map F(v)∈R. C×T×H×W C, T, H, and W represent information such as the channel, time, length, and width of the feature map, respectively.

[0049] Local features in the feature map are weighted and averaged using the following formula based on a self-attention mechanism:

[0050]

[0051] in, This represents a Softmax-based normalization function that normalizes the weights at different spatiotemporal locations to a range between 0 and 1. n ,b n} represents the learnable parameters for the nth level, used to locate the spatiotemporal regions required for different levels, f (i) This represents the feature point information in the feature map F(v).

[0052] After the above weighted averaging is performed, the feature information is linearly mapped to the target feature space.

[0053] Step 203: For each layer in the preset layer set other than the target layer, predict the category information of the video image under that layer.

[0054] In this embodiment, the execution entity can predict the category information of a video image at each level in the preset level set, excluding the target level. The target level represents the level to which the category information represented by the video image's label belongs.

[0055] As an example, the aforementioned execution entity can predict the category information of a video image at each level other than the target level based on the pre-set correspondence between feature information and the category information to which the feature information belongs at each level.

[0056] In some optional implementations of this embodiment, for each layer in the preset layer set other than the target layer, the aforementioned execution entity can predict the category to which the video image belongs at that layer using the classification model corresponding to that layer, and map the category to the feature space to obtain the category information of the video image at that layer. The classification model is used to characterize the correspondence between the video image and the category to which the video image belongs at that layer.

[0057] When the information learned at level m is transferred to video v at level n... n At that time, we use a classification model pre-trained on the m-th level data to predict video v. n Response probability at the m-th level The ultimate goal is to make the video's feature information more closely approximate the categories with high response probabilities at the m-th level. Therefore, we use a weighted average of the response probabilities to represent the category-mapped features, as the video v. n Category information at levels other than the target level:

[0058]

[0059] Among them, S m (v n ) indicates video v n Category information at levels other than the target level. This indicates the category at the m-th level (other levels besides the target level). Indicates category Category mapping function, Indicates video v n The response probability at the m-th level, L m It refers to all categories at the m-th level.

[0060] Step 204: Update the initial feature extraction network according to the preset loss function.

[0061] In this embodiment, the aforementioned execution entity can update the initial feature extraction network according to a preset loss function. The loss function is used to characterize minimizing the distance between the corresponding level feature information and category information of the video image at each level.

[0062] As an example, the loss function consists of two parts. The first part represents minimizing the distance between the layer feature information corresponding to the target layer and the category information corresponding to the label. The second part represents minimizing the distance between the layer feature information and the category information corresponding to each layer in the preset layer set, excluding the target layer.

[0063] In this context, distance represents the distance between layer feature information and the corresponding category information in the feature space. For example, distance can be Euclidean distance, Minkowski distance, Manhattan distance, etc.

[0064] The primary objective of the first part of the loss function is to align the layered feature information and category information mapped from the video image in the feature space. Given a video-category pair (v...) from the nth layer... n ,l n ), where v n It is video data labeled with tags representing categories at the nth level, l n This is the category it is labeled at the nth level. The distance between the layer feature information and the category information at the target level can be calculated using the L2 distance in the feature space:

[0065]

[0066] Among them, L intra-l2 (v n ,l n) represents the distance between the layer feature information and the category information corresponding to the target layer, and F(·), P(·) and S(·) represent the feature information extraction function, the feature information mapping function and the category information mapping function, respectively.

[0067] For the second part of the loss function, the distance between the layer feature information of layers other than the target layer and the predicted category information can be calculated using the L2 distance in the feature space:

[0068]

[0069] Among them, L inter (v n P represents the distance between the feature information of a layer outside the target layer and the corresponding category information. m (·) represents the layer feature information corresponding to the mapping of the m-th layer, S m (·) represents the category information obtained by mapping to the m-th level.

[0070] In some optional implementations of this embodiment, the execution entity updates the initial feature extraction network according to a first loss function and a second loss function. The first loss function represents minimizing the distance between the layer feature information corresponding to the target layer and the category information corresponding to the label, and maximizing the distance between the layer feature information corresponding to the target layer and the category information under the target layer excluding the category information corresponding to the label. The second loss function represents minimizing the distance between the layer feature information and the category information corresponding to each layer in the preset layer set excluding the target layer.

[0071] If the loss function described in the first part is used for training, the trained feature information may lack discriminability among different categories at the target level. That is, the distance between the feature information and other categories at the target level, other than the category represented by the label, may even be closer than the distance between the feature information and the category represented by the label. To solve this problem, we introduce the concept of contrastive learning into the first loss function, which allows the layer feature information corresponding to the target level to be close to the category information represented by the label while keeping it as far away as possible from the category information of other categories at the target level.

[0072] Specifically, the feature information P(F(v) mapped from the video image in the feature space n The category information S(l) represented by the label and mapped onto the feature space n ) as positive sample pairs, and the feature information P(F(v) nThe target class and other class information are used as negative sample pairs. The main goal of contrastive learning is to bring positive sample pairs closer together while simultaneously distancing negative sample pairs as much as possible. To achieve this, the first loss function can be formulated as the following Softmax form:

[0073]

[0074] Among them, L n It refers to all categories at the nth level.

[0075] For the second loss function, refer to the second part of the loss function mentioned above.

[0076] In this embodiment, the execution entity repeatedly executes steps 201-204 until the trained feature extraction network is obtained. As an example, the training termination condition could be the loss function converging to its minimum, the number of training iterations exceeding a preset threshold, or the training time exceeding a preset time threshold.

[0077] See also Figure 3 , Figure 3 This is a schematic diagram 300 illustrating an application scenario of the video image data processing method according to this embodiment. Figure 3 In the application scenario, the following training operations are performed until the trained feature extraction network is obtained: First, video image 302 is selected from video image data 301, and feature information 304 of the video image is extracted through the initial feature extraction network 303. Then, the feature information 304 is determined to correspond to the layer feature information of each layer in the preset layer set, including layer feature information 3041, 3042, 3043, 3044, 3045, and 3046 corresponding to action, time, interaction, motion, object, and scene respectively. Then, for each layer in the preset layer set except the target layer, the category information to which the video image belongs under that layer is predicted, including 3052, 3053, 3054, 3055, and 3056. Here, the target layer represents the layer to which the category information 3051 represented by the label of the video image belongs. Finally, the initial feature extraction network is updated according to the preset loss function. Here, the loss function is used to represent minimizing the distance between the layer feature information and the category information corresponding to each layer of the video image.

[0078] The method provided in the above embodiments of this application, through the following training operations, until a trained feature extraction network is obtained: selecting video images from video image data and extracting feature information of the video images through an initial feature extraction network; determining the layer feature information corresponding to each layer in a preset layer set; predicting the category information of the video image at each layer in the preset layer set except for the target layer, wherein the target layer represents the layer to which the category information represented by the label of the video image belongs; updating the initial feature extraction network according to a preset loss function, wherein the loss function is used to characterize minimizing the distance between the layer feature information and the category information corresponding to each layer of the video image, thereby providing a method for processing video image data and improving the comprehensiveness of the feature information extracted by the feature extraction network.

[0079] In some optional implementations of this embodiment, the execution entity can take the acquired video to be processed as input and perform feature extraction through a trained feature extraction network to obtain feature information of the video to be processed. The feature information extracted by the feature extraction network trained based on the above training method has descriptive capabilities at different levels.

[0080] In some optional implementations of this embodiment, the execution entity can also perform information recognition based on feature information to determine the target object in the video to be processed. As an example, the execution entity can use the trained feature extraction network as the feature extraction layer of a neural network model used for tasks such as machine vision and image analysis, and use the feature information as input to subsequent network structure layers to perform information recognition and determine the target object in the video to be processed, thereby improving the accuracy of the information recognition results. The target object can be any object, including but not limited to objects, people, animals, etc.

[0081] Continue to refer to Figure 4 The illustration shows a schematic flow 400 of an embodiment of a video image data processing method according to this application, which performs the following training operations until a trained feature extraction network is obtained:

[0082] Step 401: Select video images from the video image data and extract feature information of the video images through the initial feature extraction network.

[0083] Step 402: Based on the feature mapping layer including the attention mechanism, the feature information is mapped to the feature space to obtain the layer feature information corresponding to each layer in the preset layer set.

[0084] Step 403: For each layer in the preset layer set other than the target layer, predict the category of the video image under that layer using the classification model corresponding to that layer, and map the category to the feature space to obtain the category information of the video image under that layer.

[0085] The classification model is used to characterize the correspondence between video images and the categories to which they belong at that level.

[0086] Step 404: Update the initial feature extraction network according to the first loss function and the second loss function.

[0087] The first loss function represents minimizing the distance between the layer feature information corresponding to the target layer and the category information corresponding to the label, and maximizing the distance between the layer feature information corresponding to the target layer and the category information under the target layer other than the category information corresponding to the label. The second loss function represents minimizing the distance between the layer feature information and the category information corresponding to each layer in the preset layer set other than the target layer.

[0088] like Figure 5 The diagram illustrates the network structure of a method for processing video image data. First, a three-dimensional convolutional neural network 502 extracts features from the video image 501, obtaining feature information 503. Then, through feature mapping layers 504 (including attention mechanisms) corresponding to each layer in a preset layer set, the layer feature information of feature information 503 at each layer is obtained and mapped to the feature space 505. Next, for each layer in the preset layer set except the target layer, the classification model 506 corresponding to that layer predicts the category of the video image at that layer and maps the category to the feature space 505, obtaining the category information of the video image at that layer. Finally, the initial feature extraction network is updated according to a first loss function and a second loss function.

[0089] As can be seen from this embodiment, with Figure 2 Compared to the corresponding embodiments, the process flow 400 of the video image data processing method in this embodiment specifically describes the process of determining feature information at each level, the prediction process of category information of video images at other levels besides the target level, and the update process of the initial feature extraction network, which further improves the comprehensiveness of the feature information extracted by the feature extraction network.

[0090] Continue to refer to Figure 6 As an implementation of the methods shown in the above figures, this application provides an embodiment of a video image data processing apparatus, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0091] like Figure 6 As shown, the video image data processing device performs the following training operations on the feature extraction network based on the video image data until a trained feature extraction network is obtained: Extraction unit 601 is configured to select video images from the video image data and extract feature information of the video images through the initial feature extraction network; Determination unit 602 is configured to determine the layer feature information corresponding to each layer in the preset layer set; Prediction unit 603 is configured to predict the category information of the video image at each layer in the preset layer set except for the target layer, wherein the target layer represents the layer to which the category information represented by the label of the video image belongs; Update unit 604 is configured to update the initial feature extraction network according to a preset loss function, wherein the loss function is used to characterize minimizing the distance between the layer feature information and the category information corresponding to each layer of the video image.

[0092] In some embodiments, the determining unit 602 is further configured to: map feature information to a feature space based on a feature mapping layer including an attention mechanism, to obtain layer feature information corresponding to each layer in a preset layer set.

[0093] In some embodiments, the prediction unit 603 is further configured to: for each layer in the preset layer set other than the target layer, predict the category to which the video image belongs under that layer using the classification model corresponding to that layer, and map the category to the feature space to obtain the category information of the video image under that layer, wherein the classification model is used to characterize the correspondence between the video image and the category to which the video image belongs under that layer.

[0094] In some embodiments, the updating unit 604 is further configured to: update the initial feature extraction network according to a first loss function and a second loss function, wherein the first loss function represents minimizing the distance between the layer feature information corresponding to the target layer and the category information corresponding to the label, and maximizing the distance between the layer feature information corresponding to the target layer and the category information under the target layer other than the category information corresponding to the label, and the second loss function represents minimizing the distance between the layer feature information and the category information corresponding to each layer in the preset layer set other than the target layer.

[0095] In some embodiments, the above apparatus further includes: a feature extraction unit (not shown in the figure), configured to take the acquired video to be processed as input, and perform feature extraction through a trained feature extraction network to obtain feature information corresponding to the video to be processed.

[0096] In some embodiments, the above apparatus further includes a processing unit (not shown in the figure), configured to perform information recognition based on feature information to determine a target object in the video to be processed.

[0097] In this embodiment, the extraction unit in the video image data processing device selects video images from the video image data and extracts feature information of the video images through an initial feature extraction network; the determination unit determines the layer feature information corresponding to each layer in a preset layer set; the prediction unit predicts the category information to which the video image belongs at each layer in the preset layer set except for the target layer, wherein the target layer represents the layer to which the category information represented by the label of the video image belongs; the update unit updates the initial feature extraction network according to a preset loss function, wherein the loss function is used to characterize minimizing the distance between the layer feature information and the category information corresponding to each layer of the video image, thereby providing a video image data processing device that improves the comprehensiveness of the feature information extracted by the feature extraction network.

[0098] The following is for reference. Figure 7 It illustrates a device suitable for implementing embodiments of this application (e.g., Figure 1 The diagram shows the structure of the computer system 700 of the devices 101, 102, 103, and 105. Figure 7 The device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0099] like Figure 7 As shown, the computer system 700 includes a processor (e.g., CPU, Central Processing Unit) 701, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 702 or programs loaded from storage section 708 into random access memory (RAM) 703. The RAM 703 also stores various programs and data required for the operation of the system 700. The processor 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0100] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0101] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 709, and / or installed from removable medium 711. When the computer program is executed by processor 701, it performs the functions defined in the methods of this application.

[0102] It should be noted that the computer-readable medium of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0103] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—as well as conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the client computer, partially on the client computer, as a standalone software package, partially on the client computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the client computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0104] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0105] The units described in the embodiments of this application can be implemented in software or hardware. The described units can also be housed in a processor; for example, it can be described as: a processor including an extraction unit, a determination unit, a prediction unit, and an update unit. The names of these units do not necessarily limit the unit itself; for example, the update unit can also be described as "a unit that updates the initial feature extraction network according to a preset loss function."

[0106] In another aspect, this application also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, which, when executed by the device, cause the computer device to: perform the following training operations on a feature extraction network based on video image data until a trained feature extraction network is obtained: selecting video images from the video image data and extracting feature information of the video images through an initial feature extraction network; determining the layer feature information corresponding to each layer in a preset layer set; predicting the category information of the video image at each layer in the preset layer set, except for the target layer, wherein the target layer represents the layer to which the category information represented by the label of the video image belongs; and updating the initial feature extraction network according to a preset loss function, wherein the loss function is used to characterize minimizing the distance between the layer feature information and the category information corresponding to each layer of the video image.

[0107] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A method of processing video image data, characterized by, The method comprises the following steps of: performing the following training operation on a feature extraction network according to video image data until a trained feature extraction network is obtained: selecting a video image from the video image data and extracting feature information of the video image through an initial feature extraction network; determining layer feature information corresponding to each layer in a preset layer set of the feature information, wherein the preset layer set comprises an action layer, a time layer, an interaction layer, a motion layer, an object layer and a scene layer; for each layer in the preset layer set except a target layer, predicting category information to which the video image belongs at the layer, wherein the target layer represents a layer of category information represented by a label of the video image; updating the initial feature extraction network according to a first loss function and a second loss function, wherein the first loss function represents minimizing a distance between layer feature information corresponding to the target layer and category information corresponding to the label and maximizing a distance between the layer feature information corresponding to the target layer and category information other than the category information corresponding to the label at the target layer, and the second loss function represents minimizing a distance between layer feature information corresponding to each layer in the preset layer set except the target layer and category information.

2. The method of claim 1, wherein, The determination of the layer feature information corresponding to each layer in the preset layer set of the feature information comprises: mapping the feature information to a feature space based on a feature mapping layer comprising an attention mechanism to obtain the layer feature information corresponding to each layer in the preset layer set.

3. The method of claim 1, wherein, The prediction of the category information to which the video image belongs at each layer in the preset layer set except the target layer comprises: for each layer in the preset layer set except the target layer, predicting a category of the video image at the layer through a classification model corresponding to the layer and mapping the category to a feature space to obtain category information of the video image at the layer, wherein the classification model is used to represent a corresponding relationship between a video image and a category to which the video image belongs at the layer.

4. The method of claim 1, wherein, The method further comprises the following steps of: performing feature extraction on a to-be-processed video through the trained feature extraction network to obtain feature information corresponding to the to-be-processed video.

5. The method of claim 4, wherein, The method further comprises the following steps of: performing information identification based on the feature information to determine a target object in the to-be-processed video.

6. An apparatus for processing video image data, characterized by The method comprises the following steps of: performing the following training operation on a feature extraction network according to video image data until a trained feature extraction network is obtained: an extraction unit configured to select a video image from the video image data and extract feature information of the video image through an initial feature extraction network; a determination unit configured to determine layer feature information corresponding to each layer in a preset layer set of the feature information, wherein the preset layer set comprises an action layer, a time layer, an interaction layer, a motion layer, an object layer and a scene layer; The prediction unit is configured to predict the category information of the video image at each of the preset layers except for the target layer, wherein the target layer represents the layer to which the category information represented by the label of the video image belongs; The update unit is configured to update the initial feature extraction network according to a first loss function and a second loss function, wherein the first loss function represents minimizing the distance between the layer feature information corresponding to the target layer and the category information corresponding to the label, and maximizing the distance between the layer feature information corresponding to the target layer and the category information under the target layer other than the category information corresponding to the label, and the second loss function represents minimizing the distance between the layer feature information and the category information corresponding to each layer in the preset layer set other than the target layer.

7. The apparatus of claim 6, wherein, The determining unit is further configured to: Based on a feature mapping layer including an attention mechanism, the feature information is mapped to a feature space to obtain layer feature information corresponding to each layer in the preset layer set.

8. The apparatus of claim 6, wherein, The prediction unit is further configured to: For each layer in the preset layer set other than the target layer, the classification model corresponding to that layer is used to predict the category to which the video image belongs at that layer, and the category is mapped to the feature space to obtain the category information of the video image at that layer. The classification model is used to characterize the correspondence between the video image and the category to which the video image belongs at that layer.

9. The apparatus of claim 6, wherein, Also includes: The feature extraction unit is configured to take the acquired video to be processed as input, and perform feature extraction through the trained feature extraction network to obtain the feature information corresponding to the video to be processed.

10. The apparatus of claim 9, wherein, Also includes: The processing unit is configured to perform information recognition based on the feature information to determine the target object in the video to be processed.

11. A computer readable medium having stored thereon a computer program, wherein, When the program is executed by the processor, it implements the method as described in any one of claims 1-5.

12. An electronic device, comprising: One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-5.

Images