Video object detection method, device and storage medium

Abstract

The embodiment of the present invention discloses a video target detection method, device and storage medium, which are applied to the field of information processing technology. In the method of this embodiment, after extracting the features of each frame image in the video to be detected, the video target detection device performs weighted processing on the features of each channel in the obtained feature map to obtain a weighted feature map, thereby weakening the features of some unimportant channels and strengthening the features of other important channels. In this way, in the process of fusing the features of adjacent frame images or adjacent interval frame images in the video to be detected in the weighted feature map, the feature fusion operation can establish a temporal feature association on the basis of good feature quality, taking into account the relationship between each frame image in the video to be detected, and then more accurately detecting the target according to the fused feature map.

Description

Technical Field

[0001] The present invention relates to the field of information processing technology, and in particular to a video target detection method, device and storage medium. Background Art

[0002] Video object detection is similar to image object detection. Both require the location detection and classification of objects in the image. However, in the process of video object detection, the timing information and context information of consecutive frame images need to be processed. If the video is treated as a single frame image for object detection, the timing information and context information will be lost. The current video object methods can use timing information and context information for video object detection. The existing methods mainly include: object connection and feature propagation, or a combination of the two.

[0003] An existing video target detection method is a method based on Region Convolution Neural Networks (RCNN). The RCNN network can extract features from each frame image in the video, and then detect and classify the target.

[0004] In the existing video target detection method, it is necessary to compress the size of each frame image input to the RCNN network, and use fewer and smaller feature maps to maintain low computational complexity and improve speed. However, the amount of information extracted from the feature map is relatively small, which affects the detection accuracy. Summary of the invention

[0005] The embodiments of the present invention provide a video target detection method, device and storage medium, which realize channel weighting and feature fusion processing of features of each frame image in a video to be detected.

[0006] A first aspect of an embodiment of the present invention provides a video object detection method, comprising:

[0007] Extracting features of each frame image in the video to be detected to obtain a feature map, wherein the feature map includes features of multiple channels;

[0008] Performing weighted processing on the features of each channel in the multiple channels to obtain a weighted feature map;

[0009] In the weighted feature map, features of adjacent frame images or adjacent interval frame images in the video to be detected are fused to obtain a fused feature map;

[0010] Perform object detection on the video to be detected according to the fused feature map.

[0011] A second aspect of an embodiment of the present invention provides a video object detection device, comprising: a feature extraction unit, at least one attention unit, at least one feature association unit and a detection unit, wherein:

[0012] The feature extraction unit is used to extract the features of each frame image in the video to be detected to obtain a feature spectrum, wherein the feature spectrum includes features of multiple channels;

[0013] The attention unit is used to perform weighted processing on the features of each channel in the multiple channels to obtain a weighted feature map;

[0014] The feature association unit is used to fuse the features of adjacent frame images or adjacent interval frame images in the video to be detected in the weighted feature map to obtain a fused feature map;

[0015] The detection unit is used to perform target detection on the video to be detected according to the fused feature map.

[0016] A third aspect of an embodiment of the present invention provides a storage medium, wherein the storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded by a processor and executing the video object detection method as described in the first aspect of the embodiment of the present invention.

[0017] A fourth aspect of an embodiment of the present invention provides a terminal device, including a processor and a storage medium, wherein the processor is configured to implement various instructions;

[0018] The storage medium is used to store a plurality of instructions, and the instructions are used to be loaded by a processor and execute the video object detection method as described in the first aspect of the embodiment of the present invention.

[0019] It can be seen that in the method of this embodiment, after extracting the features of each frame image in the video to be detected, the video target detection device performs weighted processing on the features of each channel in the multiple channels in the obtained feature map to obtain a weighted feature map, thereby weakening the features of some unimportant channels and strengthening the features of other important channels. In this way, in the process of fusing the features of adjacent frame images or adjacent interval frame images in the video to be detected in the weighted feature map, the feature fusion operation can establish a temporal feature association on the basis of good feature quality, taking into account the relationship between each frame image in the video to be detected, and then more accurately detecting the target according to the fused feature map. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0021] Figure 1 is a schematic diagram of a video target detection method provided by an embodiment of the present invention;

[0022] Figure 2 is a flow chart of a video target detection method provided by an embodiment of the present invention;

[0023] Figure 3 is a flow chart of a video object detection method trained in an embodiment of the present invention;

[0024] Figure 4 is a structural schematic diagram of a video target detection device in an application embodiment of the present invention;

[0025] Figure 5 is a schematic diagram of the structure of a channel attention module in an application embodiment of the present invention;

[0026] Figure 6 is a structural diagram of a timing feature association module in an application embodiment of the present invention;

[0027] Figure 7 is a structural schematic diagram of a video object detection device provided by an embodiment of the present invention;

[0028] Figure 8 It is a structural diagram of a terminal device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] The embodiment of the present invention provides a video target detection method, which is mainly applied to a CNN network. Figure 1 As shown, the video target detection device can specifically implement position detection and classification of the target in the video to be detected by the following method:

[0032] The features of each frame image in the video to be detected are extracted to obtain a feature map, wherein the feature map includes features of multiple channels; the features of each channel in the multiple channels are weighted to obtain a weighted feature map; the features of adjacent frame images or adjacent interval frame images in the video to be detected in the weighted feature map are fused to obtain a fused feature map; and target detection is performed on the video to be detected according to the fused feature map.

[0033] Among them, the features of each channel in the multiple channels in the feature map are weighted to obtain a weighted feature map, thereby weakening the features of some unimportant channels and strengthening the features of other important channels. In the process of fusing the features of adjacent frame images or adjacent interval frame images in the video to be detected in the weighted feature map, the feature fusion operation can establish a temporal feature association on the basis of good feature quality, taking into account the relationship between each frame image in the video to be detected, and then more accurately detecting the target based on the fused feature map.

[0034] An embodiment of the present invention provides a video object detection method, which is mainly performed by a video object detection device. The flow chart is as follows: Figure 2 As shown, including:

[0035] Step 101: A video object detection device extracts features of each frame image in a video to be detected to obtain a feature map, wherein the feature map includes features of multiple channels.

[0036] It can be understood that the user can operate the video target detection device so that the video target detection device initiates the method of this embodiment. In one case, the video target detection device can provide a video input interface, and the user can input the video to be detected through the video input interface, and the video target detection device can directly execute steps 101 to 104 for the input video to be detected; or the video target detection device first pre-processes the input video to be detected, and then executes steps 101 to 104 for the pre-processed video to be detected. Among them, the pre-processing of the input video to be detected may include: intercepting part of the video in the input video to be detected, so that the time length of the intercepted part of the video is within a certain range (for example, within 30 seconds); removing part of the video in the input video to be detected that is more severely jittered, for example, if the jitter is more severe within a period of time at the beginning of the input video to be detected, then remove part of the video within this period of time, etc.

[0037] In another case, the video target detection device can obtain collected videos from other video acquisition devices at a fixed period (or other video acquisition devices can actively send the collected videos to the video target detection device), and use the collected videos as the video to be detected, and directly initiate the process of steps 101 to 104 in this embodiment.

[0038] When the video target detection device of this embodiment performs video target detection on the video to be detected, it is mainly based on the detection method of convolutional neural network. Generally, the operation logic of convolutional neural network is preset in the video target detection device. In this way, the convolutional neural network can be used to extract features of the video to be detected in real time, and then the target position is detected and classified according to the final extracted features.

[0039] The feature extraction of the video to be detected by the convolutional neural network may include extracting all features of each frame image, selecting important features of each frame image, etc. In this embodiment, after extracting all features of each frame image, the following steps 102 to 104 may be performed based on the feature map composed of all features; or after selecting important features of each frame image, the following steps 102 to 104 may be performed based on the feature map composed of important features.

[0040] It should be noted that the features of any frame image extracted by the convolutional neural network may include features of multiple dimensions, such as texture dimension, color dimension, etc., and the features of all frame images in each dimension constitute the features of a channel.

[0041] Step 102: The video object detection device performs weighted processing on the features of each channel in the multiple channels to obtain a weighted feature map.

[0042] Specifically, the video target detection device can adopt a squeeze-and-excitation (SE) network structure for weighted processing, that is, firstly perform global average pooling on the features of each channel in multiple channels to obtain the global features of each channel; then determine the weight information of each channel feature according to the global features of each channel, and multiply the weight information of each channel feature by the features of the corresponding channel in the above feature map to obtain a weighted feature map.

[0043] The weighted feature map also includes features of multiple channels, but the features of each channel are weighted features. In this way, the features of some unimportant channels can be weakened and the features of other important channels can be strengthened.

[0044] Step 103, the video object detection device fuses the features of adjacent frame images or adjacent interval images in the video to be detected in the weighted feature map to obtain a fused feature map.

[0045] Specifically, the video target detection device can first divide the features of each channel in the weighted feature map into multiple groups of features, for example, according to the time period, so that the features in the weighted feature map add a time dimension; then calculate the similarities between adjacent group features or adjacent interval group features of each channel respectively to obtain the similarities corresponding to each feature in each channel; then weight the similarities of each feature in each channel and add them to the corresponding features to obtain a fused feature map.

[0046] Among them, when the video target detection device divides the features of any channel in the weighted feature map, the features of each frame image within every T seconds can be divided into a group. For example, the features of a certain channel are c1, c2, ..., cn. When dividing, the features c1, c2, ..., ci can be divided into a group; the features ci+1, ci+1, ..., cj can be divided into another group; ...; the features ck, ck+1, ..., cn can be divided into a group.

[0047] Since each group includes multiple features, when calculating the similarity between adjacent group features or adjacent interval group features, the video object detection device can calculate the similarity between a certain feature in a group and each feature in another group, and accumulate these similarities to obtain the similarity of the feature. For example, if a certain group of features is c11, c12, ..., c1i, and its adjacent group of features is c21, c22, ..., c2j, then the similarity corresponding to feature c11 can be the sum of the similarities between feature c11 and features c21, c22, ..., c2j.

[0048] When the video target detection device weights the similarity of each feature in each channel and adds them to the corresponding feature, the similarity of each feature can be multiplied by the corresponding weight value to obtain the similarity weighted value of each feature, and then the similarity weighted value of each feature is added to the corresponding feature.

[0049] Furthermore, in order to reduce the amount of calculation, after the video target detection device divides the features of each channel in the weighted feature map into multiple groups of features, it can also perform dimensionality reduction processing on the features in the weighted feature map, so that the number of channels of the reduced-dimensional features is less than the number of channels of the weighted feature map, for example, it can be half of the number of channels of the weighted feature map, that is, the features of a certain number of channels are removed; in this way, when the video target detection device calculates the similarity between adjacent groups of features or adjacent interval groups of features of each channel respectively, it calculates the similarity between adjacent groups of features or adjacent interval groups of features of each channel in the reduced-dimensional features respectively; and in the process of weighting the similarity of each feature in each channel and adding it to the corresponding feature, the dimension of the feature can be restored. Specifically, when adding the weighted value of the similarity of each feature in the remaining channel to the corresponding feature, each feature in the removed channel is added with zero, that is, each feature in the removed channel is retained.

[0050] Furthermore, after the video object detection device weights the similarity of each feature in each channel and adds the features to the corresponding features, it can also normalize the features obtained after the addition, and form the normalized features into a fused feature map. In this way, the distribution range of the features in the fused feature map and the weighted feature map remains unchanged.

[0051] Step 104, the video object detection device performs object detection on the video to be detected based on the fused feature map, mainly detecting the position and type of each object in the video.

[0052] It should be noted that if the feature in the feature map obtained by the video target detection device in the above step 101 is the final feature of each frame image, that is, the most important feature of each frame image, then when executing step 104, target detection can be directly performed based on the features in the fused feature map; if the feature in the feature map obtained by the video target detection device in the above step 101 is the intermediate feature of each frame image, then when executing step 104, other operations need to be performed on the features in the fused feature map, such as selecting the most important features, etc., to obtain the final features and then perform target detection based on the final features.

[0053] It can be seen that in the method of this embodiment, after extracting the features of each frame image in the video to be detected, the video target detection device performs weighted processing on the features of each channel in the multiple channels in the obtained feature map to obtain a weighted feature map, thereby weakening the features of some unimportant channels and strengthening the features of other important channels. In this way, in the process of fusing the features of adjacent frame images or adjacent interval frame images in the video to be detected in the weighted feature map, the feature fusion operation can establish a temporal feature association on the basis of good feature quality, taking into account the relationship between each frame image in the video to be detected, and then more accurately detecting the target according to the fused feature map.

[0054] In a specific embodiment, the video object detection device can be trained to obtain a convolutional neural network that performs the above steps 101 to 104 through the following steps. The flowchart is as follows: Figure 3 As shown, including:

[0055] In step 201, the video target detection apparatus determines an initial convolutional neural network that performs feature extraction, weighted processing, feature fusion and target detection steps, that is, determines an initial convolutional neural network that performs the above steps 101 to 104.

[0056] Specifically, when determining the initial convolutional neural network, the video target detection device determines the multi-layer structure included in the initial convolutional neural network and the initial values ​​of the fixed parameters in each layer structure.

[0057] The initial convolutional neural network may include a feature module and a target detection module, wherein the feature module is used to execute the above steps 101 to 103, and the target detection module is used to perform target detection on the video to be detected according to the features finally obtained by the feature module.

[0058] The above-mentioned fixed parameters refer to the fixed parameters used in the calculation process of each layer structure in the initial convolutional neural network, which do not need to be assigned at any time, such as weights, angles and other parameters. The process of the video object detection device determining the initial value of the fixed parameter is the process of initializing the initial convolutional neural network.

[0059] In step 202, the video target detection device determines a training sample, which includes: multiple groups of training images, each group of training images includes multiple continuous frame images (i.e., each frame image of a short video), and the training sample also includes the target position identification of each frame image in the multiple groups of training images.

[0060] In step 203, the video target detection device performs target detection on each frame image in the training sample through an initial convolutional neural network to obtain an initial result of the target position, and the initial result of the target position includes the target position information of each target in each frame image in the training sample.

[0061] Step 204: The video target detection device adjusts the fixed parameter values ​​in the initial convolutional neural network according to the initial result of the target position and the target position identifier of each frame image in the training sample.

[0062] Specifically, the video target detection device can first calculate the loss function related to the initial convolutional neural network based on the initial result of the target position and the target position identification of each frame image in the training sample. The loss function is used to represent the target position information of each target in each frame image in the training sample determined according to the initial convolutional neural network, and the difference between the actual position of each target in each frame image in the training sample (obtained according to the target position identification of each frame image in the training sample), that is, the error. The mathematical expression of the error usually uses cross entropy to establish the loss function, and the training process of the initial convolutional neural network is to minimize the value of the error. The training process is to continuously optimize the initial value of the fixed parameter determined in the above step 201 through a series of mathematical optimization means such as back propagation derivation and gradient descent, and minimize the calculated value of this loss function. Therefore, after calculating the loss function related to the initial convolutional neural network, the fixed parameter value in the initial convolutional neural network will be adjusted according to the loss function value to obtain the final initial convolutional neural network.

[0063] Specifically, when adjusting the fixed parameter value, the video object detection device adjusts the initial value of the fixed parameter determined above according to the loss function value. If the function value of the calculated loss function is large, such as greater than a preset value, the fixed parameter value needs to be changed, such as reducing the weight value of a certain weight, so that the function value of the loss function calculated according to the adjusted fixed parameter value is reduced.

[0064] It should be noted that the above steps 203 to 204 are an adjustment of the fixed parameter values ​​in the initial convolutional neural network according to the initial result of the target position after the initial convolutional neural network performs target detection on each frame image in the training sample to obtain the initial result of the target position. In actual applications, it is necessary to continuously loop through the above steps 203 to 204 until the adjustment of the fixed parameter value meets certain stop conditions.

[0065] Therefore, after executing steps 201 to 204 of the above embodiment, the video target detection device still needs to determine whether the current adjustment of the fixed parameter value meets the preset stop condition. If so, the process ends; if not, the initial convolutional neural network after adjusting the fixed parameter value returns to execute steps 203 to 204 above.

[0066] Among them, the preset stop conditions include but are not limited to any one of the following conditions: the difference between the currently adjusted fixed parameter value and the last adjusted fixed parameter value is less than a threshold, that is, the adjusted fixed parameter value reaches convergence; and the number of adjustments to the fixed parameter value reaches a preset number, etc.

[0067] The following is a specific application example to illustrate the video object detection method of the present invention. In this embodiment, the structural diagram of the video object detection device is as follows: Figure 4 As shown, it may include: a convolutional neural network composed of a first feature extraction module 201, a second feature extraction module 202, a channel attention module 30, a temporal feature association module 40 and a target detection module 50. Among them, the channel attention module 30 and the temporal feature association module 40 can be one or more, and can be embedded in different positions in the feature extraction process, such as Figure 4 As described in, it is embedded after the intermediate features are extracted. In other embodiments, it can be embedded after the final features are extracted; and Figure 4 A channel attention module 30 and a temporal feature association module 40 are used as examples for explanation. Specifically:

[0068] The first feature extraction module 201 is used to extract all features from the original input data 10, obtain a feature map, and input it to the channel attention module 30. In this embodiment, in the process of training the convolutional neural network, the original input data 10 can be specifically a plurality of groups of sample images, each group of sample images includes a plurality of continuous frame images or continuous interval frame images; in the real-time detection process, the original input data 10 is specifically each frame image in the video to be detected.

[0069] The channel attention module 30 is used to perform weighted processing on the features of each channel in the feature map obtained by the first feature extraction module 201 to obtain a weighted feature map. In this way, the features of important channels are highlighted and the features of unimportant channels are weakened, thereby achieving the purpose of feature enhancement.

[0070] The temporal feature association module 40 is used to fuse the features of the t frame image and the t+τ frame image (i.e., adjacent frame images or adjacent interval frame images) in the weighted feature map to form a feature dependency between the t frame image and the t+τ frame image, so that the temporal information and context information between frames are associated.

[0071] The second feature extraction module 202 is used to further perform feature extraction based on the fused feature map obtained by the time series feature association module 40, such as selecting important features.

[0072] The object detection module 50 is used to perform operations such as position detection and classification of the video object according to the final features obtained by the second feature extraction module 202.

[0073] It should be noted that the structures of the first feature extraction module 201 and the second feature extraction module 202 can be implemented in various ways, such as mobilenet, darknet and the like.

[0074] Furthermore, the channel attention module 30 can be implemented as follows: Figure 5 Specifically, the channel attention module 30 can implement the weighted operation of features through the following steps:

[0075] (1) The channel attention module 30 first passes through the global average pooling function F sq (·) Perform global average pooling on the features of each channel in the multiple channels extracted by the first feature extraction module 201 to obtain the global features of each channel. Specifically, it can be expressed by the following formula 1:

[0076]

[0077] Wherein, Uc represents the feature spectrum extracted by the first feature extraction module 201 (in Figure 5 U in the figure), H is the height of the feature map, W is the width of the feature map, C is the number of channels of the feature map; i and j represent the position; z c It is the result of global average pooling, and its dimension is 1×1×C.

[0078] (2) The channel attention module 30 is further controlled by a simple gating mechanism F ex (·,W), the weight information of each channel feature is determined according to the global feature of each channel. It can be specifically expressed by the following formula 2:

[0079] s c =F ex (z,W)=σ(g(z,W))=σ(W2δ(W1z)) (2)

[0080] Among them, z represents the global feature of each channel obtained by the above formula 1; W1 and W2 are two fully connected layers, which can improve the feature expression of the network, δ is the activation function, and the Relu activation function is used to obtain sparse features. σ is the sigmoid activation function to normalize the weights; the final weight sc has a dimension of 1×1×C.

[0081] (3) The channel attention module 30 multiplies the weight information of each channel feature with the feature of the corresponding channel in the feature map to obtain a weighted feature map (exist Figure 5 Medium It can be specifically expressed by the following formula 3:

[0082]

[0083] Furthermore, the above-mentioned time series feature association module 40 can be implemented as follows: Figure 6 Specifically, the time series feature association model 40 can implement feature fusion according to the following steps:

[0084] (1) The time series feature association model 40 performs dimension conversion on the features of each channel in the weighted feature map. Specifically, the features of each channel can be divided into multiple groups of features.

[0085] Since the framework of video target detection often uses 2D convolution kernels, but the temporal feature association module 40 adopts 3D convolution kernels to obtain the feature connection of the weighted feature map in time series, it is necessary to first transform the dimension of the input x (i.e., the features in the weighted feature map), for example, by adding the time dimension T. Specifically, the features of each channel can be divided according to the time period T, and the dimension of the input x can be converted from N×C×H×W to N / T×C×T×H×W.

[0086] Among them, x is the feature in the weighted feature map, N is the feature length, C is the number of channels of the weighted feature map, H is the height of the weighted feature map, and W is the width of the weighted feature map.

[0087] (2) In order to reduce the amount of calculation, the time series feature association model 40 can perform dimensionality reduction processing on the features in the weighted feature map so that the number of channels of the reduced-dimensional features is smaller than the number of channels of the weighted feature map.

[0088] In this embodiment, several linear mapping functions, θ, φ, and g, are used to reduce the number of channels oc of the input x' after the above-mentioned dimensional conversion by half, for example, to C / 2, and the convolution kernel size of each function is 1×1×1.

[0089] It should be noted that in order to further reduce the amount of calculation, the time series feature association model 40 can first perform step (2) to reduce the dimension of the features in the weighted feature map, and then perform step (1) to convert the dimension of the reduced features, so that only the features of the remaining channels are converted in dimension, without converting the features of the reduced channels in dimension, thereby reducing the amount of calculation for the dimensional conversion.

[0090] (3) The time series feature association model 40 calculates the similarity between adjacent group features or adjacent interval group features of each channel in the reduced dimension feature to obtain the similarity corresponding to each feature in each channel. Specifically, it can be expressed by the following formula 4 and formula 5:

[0091]

[0092] f(x i ,x j )=θ(x i ) T φ(x j ),C(x)=N (5)

[0093] Among them, function f is used to calculate two features x i and x j C(x) is the normalization factor to ensure that the similarity is between 0 and 1, y i There are many ways to implement the similarity calculation function f, such as embedded Gaussian, Gaussian, dot product, etc. The above formula 4 takes dot product as an example.

[0094] (4) The time series feature association model 40 weights the similarity of each feature in each channel and adds it to the corresponding feature, which can be specifically expressed by the following formula 6:

[0095] z i =W z y i +x i (6)

[0096] Among them, W z is the weight matrix, the similarity weight of the feature, its channel number is C / 2, and the channel number of the feature xi is C. In this way, the channel number of the output Zi calculated by the above formula 6 is restored to C; the convolution kernel size is 1×1×1.

[0097] (5) The time series feature association model 40 normalizes the added features and combines the normalized features into a fused feature map. Specifically, it can be expressed by the following formula 7:

[0098] Z BN =BN(z) (7)

[0099] Specifically, a batch normalization (BN) layer is used to normalize the features obtained after the above addition, so that the data stream distribution after processing by the temporal feature association module 40 is consistent with the data stream distribution of the features obtained by the above channel attention module 30.

[0100] It should be noted that, since the similarity between any two features can be calculated directly, the features obtained by the above-mentioned channel attention module 30 are not necessarily optimal. Therefore, in the implementation process of the above-mentioned time series feature association module 40, it is necessary to first extract the features again through the functions of θ, φ, and g, that is, to remove the features of certain channels, and then calculate the similarity. On the one hand, it can improve the expressiveness of the features, and at the same time reduce the amount of calculation.

[0101] The embodiment of the present invention also provides a video object detection device, the structural diagram of which is shown in FIG. Figure 7 As shown, it may specifically include: a feature extraction unit 11, at least one attention unit 12, at least one feature association unit 13 and a detection unit 14, Figure 7 In the example, an attention unit 12 and a feature association unit 13 are used as examples, where:

[0102] The feature extraction unit 11 is used to extract features of each frame image in the video to be detected to obtain a feature map, and the feature map includes features of multiple channels.

[0103] The attention unit 12 is used to perform weighted processing on the features of each channel in the multiple channels in the feature map obtained by the feature extraction unit 11 to obtain a weighted feature map.

[0104] The attention unit 12 is specifically used to perform global average pooling on the features of each channel in the multiple channels to obtain the global features of each channel; determine the weight information of each channel feature according to the global features of each channel; multiply the weight information of each channel feature with the features of the corresponding channel in the feature map to obtain a weighted feature map.

[0105] The feature association unit 13 is used to fuse the features of adjacent frame images or adjacent interval frame images in the video to be detected in the weighted feature map obtained by the attention unit 12 to obtain a fused feature map.

[0106] The feature association unit 13 is specifically used to divide the features of each channel in the weighted feature map into multiple groups of features; calculate the similarities between adjacent groups of features or adjacent interval groups of features of each channel to obtain the similarities corresponding to each feature in each channel; weight the similarities of each feature in each channel and add them to the corresponding features to obtain a fused feature map.

[0107] The feature association unit 13 is further used to, after the features of each channel in the weighted feature map are divided into a plurality of groups of features, first perform dimensionality reduction processing on the features in the weighted feature map, so that the number of channels of the dimensionality reduced features is less than the number of channels of the weighted feature map, and respectively calculate the similarity between the adjacent group features or the adjacent interval group features of each channel in the dimensionality reduced features.

[0108] The feature association unit 13 is further used to normalize the added features and form the fused feature map with the normalized features.

[0109] The detection unit 14 is used to perform target detection on the video to be detected according to the fused feature map obtained by the feature association unit 13.

[0110] Furthermore, the device of this embodiment also includes: a training unit 15, which is used to determine an initial convolutional neural network that performs the feature extraction, weighted processing, feature fusion and target detection steps; determine training samples, wherein the training samples include: multiple groups of training images, each group of training images includes multiple continuous frame images, and target position identification of each frame image in the multiple groups of training images; perform target detection on each frame image in the training sample through the initial convolutional neural network to obtain an initial result of the target position; adjust the fixed parameter value in the initial convolutional neural network according to the initial result of the target position and the target position identification of each frame image in the training sample.

[0111] Furthermore, the training unit 15 is also used to stop adjusting the fixed parameter value if the adjustment of the fixed parameter value meets any of the following stopping conditions: the number of adjustments to the fixed parameter value reaches a preset number, the difference between the current adjusted fixed parameter value and the last adjusted fixed parameter value is less than a threshold, etc.

[0112] It can be seen that in the video target detection device of the present embodiment, after the feature extraction unit 11 extracts the features of each frame image in the video to be detected, the attention unit 12 performs weighted processing on the features of each channel in the multiple channels in the obtained feature map to obtain a weighted feature map, thereby weakening the features of some unimportant channels and strengthening the features of other important channels. In this way, the feature association unit 13, in the process of fusing the features of adjacent frame images or adjacent interval frame images in the video to be detected in the weighted feature map, enables the feature fusion operation to establish a temporal feature association on the basis of good feature quality, taking into account the relationship between each frame image in the video to be detected, and thus the detection unit 14 is more accurate when performing target detection based on the fused feature map.

[0113] The embodiment of the present invention further provides a terminal device, the structural diagram of which is shown in FIG. Figure 8 As shown, the terminal device may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 20 (for example, one or more processors) and memory 21, and one or more storage media 22 (for example, one or more mass storage devices) storing application programs 221 or data 222. Among them, the memory 21 and the storage medium 22 can be short-term storage or permanent storage. The program stored in the storage medium 22 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the terminal device. Furthermore, the central processing unit 20 can be configured to communicate with the storage medium 22 to execute a series of instruction operations in the storage medium 22 on the terminal device.

[0114] Specifically, the application 221 stored in the storage medium 22 includes an application for video target detection, and the program may include the feature extraction unit 11, at least one attention unit 12, at least one feature association unit 13, the detection unit 14 and the training unit 15 in the above-mentioned video target detection device, which are not described in detail here. Furthermore, the central processor 20 can be configured to communicate with the storage medium 22, and execute a series of operations corresponding to the application for video target detection stored in the storage medium 22 on the terminal device.

[0115] The terminal device may also include one or more power supplies 23, one or more wired or wireless network interfaces 24, one or more input and output interfaces 25, and / or one or more operating systems 223, such as Windows Server TM, Mac OS X TM, Unix TM, Linux TM, FreeBSD TM, etc.

[0116] The steps performed by the video object detection device described in the above method embodiment can be based on the Figure 8 The structure of the terminal device shown.

[0117] The present invention also provides a storage medium, wherein the storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded by a processor and executing the video object detection method executed by the video object detection device as described above.

[0118] An embodiment of the present invention further provides a terminal device, including a processor and a storage medium, wherein the processor is used to implement various instructions;

[0119] The storage medium is used to store a plurality of instructions, and the instructions are used to be loaded by a processor and execute the video object detection method executed by the video object detection device as described above.

[0120] A person skilled in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing the relevant hardware through a program, and the program may be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0121] The above is a detailed introduction to a video target detection method, device and storage medium provided by an embodiment of the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A video target detection method, characterized in that: include: Extracting features of each frame image in the video to be detected to obtain a feature map, wherein the feature map includes features of multiple channels; Performing weighted processing on the features of each channel in the multiple channels to obtain a weighted feature map; In the weighted feature map, features of adjacent frame images or adjacent interval frame images in the video to be detected are fused to obtain a fused feature map; Performing target detection on the video to be detected according to the fused feature map; The step of fusing the features of adjacent frame images or adjacent interval frame images in the to-be-detected video in the weighted feature map to obtain a fused feature map specifically includes: Dividing the features of each channel in the weighted feature map into multiple groups of features; Calculating the similarities between adjacent groups of features or adjacent interval groups of features of each channel respectively, so as to obtain the similarities corresponding to each feature in each channel; The similarities of the features in the channels are weighted and added to the corresponding features to obtain a fused feature map.

2. The method according to claim 1, characterized in that The weighted processing of the features of each channel in the multiple channels to obtain a weighted feature map specifically includes: Performing global average pooling on the features of each channel in the multiple channels to obtain a global feature of each channel; Determine the weight information of each channel feature according to the global feature of each channel; The weight information of each channel feature is multiplied by the feature of the corresponding channel in the feature map to obtain a weighted feature map.

3. The method according to claim 1, characterized in that After the features of each channel in the weighted feature map are divided into a plurality of feature groups, the method further includes: Performing dimensionality reduction processing on the features in the weighted feature map so that the number of channels of the features after dimensionality reduction is less than the number of channels of the weighted feature map; Then, respectively calculating the similarities between adjacent group features or adjacent interval group features of each channel specifically includes: respectively calculating the similarities between adjacent group features or adjacent interval group features of each channel in the reduced-dimensional features.

4. The method according to claim 1, characterized in that After weighting the similarities of the features in the channels and adding the weighted similarities to the corresponding features, the method further includes: The added features are normalized, and the normalized features are combined into the fused feature map.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Determine the initial convolutional neural network that performs feature extraction, weighting, feature fusion, and object detection steps; Determine a training sample, wherein the training sample includes: a plurality of groups of training images, each group of training images includes a plurality of continuous frame images, and a target position identifier of each frame image in the plurality of groups of training images; Performing target detection on each frame image in the training sample by using the initial convolutional neural network to obtain an initial result of the target position; According to the initial result of the target position and the target position identifier of each frame image in the training sample, the fixed parameter value in the initial convolutional neural network is adjusted.

6. The method according to claim 5, characterized in that If the adjustment of the fixed parameter value satisfies any of the following stop conditions, the adjustment of the fixed parameter value is stopped: The number of adjustments to the fixed parameter value reaches a preset number, and the difference between the currently adjusted fixed parameter value and the last adjusted fixed parameter value is less than a threshold.

7. A video object detection device, characterized in that: include: A feature extraction unit, at least one attention unit, at least one feature association unit, and a detection unit, wherein: The feature extraction unit is used to extract the features of each frame image in the video to be detected to obtain a feature spectrum, wherein the feature spectrum includes features of multiple channels; The attention unit is used to perform weighted processing on the features of each channel in the multiple channels to obtain a weighted feature map; The feature association unit is used to fuse the features of adjacent frame images or adjacent interval frame images in the video to be detected in the weighted feature map to obtain a fused feature map; The detection unit is used to perform target detection on the video to be detected according to the fused feature map; The feature association unit is specifically used to divide the features of each channel in the weighted feature map into multiple groups of features; calculate the similarities between adjacent groups of features or adjacent interval groups of features of each channel to obtain the similarities corresponding to each feature in each channel; weight the similarities of each feature in each channel and add them to the corresponding features to obtain a fused feature map.

8. The device according to claim 7, characterized in that The attention unit is specifically used to perform global average pooling on the features of each channel in the multiple channels to obtain the global features of each channel; determine the weight information of each channel feature according to the global features of each channel; multiply the weight information of each channel feature with the features of the corresponding channel in the feature map to obtain a weighted feature map.

9. The device according to claim 7, characterized in that The feature association unit is also used to perform dimensionality reduction processing on the features in the weighted feature map after the features of each channel in the weighted feature map are divided into multiple groups of features, so that the number of channels of the reduced-dimensional features is less than the number of channels of the weighted feature map.

10. The device according to any one of claims 7 to 9, characterized in that Also includes: A training unit for determining an initial convolutional neural network for performing feature extraction, weighting processing, feature fusion and target detection steps; Determine a training sample, wherein the training sample includes: multiple groups of training images, each group of training images includes multiple continuous frame images, and target position identification of each frame image in the multiple groups of training images; perform target detection on each frame image in the training sample through the initial convolutional neural network to obtain an initial result of the target position; and adjust the fixed parameter value in the initial convolutional neural network according to the initial result of the target position and the target position identification of each frame image in the training sample.

11. A storage medium, characterized in that: The storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the video object detection method according to any one of claims 1 to 6.

12. A terminal device, characterized in that: It includes a processor and a storage medium, wherein the processor is used to implement various instructions; The storage medium is used to store a plurality of instructions, and the instructions are used to be loaded by a processor and execute the video object detection method according to any one of claims 1 to 6.

Images