End-to-end video target detection method for solving target occlusion based on bionic memory chain

Through the end-to-end video object detection method based on bionic memory chain, the problems of low feature fusion efficiency and poor adaptability to long-term occlusion in the prior art are solved, and efficient timing feature fusion and robustness detection are achieved, especially in occlusion Significant improvement in the scenario.

CN118918311AActive Publication Date: 2024-11-08HUAZHONG UNIV OF SCI & TECH

Patent Information

Application Number
CN202410933365.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-12
Publication Date
2024-11-08
Estimated Expiration
2044-07-12

AI Technical Summary

Technical Problem

When the existing video object detection methods deal with occlusion scenarios, stacked or streaming modeling leads to low feature fusion efficiency, unable to effectively utilize visual features outside the real-time window, and have poor adaptability to long-term occlusion and limited accuracy.

Method used

Using end-to-end video object detection method based on bionic memory chain, design loss function and training through feature pyramid structure network, cross-scale encoder, bionic memory chain, memory buffer and detection head, instance-level annotation and occlusion target instance supervision, design loss function and training Model to improve feature fusion efficiency and robustness.

Benefits of technology

More efficient timing feature fusion is achieved, and the robustness and detection accuracy of occluded targets are improved, especially in long-term occlusion scenarios, and the training speed is also faster.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918311A_ABST
    Figure CN118918311A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of computer vision, and provides an end-to-end video target detection method for solving target occlusion based on a bionic memory chain. The method comprises the steps of performing instance level labeling on targets of interested categories of all images in an original video data set to obtain a standard training data set with labels; defining an end-to-end video target detection algorithm model based on the bionic memory chain and occlusion target instance supervision; according to the standard training data set with the label, utilizing back propagation and gradient descent algorithms to train a model; and performing video target detection on a to-be-detected video by using the trained model. According to the method, time sequence features including the current frame and all historical frames are fused more efficiently, end-to-end training is carried out, the training speed is high, all historical frame information can be represented, weighted memory is adaptively carried out according to the correlation and time proximity of the historical frames and the current frame, and the method has high robustness for a shielded target.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision, and in particular to an end-to-end video target detection method for solving target occlusion based on a bionic memory chain. Background Art

[0002] In the field of computer vision, video object detection is a very active and challenging research direction, which has made significant progress in recent years. Video object detection aims to locate and classify objects in continuous video frames, which is of vital importance to the development of autonomous driving, video surveillance, and general video understanding. However, objects in videos are often occluded by other objects or environmental factors, which poses a huge challenge to detection. It is worth noting that the single-frame features in scenes with occlusion are very limited, and frame-by-frame detection using general object detection methods is often less accurate.

[0003] In order to solve the defects in single-frame features, especially occlusion, existing video target detection methods usually use stacked or streaming modeling to fuse temporal features. Among them, stacked modeling fuses the current frame with historical frames within the visible time range (fixed size) at different times t to detect targets in the current frame. This method has a large amount of calculation and does not utilize visual features outside the immediate time window, resulting in low feature fusion efficiency and reduced adaptability to long-term occlusion. Streaming modeling represents all historical frame information with fixed-size feature blocks. At time t, the feature block at time t-1 is inherited and fused with the current frame. The fused feature block is used to detect targets in the current frame. This method usually cannot assign different weights to all historical frames, resulting in low feature fusion efficiency and limited accuracy.

[0004] In view of this, overcoming the defects of the prior art is an urgent problem to be solved in the field of this technology. Summary of the invention

[0005] The technical problem to be solved by the present invention is that the existing video target detection methods usually use stacked or streaming modeling to fuse temporal features. Stacked modeling has a large amount of calculation and does not utilize visual features outside the immediate time window, resulting in low feature fusion efficiency and reduced adaptability to long-term occlusion. Streaming modeling cannot assign different weights to all historical frames, resulting in low feature fusion efficiency and limited accuracy.

[0006] The present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides an end-to-end video target detection method for solving target occlusion based on a bionic memory chain, comprising:

[0008] Annotate the objects of the category of interest in all images in the original video dataset at the instance level, with the labels being the upper left and lower right vertices of the bounding box at the instance level and the object category corresponding to the instance, to obtain a standard training dataset with annotations; wherein the instance level annotation includes annotation of partially occluded objects;

[0009] Define an end-to-end video target detection algorithm model based on bionic memory chain and occluded target instance supervision; calculate the predicted label according to the labeled standard training data set, design the loss function, and train the end-to-end video target detection algorithm model based on bionic memory chain and occluded target instance supervision using back propagation and gradient descent algorithm;

[0010] Use the trained model to perform video target detection on the video to be detected.

[0011] Preferably, the definition is based on an end-to-end video target detection algorithm model supervised by a bionic memory chain and occluded target instances, specifically including:

[0012] The end-to-end video target detection algorithm model based on bionic memory chain and occluded target instance supervision includes a feature pyramid structure network, a cross-scale encoder, a bionic memory chain, a memory buffer, a memory decoder and a detection head; wherein the detection head includes a frame regression network and a category discrimination network;

[0013] The feature pyramid structure network is based on the HGNetV2-L network. During training, all convolution parameters of the batch normalization layer and the first layer are frozen, and the learning rates of the {1, 2, 3, 4, 5} layers are configured with {0, 0.05, 0.05, 0.05, 0.05} respectively. The features of the {3, 4, 5} layers are output to obtain multi-scale features. The multi-scale features include features of three layers of different scales.

[0014] The cross-scale encoder uses self-attention to calculate the intra-scale interaction features {I1, I2, I3} of the three layers of features of different scales output by the feature pyramid network, and calculates the cross-scale fusion features as output through three layers of continuous combination operations; wherein the combination operation includes: convolution and activation function, batch normalization, matrix addition and series connection;

[0015] The bionic memory chain is composed of four parts: current frame features, previous frame features, strong impressions, and fragmented memories, which are connected in series at a channel ratio of 1:1:1:1. The cross-scale fusion feature is used as input, and the memory information written in the previous frame is read from the memory buffer as input. The features corresponding to the four parts are calculated through the defined attention, repetition, transfer, enhancement, and retrieval operations, and the four parts of the features are connected in series as a feature chain. Finally, the feature chain is used as output and written into the memory buffer for memory update in the next frame.

[0016] The memory buffer caches the memory information of the bionic memory chain at different frames, provides read and write operations, and is used for memory update of the bionic memory chain; wherein the memory information is the feature chain;

[0017] The memory decoder uses a multi-head cross attention calculation with 8 heads to obtain a cross-scale temporal fusion feature on the feature chain output by the bionic memory chain;

[0018] The detection head predicts the bounding box of the instance of interest by using a bounding box regression network for the cross-scale temporal fusion features, and obtains the category discrimination of the instance of interest by using a category discrimination network for the cross-scale temporal fusion features;

[0019] The algorithm model contains a total of 6 detection task heads. After the 6 detection task heads complete the detection of the instance, the final prediction result of the algorithm model is obtained.

[0020] Preferably, the calculating the predicted label according to the annotated standard training data set specifically includes:

[0021] Using standard training data set I tr As the input of the recognition network model, the feature pyramid network module is used to extract multi-scale features;

[0022] According to the multi-scale features output by the feature pyramid module, the cross-scale encoder is used to encode the cross-scale fusion features;

[0023] The cross-scale fusion features encoded by the cross-scale encoder are used as input, the memory information corresponding to the previous frame read from the memory buffer is used as input, and the memory features corresponding to the current frame are updated using the bionic memory chain;

[0024] According to the memory feature corresponding to the current frame obtained by the bionic memory chain, the memory feature is written into the memory buffer for memory update of the next frame;

[0025] According to the memory features corresponding to the current frame obtained by the bionic memory chain, the memory decoder is used to decode and obtain the cross-scale temporal fusion features corresponding to the current frame;

[0026] According to the cross-scale temporal fusion features corresponding to the current frame decoded by the memory decoder, the detection head is used to predict the bounding box and classify the instance of interest as the network prediction output.

[0027] Preferably, the standard training data set I tr As the input of the recognition network model, the feature pyramid network module is used to extract multi-scale features, including:

[0028] The standard training dataset I trIn the bottom-up HGNetV2-L network structure of the image input feature pyramid network, the convolutional layer unit that does not change the size of the feature map in the network is defined as a level, that is, the level {P1, P2, P3, P4, P5}, and the convolutional features c output at the end of each level are extracted, that is, {c1, c2, c3, c4, c5}; {c3, c4, c5} are output as multi-scale features, and the corresponding scales are {1 / 2, 1 / 4, 1 / 8} of the original image size.

[0029] Preferably, the multi-scale features output by the feature pyramid module are encoded using a cross-scale encoder to obtain cross-scale fusion features, specifically including:

[0030] The cross-scale encoder uses self-attention to calculate the intra-scale interaction features {I1, I2, I3} on the three layers of features of different scales output by the feature pyramid network, where the number of self-attention heads used in each layer is 1;

[0031] The cross-scale fusion features are calculated through three layers of continuous combination operations as output, where the convolution stride is 1 and the activation function is SiLU.

[0032] Preferably, the cross-scale fusion feature encoded by the cross-scale encoder is used as input, the memory information corresponding to the previous frame read by the memory buffer is used as input, and the memory feature corresponding to the current frame is updated by using the bionic memory chain, specifically including:

[0033] The bionic memory chain is composed of four parts: current frame features, previous frame features, strong impressions, and fragmented memories, which are connected in series at a channel ratio of 1:1:1:1, namely:

[0034] BMC = [F, S, P, C]

[0035] Among them, BMC stands for bionic memory chain, C stands for current frame features, P stands for previous frame features, S stands for strong impression, and F stands for fragmented memory; the previous frame features and the current frame features are the shallow part of the memory chain, which are used to build short-term memory; fragmented memory and strong impression are the deep part of the memory chain, which are used to build long-term memory;

[0036] The corresponding current frame I output by the bionic memory chain t The memory feature of the component is recorded as [BMC]t, and the corresponding memory feature of each component is recorded as F t , S t , P t With C t , t is the frame number;

[0037] When the current frame number is t, that is, the current frame is I t According to the cross-scale fusion feature f output by the cross-scale encoder t, and the corresponding memory information of the previous frame read from the memory buffer [BMC] t-1 , computed through the defined attention, paraphrase, transfer, enhancement, and retrieval operations [BMC] t .

[0038] Preferably, the calculation of the attention, repetition, transfer, enhancement and retrieval operations defined [BMC] t , specifically including:

[0039] Corresponding memory information from the previous frame [BMC] t-1 Get F t-1 , S t-1 , P t-1 With C t-1 ;

[0040] Use a convolution operator with a convolution kernel size of 1 and a stride of 0 for f t Convolution is performed to obtain C t ; Among them, f t To fuse features across scales;

[0041] Use the convolution operator with a convolution kernel size of 1 and a stride of 0 for C t-1 Perform convolution to get P t ;

[0042] C t-1 Input the cross attention module and t-1 Input the cross attention module, and C is calculated by the cross attention module t-1 Relative P t-1 , and with S t-1 Add together to get S t ;

[0043] C t-1 Input linear mapping module to halve the number of channels and compress features, while F t-1 Another linear mapping module is input to halve the number of channels and compress the features. The outputs of the two linear mapping modules are connected in series to obtain F t ;

[0044] By connecting F in series t , S t , P t With C t Get the corresponding current frame I output by the bionic memory chain t Memory characteristics of [BMC] t .

[0045] Preferably, the memory feature corresponding to the current frame obtained according to the bionic memory chain is written into the memory buffer for memory update of the next frame, specifically including:

[0046] The current frame corresponding memory feature obtained based on the bionic memory chain [BMC] t , memory buffer is decomposed by [BMC] t Get F t , S t , P t With C t , and store F one by one t , S t , P t With C t , which is used to update the memory in the next frame, that is, to update F t+1 , S t+1 , P t+1 With C t+1 .

[0047] Preferably, the cross-scale temporal fusion features of the current frame obtained by decoding the memory decoder are used to predict the bounding box and classify the instance of interest using the detection head as the network prediction output, specifically including:

[0048] According to the current frame corresponding memory feature [BMC]t obtained by the bionic memory chain, the memory decoder is [BMC] t As input, the cross-scale fusion feature f is output by the cross-scale encoder t As input, a multi-head cross attention with 8 heads is used to calculate the cross-scale temporal fusion features of the current frame;

[0049] The detection head consists of a bounding box regression network and a category discrimination network, and the number of detection heads is 6; the bounding box regression network is a network composed of multiple linear mapping layers with an output dimension of 4; the category discrimination network is a network composed of a single linear mapping layer, and its output dimension is the total number of categories of the target of interest; the 6 detection heads contain a total of 6 bounding box regression networks and 6 category discrimination networks;

[0050] The output of each frame regression network is the predicted bounding box coordinate value (χ 1 ,y 1 , χ 2 ,y 2 ), where χ 1 With y 1 The horizontal and vertical coordinates of the upper left corner of the bounding box, χ 2 With y 2 The horizontal and vertical coordinate values ​​of the vertex of the lower right corner of the corresponding bounding box;

[0051] The output of each category discrimination network is an n-dimensional vector, where n is the standard training data set I tr The total number of categories of the target of interest in the , the value of each dimension corresponds to the probability that the target belongs to each category, and the category with the largest probability is the category c to be determined;

[0052] After the six detection task heads complete the instance detection, the final prediction result of the algorithm model is obtained (χ 1 ,y 1 , χ 2 ,y 2 , c).

[0053] Preferably, the design loss function uses back propagation and gradient descent algorithms to train the end-to-end video target detection algorithm model based on bionic memory chain and occluded target instance supervision, specifically including:

[0054] To predict the label (χ 1 ,y 1 , χ 2 ,y 2 , c) is the network prediction output, taking the sample labels in the standard data set as the network expected output, and constructing the corresponding relationship between 300 network prediction outputs and sample labels through the Hungarian matching algorithm;

[0055] Take the training label gt as the expected output of the network and predict the label For the network prediction output, for the constructed network model, the target loss function between the expected output and the predicted output is calculated: based on the matching result of the Hungarian matching algorithm, the loss function is calculated between the network expected output and the network predicted output. The overall target loss function is composed of the bounding box regression loss and the classification loss.

[0056] According to the target loss function, the model is iteratively trained using back propagation and gradient descent algorithms to minimize the target loss function and achieve the optimal network model and parameters.

[0057] In a second aspect, the present invention further provides an end-to-end video target detection device for solving target occlusion based on a bionic memory chain, which is used to implement the end-to-end video target detection method for solving target occlusion based on a bionic memory chain according to the first aspect, and the device comprises:

[0058] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the end-to-end video target detection method for solving target occlusion based on bionic memory chain as described in the first aspect.

[0059] In a third aspect, the present invention further provides a non-volatile computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions are executed by one or more processors to complete the method described in the first aspect.

[0060] In a fourth aspect, a chip is provided, comprising: a processor and an interface, for calling and running a computer program stored in the memory from a memory, and executing the method of the first aspect.

[0061] According to a fifth aspect, a computer program product comprising instructions is provided. When the instructions are executed on a computer or a processor, the computer or the processor executes the method according to the first aspect.

[0062] The present invention constructs an end-to-end video target detection algorithm model based on bionic memory chain and occluded target instance supervision, which can more efficiently fuse temporal features including the current frame and all historical frames, and can perform end-to-end training with a fast training speed. It can also characterize all historical frame information and adaptively weighted memory according to the correlation and temporal proximity between historical frames and current frames. It has strong robustness against occluded targets, especially those that have been occluded for a long time. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0064] Figure 1 It is a flow chart of a first end-to-end video target detection method for solving target occlusion based on a bionic memory chain provided by an embodiment of the present invention;

[0065] Figure 2 It is a structural schematic diagram of an end-to-end video target detection algorithm model based on bionic memory chain and occluded target instance supervision provided by an embodiment of the present invention;

[0066] Figure 3 It is a flow chart of a second end-to-end video target detection method for solving target occlusion based on a bionic memory chain provided by an embodiment of the present invention;

[0067] Figure 4 It is a structural schematic diagram of a feature pyramid module in an end-to-end video target detection method for solving target occlusion based on a bionic memory chain provided by an embodiment of the present invention;

[0068] Figure 5 It is a schematic diagram of the structure of a cross-scale encoder in an end-to-end video target detection method for solving target occlusion based on a bionic memory chain provided by an embodiment of the present invention;

[0069] Figure 6It is a flow chart of a third end-to-end video target detection method for solving target occlusion based on a bionic memory chain provided by an embodiment of the present invention;

[0070] Figure 7 It is a flow chart of a fourth end-to-end video target detection method for solving target occlusion based on a bionic memory chain provided by an embodiment of the present invention;

[0071] Figure 8 It is a schematic diagram of an end-to-end video target detection method for solving target occlusion based on a bionic memory chain provided by an embodiment of the present invention;

[0072] Fig. 9 It is a structural schematic diagram of a memory decoder in an end-to-end video target detection method for solving target occlusion based on a bionic memory chain provided by an embodiment of the present invention;

[0073] Fig.10 It is a flowchart diagram of a fifth end-to-end video target detection method for solving target occlusion based on a bionic memory chain provided by an embodiment of the present invention;

[0074] Fig.11 It is a schematic diagram of the architecture of an end-to-end video target detection device for solving target occlusion based on a bionic memory chain provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0075] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0076] Unless the context requires otherwise, throughout the specification and claims, the term "including" is to be interpreted as open inclusion, that is, "including, but not limited to". In the description of the specification, the terms "one embodiment", "some embodiments", "exemplary embodiments", "examples", "specific examples" or "some examples" and the like are intended to indicate that specific features, structures, materials or characteristics associated with the embodiment or example are included in at least one embodiment or example of the present disclosure. The schematic representation of the above terms does not necessarily refer to the same embodiment or example. In addition, the specific features, structures, materials or characteristics may be included in any one or more embodiments or examples in any appropriate manner, that is, although they may be carried in the embodiments or examples of the above terms due to reasons such as the order and position of appearance, it is not limited to that they can be carried in combination by one embodiment or example.

[0077] In the description of the present invention, the terms "first" and "second" are used only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, unless otherwise specified, the meaning of "multiple" is two or more. In addition, for example, the same type of nouns may be described as two independent individuals by adding "A" and "B" at the end. In this case, the corresponding features defined as "A" and "B" are only used to distinguish the same type of individuals for description purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features.

[0078] In the description of the present invention, the expression "A and / or B" (where A and B are used to formally represent specific characteristic contents) will be involved, and the corresponding expressions include the following three combinations: only A, only B, and a combination of A and B.

[0079] As used herein, "about," "substantially," or "approximately" includes the stated value and an average value that is within an acceptable range of deviation from the particular value as determined by one of ordinary skill in the art taking into account the measurements in question and the errors associated with the measurement of the particular quantity (i.e., the limitations of the measurement system).

[0080] In order to make the technical solution of the present invention more clear, the professional terms used in the present invention are explained here:

[0081] HGNetV2-L: A neural network that can be used for classification. The network uses a multi-level feature extraction mechanism, including convolutional layers, pooling layers, and residual connection layers. The convolutional layer is used to extract image features; the pooling layer is used to reduce the dimension of the feature vector output by the convolutional layer to reduce overfitting; the residual connection layer is used to transfer gradients to solve the vanishing and exploding gradient problems. The network parameters can be updated through backpropagation and gradient descent algorithms.

[0082] SiLU: An activation function that has been widely used in recent years. Its mathematical expression is SiLU(x)=x*sigmoid(x), where sigmoid(x) is the standard Sigmoid function. Compared with traditional activation functions such as ReLU, SiLU has smooth nonlinear characteristics, can more stably transmit gradients and retain negative information, thereby enhancing the expressiveness of neural networks.

[0083] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0084] Embodiment 1:

[0085] Embodiment 1 of the present invention provides an end-to-end video target detection method for solving target occlusion based on a bionic memory chain, such as Figure 1 As shown, including:

[0086] In step 201, the targets of the category of interest in all images in the original video dataset are annotated at the instance level, and the labels are the upper left and lower right vertices of the instance-level bounding box and the target category corresponding to the instance, so as to obtain a standard training dataset with annotations; wherein the instance-level annotation includes annotating partially occluded targets.

[0087] In step 202, an end-to-end video target detection algorithm model based on bionic memory chain and occluded target instance supervision is defined; based on the labeled standard training data set, the predicted label is calculated, and the loss function is designed, and the end-to-end video target detection algorithm model based on bionic memory chain and occluded target instance supervision is trained using back propagation and gradient descent algorithms.

[0088] In step 203, the trained model is used to perform video object detection on the video to be detected.

[0089] The present invention constructs an end-to-end video target detection algorithm model based on bionic memory chain and occluded target instance supervision, which can more efficiently fuse temporal features including the current frame and all historical frames, and can perform end-to-end training with a fast training speed. It can also characterize all historical frame information and adaptively weighted memory according to the correlation and temporal proximity between historical frames and current frames. It has strong robustness against occluded targets, especially those that have been occluded for a long time.

[0090] The definition is based on an end-to-end video target detection algorithm model supervised by bionic memory chain and occluded target instance, such as Figure 2 As shown, specifically including:

[0091] The end-to-end video target detection algorithm model based on bionic memory chain and occluded target instance supervision includes a feature pyramid structure network, a cross-scale encoder, a bionic memory chain, a memory buffer, a memory decoder and a detection head; wherein the detection head includes a border regression network and a category discrimination network.

[0092] The feature pyramid structure network is based on the HGNetV2-L network. During training, the batch normalization layer and all convolution parameters of the first layer are frozen, and the learning rates of the {1, 2, 3, 4, 5} layers are configured with {0, 0.05, 0.05, 0.05, 0.05} respectively. The features of the {3, 4, 5} layers are output to obtain multi-scale features. The multi-scale features include features of three layers of different scales.

[0093] The cross-scale encoder uses self-attention to calculate the intra-scale interaction features {I1, I2, I3} of the three layers of features of different scales output by the feature pyramid network, and calculates the cross-scale fusion features as output through three layers of continuous combination operations; wherein the combination operation includes: convolution and activation function, batch normalization, matrix addition and series connection.

[0094] The bionic memory chain is composed of four parts, namely, current frame features, previous frame features, strong impressions, and fragmented memories, which are connected in series at a channel ratio of 1:1:1:1. The cross-scale fusion features are used as input, and the memory information written in the previous frame is read from the memory buffer as input. The features corresponding to the four parts are calculated through the defined attention, repetition, transfer, enhancement, and retrieval operations, and the four-part features are connected in series into a feature chain. Finally, the feature chain is used as output and written into the memory buffer for memory update in the next frame.

[0095] The memory buffer caches the memory information of the bionic memory chain at different frames, provides read and write operations, and is used for memory update of the bionic memory chain; wherein the memory information is a feature chain.

[0096] The memory decoder uses a multi-head cross attention calculation with 8 heads to calculate the feature chain output by the bionic memory chain to obtain a cross-scale temporal fusion feature.

[0097] The detection head predicts the bounding box of the instance of interest using a bounding box regression network for the cross-scale temporal fusion features, and obtains the category discrimination of the instance of interest using a category discrimination network for the cross-scale temporal fusion features; the instance of interest is the target to be detected, and the category discrimination obtains the category of the target.

[0098] The algorithm model includes a total of 6 detection task heads (i.e., 6 detection heads). After the 6 detection task heads complete the detection of the instance, the final prediction result of the algorithm model is obtained. The prediction result is the bounding box of the target and the category of the target finally output by the 6 detection heads.

[0099] Wherein, the predicted label is calculated based on the labeled standard training data set, such as Figure 3 As shown, specifically including:

[0100] In step 301, a standard training data set I is used. tr As the input of the recognition network model, the feature pyramid network module is used to extract multi-scale features.

[0101] In step 302, a cross-scale encoder is used to encode the multi-scale features output by the feature pyramid module to obtain a cross-scale fusion feature.

[0102] In step 303, the cross-scale fusion features encoded by the cross-scale encoder are used as input, the memory information corresponding to the previous frame read from the memory buffer is used as input, and the memory features corresponding to the current frame are updated using the bionic memory chain.

[0103] In step 304, the memory feature corresponding to the current frame obtained by the bionic memory chain is written into the memory buffer for memory update of the next frame.

[0104] In step 305, based on the memory features corresponding to the current frame obtained by the bionic memory chain, a memory decoder is used to decode the cross-scale temporal fusion features corresponding to the current frame; the memory features are the above-mentioned feature chain.

[0105] In step 306, based on the cross-scale temporal fusion features corresponding to the current frame decoded by the memory decoder, the detection head is used to predict the bounding box and classify the instance of interest as the network prediction output.

[0106] The predicted label is the network prediction output.

[0107] In actual application scenarios, the standard training data set I tr As the input of the recognition network model, the feature pyramid network module is used to extract multi-scale features, such as Figure 4 As shown, specifically including:

[0108] The standard training dataset I tr In the bottom-up HGNetV2-L network structure of the image input feature pyramid network, the convolution layer unit that does not change the size of the feature map in the network is defined as a level, that is, the level {P1, P2, P3, P4, P5}, and the convolution feature c outputted at the end of each level is extracted, that is, {c1, c2, c3, c4, c5}; {c3, c4, c5} is output as multi-scale features, and the corresponding scales are {1 / 2, 1 / 4, 1 / 8} of the original image size.

[0109] In an optional implementation, the multi-scale features output by the feature pyramid module are encoded using a cross-scale encoder to obtain a cross-scale fusion feature, such as Figure 5 As shown, specifically including:

[0110] The cross-scale encoder uses self-attention to calculate the intra-scale interaction features {I1, I2, I3} on the three layers of features of different scales output by the feature pyramid network, where the number of self-attention heads used in each layer is 1.

[0111] The cross-scale fusion features are calculated through three layers of continuous combination operations as output, where the convolution stride is 1 and the activation function is SiLU.

[0112] The cross-scale fusion feature obtained by encoding with the cross-scale encoder is used as input, the memory information corresponding to the previous frame read from the memory buffer is used as input, and the memory feature corresponding to the current frame is updated by using the bionic memory chain, such as Figure 6 As shown, specifically including:

[0113] In step 401, the bionic memory chain is composed of four parts: current frame features, previous frame features, strong impressions, and fragmented memories, which are connected in series at a channel ratio of 1:1:1:1, that is:

[0114] BMC = [F, S, P, C]

[0115] Among them, BMC stands for bionic memory chain, that is, feature chain, C represents the current frame feature, P represents the previous frame feature, S represents strong impression, and F represents fragmented memory; the previous frame feature and the current frame feature are the shallow part of the memory chain, which are used to construct short-term memory; fragmented memory and strong impression are the deep part of the memory chain, which are used to construct long-term memory.

[0116] The corresponding current frame I output by the bionic memory chain t The memory feature of the component is recorded as [BMC]t, and the corresponding memory feature of each component is recorded as F t , S t , P t With C t , t is the frame number.

[0117] In step 402, when the current frame number is t, that is, the current frame is I t According to the cross-scale fusion feature f output by the cross-scale encoder t , and the corresponding memory information of the previous frame read from the memory buffer [BMC] t-1 , computed through the defined attention, paraphrase, transfer, enhancement, and retrieval operations [BMC] t .

[0118] Wherein, the following operations are defined: attention, repetition, transfer, enhancement and retrieval operations are calculated [BMC] t ,like Figure 7 As shown, specifically including:

[0119] In step 501, the corresponding memory information [BMC] from the previous frame t-1 Get F t-1 , S t-1 , P t-1 With C t-1 .

[0120] In step 502, a convolution operator with a convolution kernel size of 1 and a stride of 0 is used for f t Convolution is performed to obtain C t; Among them, f t It is a cross-scale fusion feature.

[0121] In step 503, a convolution operator with a convolution kernel size of 1 and a stride of 0 is used to perform the convolution on C t-1 Perform convolution to get P t .

[0122] In step 504, C t-1 Input the cross attention module and t-1 Input the cross attention module, and C is calculated by the cross attention module t-1 Relative P t-1 , and with S t-1 Add together to get S t .

[0123] In step 505, C t-1 Input linear mapping module to halve the number of channels and compress features, while F t-1 Another linear mapping module is input to halve the number of channels and compress the features. The outputs of the two linear mapping modules are connected in series to obtain F t .

[0124] In step 506, by connecting F t , S t , P t With C t Get the corresponding current frame I output by the bionic memory chain t Memory characteristics of [BMC] t ,like Figure 8 shown.

[0125] The memory feature corresponding to the current frame obtained according to the bionic memory chain is written into the memory buffer for memory update of the next frame, specifically including:

[0126] The current frame corresponding memory feature obtained based on the bionic memory chain [BMC] t , memory buffer is decomposed by [BMC] t Get F t , S t , P t With C t , and store F one by one t , S t , P t With C t , which is used to update the memory in the next frame, that is, to update F t+1 , S t+1 , P t+1 With C t+1 ,like Figure 8 shown.

[0127] The cross-scale temporal fusion features corresponding to the current frame obtained by decoding the memory decoder are used to predict the bounding box and classify the instance of interest using the detection head as the network prediction output, specifically including:

[0128] The current frame corresponding memory feature obtained based on the bionic memory chain [BMC] t , memory decoder with [BMC] t As input, the cross-scale fusion feature f is output by the cross-scale encoder t As input, a multi-head cross attention calculation with 8 heads is used to obtain the cross-scale temporal fusion features of the current frame, such as Fig. 9 shown.

[0129] The detection head consists of a bounding box regression network and a category discrimination network, and the number of detection heads is 6; the bounding box regression network is a network composed of multiple linear mapping layers with an output dimension of 4; the category discrimination network is a network composed of a single linear mapping layer, and its output dimension is the total number of categories of objects of interest; the 6 detection heads contain a total of 6 bounding box regression networks and 6 category discrimination networks.

[0130] The output of each frame regression network is the predicted bounding box coordinate value (χ 1 ,y 1 , χ 2 ,y 2 ), where χ 1 With y 1 The horizontal and vertical coordinates of the upper left corner of the bounding box, χ 2 With y 2 The horizontal and vertical coordinate values ​​corresponding to the vertex of the lower right corner of the bounding box.

[0131] The output of each category discrimination network is an n-dimensional vector, where n is the standard training data set I tr The total number of categories of the target of interest in , the value of each dimension corresponds to the probability that the target belongs to each category, and the category with the largest probability is the category c to be determined.

[0132] After the six detection task heads complete the instance detection, the final prediction result of the algorithm model is obtained (χ 1 ,y 1 , χ 2 ,y 2 , c).

[0133] The loss function is designed to train the end-to-end video target detection algorithm model based on bionic memory chain and occluded target instance supervision using back propagation and gradient descent algorithms, such as Fig.10 As shown, specifically including:

[0134] In step 601, the predicted label (1 ,y 1 , χ 2 ,y 2 , c) is the network prediction output, and the sample label in the standard data set is taken as the network expected output. Through the Hungarian matching algorithm, a corresponding relationship between 300 network prediction outputs and sample labels is constructed; the predicted label is the prediction result obtained by the model prediction.

[0135] In step 602, the training label gt is used as the network expected output, and the predicted label For the network prediction output, for the constructed network model, the target loss function between the expected output and the predicted output is calculated: based on the matching result of the Hungarian matching algorithm, the loss function is calculated between the network expected output and the network predicted output. The overall target loss function is composed of the bounding box regression loss and the classification loss. The training label gt is the annotation result. Prediction label Among them, χ 1 ,y 1 , χ 2 ,y 2 is the bounding box of the target, and c is the category of the target.

[0136] In step 603, according to the target loss function, the model is iteratively trained using back propagation and gradient descent algorithms to minimize the target loss function and achieve the optimal network model and parameters.

[0137] After the training is completed, the trained end-to-end video target detection algorithm model based on bionic memory chain and occluded target instance supervision (also referred to as the trained model) can be used for video target detection, including:

[0138] First, the video to be detected is passed through the HGNetV2-L backbone network and the feature pyramid network frame by frame to obtain the multi-scale features corresponding to the current frame. The features are then passed through the cross-scale encoder to obtain the cross-scale fusion features corresponding to the current frame.

[0139] Secondly, the cross-scale fusion features corresponding to the current frame and the memory information corresponding to the previous frame read from the memory buffer are input into the bionic memory chain to obtain the memory features corresponding to the current frame, and written into the memory buffer for the next memory update.

[0140] Again, the memory features corresponding to the current frame are passed through the memory decoder to obtain the cross-scale temporal fusion features corresponding to the current frame.

[0141] Finally, the cross-scale temporal fusion features corresponding to the current frame are input into the detection head, and the bounding box regression and category discrimination are performed through the bounding box regression network and the category discrimination network to obtain the predicted bounding box and category of the instance of interest in the current frame image. After frame-by-frame processing, the predicted bounding boxes and categories of all instances of interest in the video to be detected are obtained.

[0142] Compared with the prior art, the above technical solution conceived in this embodiment has the following beneficial effects:

[0143] (1) High efficiency of temporal feature fusion: This embodiment addresses the problem of target occlusion in video target detection by constructing an end-to-end video target detection algorithm model based on bionic memory chain and occluded target instance supervision to more efficiently fuse temporal features including the current frame and all historical frames.

[0144] (2) Fast computing speed: The end-to-end video target detection method based on bionic memory chain and occluded target instance supervision proposed in this embodiment can be trained end-to-end with a fast training speed. The inference speed on a single NVIDIA RTX 3090 GPU is not less than 16FPS.

[0145] (3) Strong robustness: This embodiment proposes an end-to-end video target detection method based on a bionic memory chain to solve video target occlusion. It simulates the long-term and short-term memory of the human brain with a fixed-size feature chain, can characterize all historical frame information, and adaptively weight the memory according to the correlation and temporal proximity between the historical frames and the current frame. It has strong robustness against occluded targets, especially those that have been occluded for a long time.

[0146] Embodiment 2:

[0147] The present invention is based on the method described in Example 1, combined with specific application scenarios, and uses technical descriptions in related scenarios to illustrate the implementation process of the present invention in characteristic scenarios.

[0148] In view of the defects of the prior art, this embodiment aims to design an economical and efficient end-to-end video target detection algorithm based on bionic memory chain to solve video target occlusion. This embodiment provides an economical and efficient end-to-end video target detection algorithm based on bionic memory chain to solve video target occlusion. This embodiment simulates the long-term and short-term memory of the human brain with a fixed-size feature chain, which can characterize all historical frame information and adaptively weight the memory according to the correlation and temporal proximity between the historical frame and the current frame. At time t, the current frame is used as a stimulus to update the memory of time t-1, so as to achieve the re-weighting of all historical features. It is worth noting that the bionic memory chain module we designed does not perform the re-weighting operation explicitly, which is very time-consuming and has no practical significance. The bionic memory chain module is composed of four parts, namely "fragmented memory", "strong impression", "previous frame feature" and "current frame feature", which are connected in series in a chain form, emphasizing the characterization of different areas of the human brain by different positions of the memory chain, and simulating the natural update of memory by the human brain from top to bottom and implicitly.

[0149] The end-to-end video target detection method for solving video target occlusion based on the bionic memory chain in this embodiment includes the following sub-steps:

[0150] (1) Training an end-to-end video object detection algorithm model based on bionic memory chain and occluded object instance supervision includes the following sub-steps:

[0151] (1.1) Perform instance-level annotation on the objects of the interested category (including partially occluded objects) in all images in the original video dataset. The labels are the upper left and lower right vertices of the instance-level bounding box and the object category corresponding to the instance, and a standard training dataset with annotations is obtained.

[0152] (1.2) defining an end-to-end video target detection algorithm model based on bionic memory chain and occluded target instance supervision, wherein the end-to-end video target detection algorithm model based on bionic memory chain and occluded target instance supervision is composed of a feature pyramid structure network, a cross-scale encoder, a bionic memory chain (BMC), a memory buffer, a memory decoder, and a detection head (including a bounding box regression network and a category discrimination network). According to the annotated standard training data set in (1.1), the training labels are calculated, and the loss function is designed. The end-to-end video target detection algorithm model based on bionic memory chain and occluded target instance supervision is trained using back propagation and gradient descent algorithms. Specifically, the model includes the following sub-steps:

[0153] (1.2.1) Constructing an end-to-end video target detection algorithm model based on bionic memory chain and occluded target instance supervision: The end-to-end video target detection algorithm model based on bionic memory chain and occluded target instance supervision consists of a feature pyramid structure network, a cross-scale encoder, a bionic memory chain (BMC), a memory buffer, a memory decoder, and a detection head; the feature pyramid structure network is based on the HGNetV2-L network, and all convolution parameters of the batch normalization layer and the first layer are frozen during training, and the {1, 2, 3, 4, 5} layers are configured respectively. The learning rate is set to {0, 0.05, 0.05, 0.05, 0.05}, and the features of the {3rd, 4th, 5th} layers are output to obtain multi-scale features; the cross-scale encoder uses self-attention to calculate the intra-scale interaction features {I1, I2, I3} of the three layers of features of different scales output by the feature pyramid network, and calculates the cross-scale fusion features as output through three layers of continuous combination operations (including convolution and activation functions, batch normalization, matrix addition, and series connection); the bionic memory chain consists of "current frame features", "previous frame features", "strong The four parts of "impression" and "fragmented memory" are connected in series with a channel ratio of 1:1:1:1, with cross-scale fusion features as input, and the memory information written in the previous frame is read from the memory buffer as input. The features corresponding to the four parts are calculated through the defined "attention", "retelling", "transfer", "enhancement" and "retrieval" operations, and the four parts of the features are connected in series as a feature chain (i.e., memory information), and finally the feature chain is used as output and written into the memory buffer for memory update in the next frame; the memory buffer can cache the memory information of the bionic memory chain in different frames, provide read and write operations, and be used for memory update of the bionic memory chain; the memory decoder uses a multi-head cross attention calculation with a head number of 8 to obtain the cross-scale temporal fusion feature for the feature chain output by the bionic memory chain; the detection head predicts the bounding box of the instance of interest by the border regression network for the cross-scale temporal fusion feature, and obtains the category discrimination of the instance of interest by the category discrimination network for the cross-scale temporal fusion feature; the algorithm model contains a total of 6 detection task heads, and after the 6 detection task heads complete the detection of the instance, the final prediction result of the algorithm model is obtained.

[0154] (1.2.2) Using standard training data set I tr As the input of the recognition network model, the feature pyramid network module is used to extract multi-scale features: the standard training dataset I trIn the bottom-up HGNetV2-L network structure of the image input feature pyramid network, the convolution layer unit that does not change the size of the feature map in the network is defined as a level, that is, the level {P1, P2, P3, P4, P5}, and the convolution feature c outputted at the end of each level is extracted, that is, {c1, c2, c3, c4, c5}; {c3, c4, c5} is output as a multi-scale feature, and the corresponding scales are {1 / 2, 1 / 4, 1 / 8} of the original image size.

[0155] (1.2.3) According to the multi-scale features output by the feature pyramid module, the cross-scale encoder is used to encode the cross-scale fusion features: the cross-scale encoder uses self-attention to calculate the intra-scale interaction features {I1, I2, I3} of the three layers of features of different scales output by the feature pyramid network, where the number of self-attention heads used in each layer is 1; and the cross-scale fusion features are calculated as output through three layers of continuous combination operations (including convolution and activation function, batch normalization, matrix addition, and serial connection), where the stride of the convolution is 1 and the activation function is SiLU.

[0156] (1.2.4) Taking the cross-scale fusion feature encoded by the cross-scale encoder as input and the memory information corresponding to the previous frame read from the memory buffer as input, the memory feature corresponding to the current frame is updated by using the bionic memory chain: the bionic memory chain (BMC) is composed of four parts, namely, "current frame feature" (C), "previous frame feature" (P), "strong impression" (S), and "fragmented memory" (F), which are connected in series at a channel ratio of 1:1:1:1, that is:

[0157] BMC=[F, S, P, C]

[0158] Among them, the previous frame feature (P) and the current frame feature (C) are the shallow part of the memory chain, corresponding to the hippocampus in the human brain, which is used to build short-term memory; fragmented memory (F) and strong impression (S) are the deep part of the memory chain, corresponding to the human cerebral cortex, which is used to build long-term memory. t The memory feature (i.e. feature chain) is denoted as [BMC] t , the corresponding memory features of each part are recorded as F t , S t , P t With C t , t is the frame number.

[0159] When the current frame number is t (i.e., the current frame is I t ), according to the cross-scale fusion feature f output by the cross-scale encoder t , and the corresponding memory information of the previous frame read from the memory buffer [BMC] t-1, calculated by defining the operations of "attention", "retelling", "transfer", "enhancement" and "retrieval" [BMC] t The detailed process is as follows:

[0160] (a) Corresponding memory information from the previous frame [BMC] t-1 Get F t-1 , S t-1 , P t-1 With C t-1 .

[0161] (b) Focus: Use the convolution operator with a kernel size of 1 and a stride of 0 for f t Convolution is performed to obtain C t .

[0162] (c) Restatement: Use a convolution operator with a kernel size of 1 and a stride of 0 to perform t-1 Perform convolution to get P t .

[0163] (d) Transfer: The transfer operation is also used to calculate F t With S t In calculating S t When C t-1 Input the cross attention module (this operation is called transfer), and at the same time t-1 Input the cross attention module, and C is calculated by the cross attention module t-1 Relative P t-1 , and with S t-1 Add together to get S t In calculating F t When C t-1 Input linear mapping module (this operation is called transfer) to halve the number of channels and compress features, while F t-1 Another linear mapping module is input to halve the number of channels and compress the features. The outputs of the two linear mapping modules are connected in series to obtain F t .

[0164] (e) Enhancement: The enhancement operation is used to calculate S t In calculating S t When C t-1 With P t-1 Input the cross attention module, and C is calculated by the cross attention module t-1 Relative P t-1 The strong similarity feature is combined with S t-1 Add (this operation is called enhancement) to get S t .

[0165] (f) Retrieval: The retrieval operation is used to calculate F t In calculating Ft When C t-1 Input linear mapping module to halve the number of channels and compress features, while F t-1 Another linear mapping module is input (this operation is called retrieval) to halve the number of channels and compress the features. The outputs of the two linear mapping modules are connected in series to obtain F t .

[0166] (g) By connecting F in series t , S t , P t With C t Get the corresponding current frame I output by the bionic memory chain t Memory characteristics of [BMC] t .

[0167] (1.2.5) According to the memory feature corresponding to the current frame obtained by the bionic memory chain, the memory feature is written into the memory buffer for memory update of the next frame: According to the memory feature corresponding to the current frame obtained by the bionic memory chain [BMC] t , memory buffer is decomposed by [BMC] t Get F t , S t , P t With C t , and store F one by one t , S t , P t With C t , which is used to update the memory in the next frame, that is, to update F t+1 , S t+1 , P t+1 With C t+1 .

[0168] (1.2.6) According to the memory feature corresponding to the current frame obtained by the bionic memory chain, the memory decoder is used to decode and obtain the cross-scale temporal fusion feature corresponding to the current frame: According to the memory feature corresponding to the current frame obtained by the bionic memory chain [BMC] t , memory decoder with [BMC] t As input, the cross-scale fusion feature f is output by the cross-scale encoder t As input, a multi-head cross attention with 8 heads is used to calculate the cross-scale temporal fusion features of the current frame.

[0169] (1.2.7) According to the cross-scale temporal fusion features corresponding to the current frame decoded by the memory decoder, the detection head is used to predict the bounding box and class discrimination of the instance of interest as the network prediction output: the detection head is composed of a bounding box regression network and a class discrimination network, where the number of detection heads is 6; the bounding box regression network is a network with an output dimension of 4 composed of multiple linear mapping layers; the class discrimination network is a network composed of a single linear mapping layer, and its output dimension is the total number of categories of the target of interest. The 6 detection heads contain a total of 6 bounding box regression networks and 6 class discrimination networks. The output of each bounding box regression network is the predicted bounding box coordinate value (χ 1 ,y 1 , χ 2 ,y 2 ), where χ 1 With y 1 The horizontal and vertical coordinates of the upper left corner of the bounding box, χ 2 With y 2 The horizontal and vertical coordinates of the vertex in the lower right corner of the bounding box. The output of each category discrimination network is an n-dimensional vector, where n is the standard training data set I tr The total number of categories of the target of interest in the , the value of each dimension corresponds to the probability that the target belongs to each category, and the category with the largest probability is the category c determined to belong to. After the six detection task heads complete the detection of the instance, the final prediction result of the algorithm model is obtained (χ 1 ,y 1 , χ 2 ,y 2 , c).

[0170] (1.2.8) Use the Hungarian matching algorithm to match the network prediction output with the training set label set one by one: use the prediction label (χ 1 ,y 1 , χ 2 ,y 2 , c) is the network prediction output, and the sample labels in the standard data set are taken as the network expected output. The corresponding relationship between 300 network prediction outputs and sample labels is constructed through the Hungarian matching algorithm.

[0171] (1.2.9) Take the training label gt as the expected output of the network and the predicted label For the network prediction output, for the constructed network model, the target loss function between the expected output and the predicted output is calculated: based on the matching result of the Hungarian matching algorithm, the loss function is calculated between the network expected output and the network predicted output. The overall target loss function is composed of the bounding box regression loss and the classification loss.

[0172] (1.2.10) According to the target loss function, the model is iteratively trained using back propagation and gradient descent algorithms to minimize the target loss function and achieve the optimal network model and parameters.

[0173] (2) Using the trained model to detect the video object, including the following sub-steps:

[0174] (2.1) First, the video to be detected is passed through the HGNetV2-L backbone network and the feature pyramid network frame by frame to obtain the multi-scale features corresponding to the current frame. The features are then passed through the cross-scale encoder to obtain the cross-scale fusion features corresponding to the current frame.

[0175] (2.2) Secondly, the cross-scale fusion features corresponding to the current frame and the memory information corresponding to the previous frame read from the memory buffer are input into the bionic memory chain to obtain the memory features corresponding to the current frame, and written into the memory buffer for the next memory update.

[0176] (2.3) Again, the memory features corresponding to the current frame are passed through the memory decoder to obtain the cross-scale temporal fusion features corresponding to the current frame.

[0177] (2.4) Finally, the cross-scale temporal fusion features corresponding to the current frame are input into the detection head, and bounding box regression and category discrimination are performed through the bounding box regression network and the category discrimination network to obtain the predicted bounding box and category of the instance of interest in the current frame image. After frame-by-frame processing, the predicted bounding boxes and categories of all instances of interest in the video to be detected are obtained.

[0178] In general, the above technical solution conceived by this embodiment has the following beneficial effects compared with the prior art:

[0179] (1) High efficiency of temporal feature fusion: This embodiment addresses the problem of target occlusion in video target detection by constructing an end-to-end video target detection algorithm model based on bionic memory chain and occluded target instance supervision to more efficiently fuse temporal features including the current frame and all historical frames.

[0180] (2) Fast computing speed: The end-to-end video target detection method based on bionic memory chain and occluded target instance supervision proposed in this embodiment can be trained end-to-end with a fast training speed. The inference speed on a single NVIDIA RTX 3090 GPU is not less than 16FPS.

[0181] (3) Strong robustness: This embodiment proposes an end-to-end video target detection method based on a bionic memory chain to solve video target occlusion. It simulates the long-term and short-term memory of the human brain with a fixed-size feature chain, can characterize all historical frame information, and adaptively weight the memory according to the correlation and temporal proximity between the historical frames and the current frame. It has strong robustness against occluded targets, especially those that have been occluded for a long time.

[0182] Embodiment 3:

[0183] like Fig.11 , is a schematic diagram of the architecture of an end-to-end video target detection device for solving target occlusion based on a bionic memory chain according to an embodiment of the present invention. The end-to-end video target detection device for solving target occlusion based on a bionic memory chain according to this embodiment includes one or more processors 21 and a memory 22. Fig.11 A processor 21 is taken as an example.

[0184] The processor 21 and the memory 22 may be connected via a bus or other means. Fig.11 The example of connecting through bus is taken in the following.

[0185] The memory 22, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs and non-volatile computer executable programs, such as the end-to-end video target detection method for solving target occlusion based on the bionic memory chain in Example 1. The processor 21 executes the end-to-end video target detection method for solving target occlusion based on the bionic memory chain by running the non-volatile software programs and instructions stored in the memory 22.

[0186] The memory 22 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some embodiments, the memory 22 may optionally include a memory remotely arranged relative to the processor 21, and these remote memories may be connected to the processor 21 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0187] The program instructions / modules are stored in the memory 22, and when executed by the one or more processors 21, the end-to-end video target detection method for solving target occlusion based on the bionic memory chain in the above-mentioned embodiment 1 is executed.

[0188] It is worth noting that the information interaction, execution process, etc. between the modules and units within the above-mentioned devices and systems are based on the same concept as the processing method embodiment of the present invention. The specific contents can be found in the description of the method embodiment of the present invention and will not be repeated here.

[0189] A person skilled in the art may understand that all or part of the steps in the various methods of the embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0190] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An end-to-end video target detection method based on bionic memory chain to solve target occlusion, characterized in that: include: Annotate the objects of the category of interest in all images in the original video dataset at the instance level, with the labels being the upper left and lower right vertices of the bounding box at the instance level and the object category corresponding to the instance, to obtain a standard training dataset with annotations; wherein the instance level annotation includes annotation of partially occluded objects; Define an end-to-end video target detection algorithm model based on bionic memory chain and occluded target instance supervision; calculate the predicted label according to the labeled standard training data set, design the loss function, and train the end-to-end video target detection algorithm model based on bionic memory chain and occluded target instance supervision using back propagation and gradient descent algorithm; Use the trained model to perform video target detection on the video to be detected.

2. The end-to-end video target detection method for solving target occlusion based on bionic memory chain according to claim 1 is characterized in that: The definition is based on an end-to-end video target detection algorithm model supervised by bionic memory chain and occluded target instance, specifically including: The end-to-end video target detection algorithm model based on bionic memory chain and occluded target instance supervision includes a feature pyramid structure network, a cross-scale encoder, a bionic memory chain, a memory buffer, a memory decoder and a detection head; wherein the detection head includes a frame regression network and a category discrimination network; The feature pyramid structure network is based on the HGNetV2-L network. During training, all convolution parameters of the batch normalization layer and the first layer are frozen, and the learning rates of the {1, 2, 3, 4, 5} layers are configured with {0, 0.05, 0.05, 0.05, 0.05} respectively. The features of the {3, 4, 5} layers are output to obtain multi-scale features. The multi-scale features include features of three layers of different scales. The cross-scale encoder uses self-attention to calculate the intra-scale interaction features {I1, I2, I3} of the three layers of features of different scales output by the feature pyramid network, and calculates the cross-scale fusion features as output through three layers of continuous combination operations; wherein the combination operation includes: convolution and activation function, batch normalization, matrix addition and series connection; The bionic memory chain is composed of four parts: current frame features, previous frame features, strong impressions, and fragmented memories, which are connected in series at a channel ratio of 1:1:1:

1. The cross-scale fusion feature is used as input, and the memory information written in the previous frame is read from the memory buffer as input. The features corresponding to the four parts are calculated through the defined attention, repetition, transfer, enhancement, and retrieval operations, and the four parts of the features are connected in series as a feature chain. Finally, the feature chain is used as output and written into the memory buffer for memory update in the next frame. The memory buffer caches the memory information of the bionic memory chain at different frames, provides read and write operations, and is used for memory update of the bionic memory chain; wherein the memory information is the feature chain; The memory decoder uses a multi-head cross attention calculation with 8 heads to obtain a cross-scale temporal fusion feature on the feature chain output by the bionic memory chain; The detection head predicts the bounding box of the instance of interest by using a bounding box regression network for the cross-scale temporal fusion features, and obtains the category discrimination of the instance of interest by using a category discrimination network for the cross-scale temporal fusion features; The algorithm model contains a total of 6 detection task heads. After the 6 detection task heads complete the detection of the instance, the final prediction result of the algorithm model is obtained.

3. The end-to-end video target detection method based on bionic memory chain to solve target occlusion according to claim 1 is characterized in that: The step of calculating the predicted label according to the labeled standard training data set specifically includes: Using standard training data set I tr As the input of the recognition network model, the feature pyramid network module is used to extract multi-scale features; According to the multi-scale features output by the feature pyramid module, the cross-scale encoder is used to encode the cross-scale fusion features; The cross-scale fusion features encoded by the cross-scale encoder are used as input, the memory information corresponding to the previous frame read from the memory buffer is used as input, and the memory features corresponding to the current frame are updated using the bionic memory chain; According to the memory feature corresponding to the current frame obtained by the bionic memory chain, the memory feature is written into the memory buffer for memory update of the next frame; According to the memory features corresponding to the current frame obtained by the bionic memory chain, the memory decoder is used to decode and obtain the cross-scale temporal fusion features corresponding to the current frame; According to the cross-scale temporal fusion features corresponding to the current frame decoded by the memory decoder, the detection head is used to predict the bounding box and classify the instance of interest as the network prediction output.

4. The end-to-end video target detection method based on bionic memory chain to solve target occlusion according to claim 3 is characterized in that: The standard training data set I tr As the input of the recognition network model, the feature pyramid network module is used to extract multi-scale features, including: The standard training dataset I tr In the bottom-up HGNetV2-L network structure of the image input feature pyramid network, the convolution layer unit that does not change the size of the feature map in the network is defined as a level, that is, the level {P1, P2, P3, P4, P5}, and the convolution feature c outputted at the end of each level is extracted, that is, {c1, c2, c3, c4, c5}; {c3, c4, c5} is output as multi-scale features, and the corresponding scales are {1 / 2, 1 / 4, 1 / 8} of the original image size.

5. The end-to-end video target detection method based on bionic memory chain to solve target occlusion according to claim 3 is characterized in that: The method of encoding the multi-scale features output by the feature pyramid module using a cross-scale encoder to obtain a cross-scale fusion feature specifically includes: The cross-scale encoder uses self-attention to calculate the intra-scale interaction features {I1, I2, I3} on the three layers of features of different scales output by the feature pyramid network, where the number of self-attention heads used in each layer is 1; The cross-scale fusion features are calculated through three layers of continuous combination operations as output, where the convolution stride is 1 and the activation function is SiLU.

6. The end-to-end video target detection method based on bionic memory chain to solve target occlusion according to claim 3 is characterized in that: The method uses the cross-scale fusion feature obtained by encoding the cross-scale encoder as input, uses the memory information corresponding to the previous frame read by the memory buffer as input, and uses the bionic memory chain to update the memory feature corresponding to the current frame, specifically including: The bionic memory chain is composed of four parts: current frame features, previous frame features, strong impressions, and fragmented memories, which are connected in series at a channel ratio of 1:1:1:1, namely: BMC=[F,S,P,C] Among them, BMC stands for bionic memory chain, C stands for current frame features, P stands for previous frame features, S stands for strong impression, and F stands for fragmented memory; the previous frame features and the current frame features are the shallow part of the memory chain, which are used to build short-term memory; fragmented memory and strong impression are the deep part of the memory chain, which are used to build long-term memory; The corresponding current frame I output by the bionic memory chain t The memory characteristic is denoted as [BMC] t , the corresponding memory features of each part are recorded as F t , S t , P t With C t , t is the frame number; When the current frame number is t, that is, the current frame is I t According to the cross-scale fusion feature f output by the cross-scale encoder t , and the corresponding memory information of the previous frame read from the memory buffer [BMC] t-1 , computed through the defined attention, paraphrase, transfer, enhancement, and retrieval operations [BMC] t .

7. The end-to-end video target detection method for solving target occlusion based on bionic memory chain according to claim 6 is characterized in that: The following are the computational operations defined by attention, paraphrase, transfer, enhancement and retrieval [BMC] t , including: Corresponding memory information from the previous frame [BMC] t-1 Get F t-1 , S t-1 , P t-1 With C t-1 ; Use a convolution operator with a convolution kernel size of 1 and a stride of 0 for f t Convolution is performed to obtain C t ; Among them, f t To fuse features across scales; Use the convolution operator with a convolution kernel size of 1 and a stride of 0 for C t-1 Perform convolution to get P t ; C t-1 Input the cross attention module and t-1 Input the cross attention module, and C is calculated by the cross attention module t-1 Relative P t-1 , and with S t-1 Add together to get S t ; C t-1 Input linear mapping module to halve the number of channels and compress features, while F t-1 Another linear mapping module is input to halve the number of channels and compress the features. The outputs of the two linear mapping modules are connected in series to obtain F t ; By connecting F t , S t , P t With C t Get the corresponding current frame I output by the bionic memory chain t Memory characteristics of [BMC] t .

8. The end-to-end video target detection method for solving target occlusion based on bionic memory chain according to claim 3 is characterized in that: The memory feature corresponding to the current frame obtained according to the bionic memory chain is written into the memory buffer for memory update of the next frame, specifically including: The current frame corresponding memory feature obtained based on the bionic memory chain [BMC] t , memory buffer is decomposed by [BMC] t Get F t , S t , P t With C t , and store F one by one t , S t , P t With C t , which is used to update the memory in the next frame, that is, to update F t+1 , S t+1 , P t+1 With C t+1 .

9. The end-to-end video target detection method for solving target occlusion based on bionic memory chain according to claim 3 is characterized in that: The cross-scale temporal fusion features corresponding to the current frame obtained by decoding the memory decoder are used to predict the bounding box and classify the instance of interest using the detection head as the network prediction output, specifically including: The current frame corresponding memory feature obtained based on the bionic memory chain [BMC] t , memory decoder with [BMC] t As input, the cross-scale fusion feature f is output by the cross-scale encoder t As input, a multi-head cross attention with 8 heads is used to calculate the cross-scale temporal fusion features of the current frame; The detection head consists of a bounding box regression network and a category discrimination network, and the number of detection heads is 6; the bounding box regression network is a network composed of multiple linear mapping layers with an output dimension of 4; the category discrimination network is a network composed of a single linear mapping layer, and its output dimension is the total number of categories of the target of interest; the 6 detection heads contain a total of 6 bounding box regression networks and 6 category discrimination networks; The output of each bounding box regression network is the predicted bounding box coordinate value (χ2, y1, χ2, y2), where χ2 and y1 correspond to the horizontal and vertical coordinate values ​​of the vertex of the upper left corner of the bounding box, and χ3 and y3 correspond to the horizontal and vertical coordinate values ​​of the vertex of the lower right corner of the bounding box; The output of each category discrimination network is an n-dimensional vector, where n is the standard training data set I tr The total number of categories of the target of interest in the , the value of each dimension corresponds to the probability that the target belongs to each category, and the category with the largest probability is the category c to be determined; After the six detection task heads complete the instance detection, the final prediction result (χ1, y1, χ2, y2, c) of the algorithm model is obtained.

10. The end-to-end video target detection method for solving target occlusion based on bionic memory chain according to claim 1, characterized in that: The design loss function uses back propagation and gradient descent algorithms to train the end-to-end video target detection algorithm model based on bionic memory chain and occluded target instance supervision, specifically including: The predicted label (χ1, y1, X2, y2, c) is used as the network prediction output, and the sample label in the standard data set is used as the network expected output. The corresponding relationship between the 300 network prediction outputs and sample labels is constructed through the Hungarian matching algorithm. Take the training label gt as the expected output of the network and predict the label For the network prediction output, for the constructed network model, the target loss function between the expected output and the predicted output is calculated: based on the matching result of the Hungarian matching algorithm, the loss function is calculated between the network expected output and the network predicted output. The overall target loss function is composed of the bounding box regression loss and the classification loss. According to the target loss function, the model is iteratively trained using back propagation and gradient descent algorithms to minimize the target loss function and achieve the optimal network model and parameters.

Citation Information

Patent Citations

  • Small target detection method by combining super-resolution reconstruction

    CN110956126A

  • Video instance segmentation method and segmentation device based on space-time memory information

    CN114241388A

  • Weak light vehicle detection method based on multi-exposure generation fusion

    CN116129375A

  • Small sample image defect target detection method based on self-supervised pre-training

    CN116994047A

  • Target detection method and device, equipment and medium

    CN117058585A

Cited By

  • Audio processing method and apparatus, and XR device

    CN120388580A

  • Online action detection method based on multi-stage memory mapping

    CN120431632A

  • Online action detection method based on multi-stage memory mapping

    CN120431632B

  • Response capability detection method and system based on online action detection, terminal and storage medium

    CN121073888A