Image feature fusion and target detection method, electronic device, and readable storage medium

Abstract

The application discloses an image feature fusion and target detection method, an electronic device and a readable storage medium. The method comprises the following steps: selecting at least one from a current query vector, a current key vector and a current value vector of an acquired current feature map as a current target vector; performing time sequence fusion processing on the current target vector and a historical target vector corresponding to at least one historical feature map, to obtain a time sequence fusion feature; and performing attention feature fusion processing on the time sequence fusion feature and other vectors in the current feature map, to obtain a target fusion feature, wherein the other vectors are vectors in the current feature map except the current target vector. The above scheme can improve the accuracy of the target fusion feature.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image feature fusion method, a target detection method, an electronic device, and a computer-readable storage medium. Background Technology

[0002] With the rapid development of computer vision and deep learning technologies, feature fusion has become a widely used feature processing technique. Especially in the field of object detection, historical features can be fused in order to better learn temporal information and obtain more accurate object detection results.

[0003] Traditional temporal feature fusion typically involves concatenating the current and historical feature maps along the channel dimension, followed by convolutional processing to obtain a fused feature map. However, this method requires an excessive number of computational parameters and focuses on overly scattered information, making it difficult to capture useful details. Summary of the Invention

[0004] The main technical problem addressed by this application is to provide an image feature fusion method, a target detection method, an electronic device, and a computer-readable storage medium that can improve the efficiency and accuracy of image feature fusion.

[0005] To address the aforementioned technical problems, this application provides an image feature fusion method. This method is applied to a temporal feature fusion module and includes: selecting at least one from the current query vector, current key vector, and current value vector of the acquired current feature map as the current target vector; performing temporal fusion processing on the current target vector and historical target vectors corresponding to at least one historical feature map to obtain temporal fusion features; and performing attention feature fusion processing on the temporal fusion features and other vectors in the current feature map to obtain target fusion features, wherein the other vectors are vectors in the current feature map other than the current target vector.

[0006] To address the aforementioned technical problems, another technical solution adopted in this application is: providing a target detection method, the target detection method further comprising: inputting an acquired current image into a target detection network, the target detection network including a feature extraction module, a temporal feature fusion module, and a detection module, wherein the temporal feature fusion module is the aforementioned temporal feature fusion module; determining the current feature map of the current image through the feature extraction module; selecting at least one as the current target vector from the current query vector, current key vector, and current value vector of the acquired current feature map through the temporal feature fusion module; performing temporal fusion processing on the current target vector and historical target vectors corresponding to at least one historical feature map to obtain temporal fusion features; performing attention feature fusion processing on the temporal fusion features and other vectors in the current feature map to obtain target fusion features, wherein the other vectors are vectors in the current feature map other than the current target vector; and performing target detection on the target fusion features through the detection module to obtain a target detection result.

[0007] To solve the above-mentioned technical problems, another technical solution adopted in this application is: to provide an electronic device, including a memory and a processor, wherein the memory stores program instructions, and the processor retrieves the program instructions from the memory to execute the above-mentioned image feature fusion method and / or target detection method.

[0008] To solve the above-mentioned technical problems, another technical solution adopted in this application is: to provide a computer-readable storage medium including program data, which, when executed by a processor, is used to implement the above-mentioned image feature fusion method and / or target detection method.

[0009] The image feature fusion method of this application selects at least one of the current query vector, current key vector, and current value vector from the acquired current feature map as the current target vector; performs temporal fusion processing on the current target vector and the historical target vector corresponding to at least one historical feature map to obtain temporal fusion features; and performs attention feature fusion processing on the temporal fusion features and other vectors in the current feature map to obtain target fusion features, where the other vectors are vectors in the current feature map other than the current target vector. This scheme, by fusing the current feature map and historical feature maps through an attention mechanism, pays more attention to relevant and effective information in both historical and current feature maps, thereby improving the accuracy of the target fusion features. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly described below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort, wherein:

[0011] Figure 1 This is a schematic flowchart of an exemplary embodiment of the image feature fusion method shown in this application;

[0012] Figure 2 yes Figure 1 A flowchart illustrating an exemplary embodiment of step S120 in the image feature fusion method is shown.

[0013] Figure 3 yes Figure 1 A flowchart illustrating an exemplary embodiment of step S130 in the image feature fusion method is shown.

[0014] Figure 4 This is a schematic diagram of an exemplary embodiment of the first attention feature fusion submodule shown in this application;

[0015] Figure 5 This is a schematic flowchart illustrating a specific embodiment of the image feature fusion method shown in this application;

[0016] Figure 6 This is a schematic diagram of the framework of an exemplary embodiment of the target detection network shown in this application;

[0017] Figure 7 This is a schematic diagram of an exemplary embodiment of the image feature fusion apparatus shown in this application;

[0018] Figure 8 This is a schematic diagram of the structure of an embodiment of the electronic device provided in this application;

[0019] Figure 9 This is a schematic diagram of an embodiment of the computer-readable storage medium provided in this application. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It is understood that the specific embodiments described herein are only for explaining this application and not for limiting it. Furthermore, it should be noted that, for ease of description, only the parts related to this application are shown in the accompanying drawings, not all structures. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0021] Please see Figure 1 , Figure 1 This is a flowchart illustrating an exemplary embodiment of the image feature fusion method shown in this application.

[0022] The execution entity of the image feature fusion method can be a terminal device, a server, or other processing device. The terminal device can be a user equipment (UE), computer, mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. The execution entity of the image feature fusion method can also be a temporal feature fusion module or an image feature fusion device. In some possible implementations, the image feature fusion method can be implemented by a processor calling computer-readable instructions stored in memory.

[0023] Specifically, the image feature fusion method in this embodiment includes the following steps:

[0024] S110: Select at least one of the current query vector, current key vector, and current value vector from the obtained current feature map as the current target vector.

[0025] First, it should be noted that the image feature fusion method in this embodiment is applied to a temporal feature fusion module. The temporal feature fusion module is used to fuse the current feature map and historical feature maps to obtain the target fused features. Exemplarily, the temporal feature fusion module is highly adaptable and can be inserted into any model to complete the corresponding feature fusion task. The temporal feature fusion module can be based on the Transformer architecture.

[0026] The current feature map includes feature information obtained after feature extraction from the current image. In some embodiments, the temporal feature fusion module may receive the current feature map from the feature extraction module. In other embodiments, the temporal feature fusion module may also retrieve the current feature map from internal storage. In still other embodiments, the temporal feature fusion module may also receive user input or retrieve the current feature map from a feature set. The source of the current feature map is not specifically limited here and can be determined according to the actual application scenario.

[0027] After obtaining the current feature map, the temporal feature fusion module performs a linear transformation on the current feature map to obtain the current query vector, the current key vector, and the current value vector of the current feature map.

[0028] The current target vector is a vector selected from the vectors of the current feature map and fused with the historical feature map. For example, the current target vector can be one or more. For instance, the current target vector can be the current query vector, the current key vector, or the current value vector. Alternatively, the current target vector can be the current query vector and the current key vector; or the current query vector and the current value vector; or the current key vector and the current value vector; or the current query vector, the current key vector, and the current value vector.

[0029] S120: Perform temporal fusion processing on the current target vector and the historical target vector corresponding to at least one historical feature map to obtain temporal fusion features.

[0030] Historical feature maps refer to feature maps processed earlier than the current feature map. For example, a video stream can be acquired, containing at least one frame of image. Each frame is processed from earliest to latest capture time, and features from the currently processed image are extracted to obtain the current feature map. Features from images extracted earlier than the current time point are used to obtain historical feature maps. There can be one or more historical feature maps. For instance, feature maps from multiple historical time points can be used as the historical feature maps to be fused in this embodiment.

[0031] The historical target vector is the vector corresponding to the current target vector. For example, when the current target vector of the current feature map is the current query vector, the historical target vector of the historical feature map corresponds to the historical query vector; when the current target vector of the current feature map is the current key vector, the historical target vector of the historical feature map corresponds to the historical key vector; and when the current target vector of the current feature map is the current value vector, the historical target vector of the historical feature map corresponds to the historical value vector. After obtaining the historical feature map, the time-series feature fusion module can perform linear transformation processing on the historical feature map according to actual needs to obtain the current query vector and / or the current key vector and / or the current value vector of the historical feature map.

[0032] Temporal fusion features refer to the features obtained by fusing the current target vector and historical target vectors. The number of temporal fusion features corresponds to the number of current target vectors and can be one or more. When there are multiple current target vectors, the temporal feature fusion module performs temporal fusion processing on each current target vector and its corresponding historical target vector to obtain multiple temporal fusion features. As an example, when the current target vector includes the current query vector and the current key vector, the current query vector is temporally fused with historical query vectors to obtain the temporal fusion feature of the query vector, and the current key vector is temporally fused with historical key vectors to obtain the temporal fusion feature of the key vector.

[0033] S130: Perform attention feature fusion processing on the temporal fusion feature and other vectors in the current feature map to obtain the target fusion feature. The other vectors are the vectors in the current feature map other than the current target vector.

[0034] Other vectors are those vectors in the current feature map that were not selected as the current target vector. As an example, if the current target vector includes the current key vector, then the other vectors are the current query vector and the current value vector. If the current target vector includes both the current query vector and the current key vector, then the other vectors include the current value vector.

[0035] The target fusion feature is the feature obtained by fusing temporal fusion features with other vectors. For example, the temporal feature fusion module performs temporal fusion processing on the current target vector and historical target vectors to obtain temporal fusion features; then, it performs feature fusion processing on the temporal fusion features and other vectors in the current feature map to obtain the target fusion feature. As an example, if the current target vector includes the current key vector, the temporal feature fusion module performs temporal fusion processing on the current key vector and historical key vectors to obtain the temporal fusion feature of the key vector; then, it fuses the current query vector, the temporal fusion features of the key vector, and the current value vector to obtain the target fusion feature. As another example, if the current target vector includes the current query vector and the current key vector, the temporal feature fusion module performs temporal fusion processing on the current query vector and historical query vectors to obtain the temporal fusion feature of the query vector; it performs temporal fusion processing on the current key vector and historical key vectors to obtain the temporal fusion feature of the key vector; and it performs attention feature fusion processing on the temporal fusion features of the query vector, the temporal fusion features of the key vector, and the current value vector to obtain the target fusion feature. As another example, if the current target vector includes the current query vector, the current key vector, and the current value vector, the temporal feature fusion module performs temporal fusion processing on the current query vector and historical query vectors to obtain the temporal fusion feature of the query vector; it performs temporal fusion processing on the current key vector and historical key vectors to obtain the temporal fusion feature of the key vector; it performs temporal fusion processing on the current value vector and historical value vectors to obtain the temporal fusion feature of the value vector; and it performs attention feature fusion processing on the temporal fusion features of the query vector, the key vector, and the value vector to obtain the target fusion feature.

[0036] As can be seen, the image feature fusion method in this embodiment selects at least one of the current query vector, current key vector, and current value vector of the acquired current feature map as the current target vector; performs temporal fusion processing on the current target vector and the historical target vector corresponding to at least one historical feature map to obtain temporal fusion features; and performs attention feature fusion processing on the temporal fusion features and other vectors in the current feature map to obtain target fusion features, where the other vectors are vectors in the current feature map other than the current target vector. The above scheme, by fusing the current feature map and historical feature maps through an attention mechanism, pays more attention to relevant effective information in the historical feature maps and the current feature map, thereby improving the accuracy of the target fusion features.

[0037] Based on the above embodiments, the embodiments of this application adopt... Figure 2 The flowchart details how to fuse the current target vector and historical target vectors to obtain temporal fusion features. Please refer to [link / reference]. Figure 2 , Figure 2 yes Figure 1 The illustrated flowchart shows an exemplary embodiment of step S120 in the image feature fusion method. Specifically, step S120, which performs temporal fusion processing on the current target vector and the historical target vector corresponding to at least one historical feature map to obtain temporal fused features, specifically includes the following steps:

[0038] S210: Perform time-position encoding on the current target vector and the historical target vector respectively to obtain the time-position encoded current target vector and the time-position encoded historical target vector.

[0039] The purpose of temporal position encoding is to add a learnable code to the vector, enabling the target fusion features to better learn the temporal information between the current target vector and the historical target vector. For example, after obtaining the current target vector and the historical target vector, the temporal feature fusion module adds learnable temporal position codes to the current target vector and the historical target vector according to the time points corresponding to the current and historical target vectors, respectively, to obtain the temporally position-encoded current target vector and the temporally position-encoded historical target vector.

[0040] In some application scenarios, the time points corresponding to the current target vector and the historical target vector can be the acquisition times of the corresponding current and historical images. In other application scenarios, the time points corresponding to the current target vector and the historical target vector can also be the playback times of the corresponding current and historical images in the video stream. As an example, if the current target vector is the current key vector and the historical target vector is the historical key vector, then the historical target vectors corresponding to n historical feature maps can be used for temporal fusion processing with the current target vector. The n historical feature maps are the feature maps of the corresponding historical images at the n historical time points. Then, the temporally position-encoded current target vector and the temporally position-encoded historical target vector can be represented as:

[0041]

[0042] in, This represents the current target vector after encoding the time position corresponding to the current time point t. Indicates the current feature map, Represents the current target vector. The time location code represents the current point in time; This represents the historical target vector encoded with the time location corresponding to a historical time point. Represents historical features. Represents the historical target vector. This represents the time location code of a historical point in time. Among them, The feature maps at these time points are selected as historical feature maps to be fused with the current target vector in this embodiment. go through Operator conversion to historical target vector .

[0043] Furthermore, when determining the number of historical target vectors, the number can be determined based on the movement speed of the detected target, which is directly proportional to the number of historical target vectors. Specifically, the faster the detected target moves, the more historical target vectors there are, and the slower the detected target moves, the fewer historical target vectors there are. This allows for the establishment of a multi-scale historical frame fusion mechanism to utilize detected targets with different movement speeds, such as high-speed vehicles and pedestrians, thereby enhancing the robustness of feature matching and reducing misfusion caused by single-frame anomalies.

[0044] S220: The current target vector encoded by time position and the historical target vector encoded by time position are concatenated to obtain the first concatenated feature.

[0045] After obtaining the time-position encoded current target vector and the time-position encoded historical target vector, the temporal feature fusion module directly concatenates the two vectors to obtain the first concatenated feature. In some embodiments, the two vectors can be concatenated along the channel dimension to obtain the first concatenated feature. For example, assuming the dimension of the current target vector is... The dimension of each historical target vector is If the number of historical target vectors can be n, then the dimension of the current target vector (after time-position encoding) and the historical target vector (after time-position encoding) concatenated along the channel dimension is: In other embodiments, the temporal feature fusion module can further flatten the current target vector and the historical target vector after time position encoding, respectively, into one-dimensional vectors to obtain the flattened current target vector and the flattened historical target vector; and concatenate the flattened current target vector and the flattened historical target vector to obtain the first concatenated feature.

[0046] S230: Perform one-dimensional convolution on the first concatenated feature to obtain the temporal fusion feature.

[0047] After obtaining the first concatenated feature, to improve the fusion effect, the temporal feature fusion module can also perform one-dimensional convolution on the first concatenated feature to obtain the temporal fusion feature. For example, the Conv1d operator can be used to perform fusion processing on the first concatenated feature to obtain the temporal fusion feature. Specifically, during the one-dimensional convolution process, the first concatenated feature can be dimensionality reduced to facilitate the subsequent fusion of the temporal fusion feature and other vectors. For example, the dimension of... The first concatenation feature processing is of dimension 1. The temporal fusion features are shown below as an example.

[0048]

[0049] in, Represents temporal fusion features, This indicates splicing / joining. This represents one-dimensional convolution processing. This represents the current target vector at time point t. Indicates a point in time The historical target vector, Indicates a point in time The historical target vector, Indicates a point in time The historical target vector.

[0050] As can be seen, the image feature fusion method in this embodiment performs temporal and positional encoding on the current target vector and the historical target vector respectively, obtaining the temporally and positionally encoded current target vector and the temporally and positionally encoded historical target vector; the temporally and positionally encoded current target vector and the temporally and positionally encoded historical target vector are concatenated to obtain the first concatenated feature; the first concatenated feature is subjected to one-dimensional convolution processing to obtain the temporal fusion feature. Thus, by adding temporal and positional encoding and fusing temporal information, the subsequent utilization of the target fusion feature can better perceive time.

[0051] Based on the above embodiments, the embodiments of this application adopt... Figure 3 The flowchart details how to fuse temporal fusion features and other vectors in the current feature map besides the current target vector to obtain the target fusion features. Please refer to [link / reference]. Figure 3 , Figure 3 yes Figure 1 The illustrated flowchart shows an exemplary embodiment of step S130 in the image feature fusion method. Specifically, step S130, which involves performing attention feature fusion processing on the temporal fusion features and other vectors in the current feature map to obtain the target fusion features, includes the following steps:

[0052] S310: In response to the current target vector including the current key vector, the temporal fusion feature and the current query vector are subjected to the first attention feature fusion process to obtain the initial attention fusion feature.

[0053] This embodiment takes the current target vector including the current key vector as an example. Of course, in other embodiments, the temporal fusion feature can also switch the current target vector to the current query vector or the current value vector; or the current target vector includes any two of the current query vector, the current key vector, and the current value vector, or the current target vector includes the current query vector, the current key vector, and the current value vector.

[0054] It should be noted that when fusing temporal fusion features and other vectors, the other vectors can also be vectors encoded with time position.

[0055] When the current target vector includes the current key vector, the temporal fusion feature obtained by fusing the current target vector with the historical target vector is fused with the current query vector to perform the first attention feature fusion, resulting in the initial attention fusion feature. In some embodiments, the temporal feature fusion module can obtain the product between the current query vector and the temporal fusion feature as the initial attention fusion feature.

[0056] In other embodiments, the temporal feature fusion module can also sample at least one feature point and its corresponding feature point weight from the temporal fusion features based on the current query vector; and determine the initial attention fusion features based on the current query vector, each feature point, and its corresponding feature point weight. This reduces the computational load through sampling.

[0057] Specifically, the temporal feature fusion module can predict sampling coordinates based on the current query vector, and sample at least one feature point from the temporal fusion features according to the sampling coordinates. For example, the current query vector can first be processed by a two-dimensional convolution to obtain a two-dimensional convolutional current query vector; then, the two-dimensional convolutional current query vector can be normalized to obtain the sampling coordinates. For example, the sampling coordinates p = Norm(Conv2d(Q(t))), where Conv2d represents two-dimensional convolution, Norm represents normalization, and Q(t) represents the current query vector. After obtaining the predicted sampling coordinates, the temporal feature fusion module can obtain the coordinates of m feature points sampled on the temporal fusion feature by mapping bilinear sampling to 0-H and 0-W and rounding. .

[0058] The temporal feature fusion module can also predict the feature point weights corresponding to each feature point based on the current query vector. For example, the current query vector can first be processed by a two-dimensional convolution to obtain a two-dimensional convolutional current query vector; then, the two-dimensional convolutional current query vector can be normalized to obtain the feature point weights of the temporal fusion features. For example, the feature point weights... Conv2d represents 2D convolution, Norm represents normalization, and Q(t) represents the current query vector. It should be noted that the convolution kernels used to calculate the sampling coordinates and feature point weights are different; the convolution kernel for sampling coordinates can be twice the size of the kernel for feature point weights.

[0059] In some embodiments, the temporal feature fusion module can also predict the feature points and feature point weights of the temporal fusion features based on the current query vector and the current key vector. In other embodiments, the temporal feature fusion module can also predict the feature points and feature point weights of the temporal fusion features based on the current value vector.

[0060] After obtaining the feature points and corresponding feature point weights of the temporal fusion features, the temporal feature fusion module fuses the current query vector, each feature point, and their corresponding feature point weights to obtain the initial attention fusion features. Specifically, it obtains the product between the current query vector and each feature point, and the corresponding feature point weights; it sums the products corresponding to each feature point to obtain the sum of the feature points; and it processes the sum of the feature points using an activation function to obtain the initial attention fusion features.

[0061] Weighted processing is performed using the feature point weights corresponding to each feature point to increase the weight of important feature points and decrease the weight of unimportant feature points, thus focusing more on the features of interest in the temporal fusion feature. For example, the calculation of the initial attention fusion feature satisfies the following equation:

[0062]

[0063] in, Indicates the initial attention fusion features, This represents the activation function. This represents the feature point weight of the i-th feature point. Indicates the current query vector. This represents the coordinates of the i-th feature point of the temporal fusion feature.

[0064] Please refer to Figure 4 , Figure 4 This is a schematic diagram of an exemplary embodiment of the first attention feature fusion submodule shown in this application. First, parameter prediction is performed using the current query vector to obtain the sampled coordinates of feature points in the temporal fusion features. and feature point weights Then, the attention distribution calculation formula was used. The initial attention fusion feature is obtained by fusing the feature points and feature point weights in the current query vector and the temporal fusion feature.

[0065] S320: Perform a second attention feature fusion process on the initial attention fusion features and the current value vector to obtain the target fusion features.

[0066] After obtaining the initial attention fusion features of the temporal fusion features and the current query vector, the temporal feature fusion module further performs a second attention feature fusion on the initial attention fusion features and the current value vector to obtain the final target fusion features. In some embodiments, a conventional attention distribution calculation formula can be used. The initial attention fusion feature and the current value vector are fused to obtain the target fusion feature, where Q is the current query vector, K is the temporal fusion feature, and V is the current value vector.

[0067] In other embodiments, the product of the initial attention fusion feature and the current value vector can be used as the attention fusion feature; a first fusion weight and a second fusion weight of the current feature map are determined based on the current feature map and the attention fusion feature; the attention fusion feature and the current feature map are fused using the first fusion weight and the second fusion weight of the current feature map to obtain the target fusion feature. Introducing fusion weights to control the fusion intensity can reduce the impact of historical noise on the current feature map and improve the effectiveness of the target fusion feature.

[0068] The first fusion weight is used to weight the attention fusion features, and the second fusion weight is used to weight the current feature map. The fusion weights control the fusion strength between the attention fusion features and the current feature map. In some embodiments, the attention fusion features and the current feature map can be normalized separately to obtain the first and second fusion weights. In other embodiments, the attention fusion features and the current feature map can be mapped separately to obtain the first and second fusion weights.

[0069] In other embodiments, the current feature map and the attention fusion feature can be concatenated to obtain a second concatenated feature; the second concatenated feature can be subjected to one-dimensional convolution to obtain a second concatenated feature after one-dimensional convolution; the second concatenated feature after one-dimensional convolution can be processed using an activation function to obtain a first fusion weight of the attention fusion feature; the difference between a preset value and the first fusion weight is determined as the second fusion weight of the current feature map. For example, the calculation of the first fusion weight satisfies the following formula:

[0070]

[0071] in, Indicates the first fusion weight. Indicates the initial attention fusion features, Represents the current value vector. Indicates the current feature map, Represents one-dimensional convolution. This represents the activation function.

[0072] After obtaining the first fusion weight of the attention fusion feature, the difference between the preset value and the first fusion weight can be determined as the second fusion weight of the current feature map. The preset value can be 1.

[0073] Then, the attention fusion features and the current feature map are fused according to the first fusion weight and the second fusion weight. Specifically, the attention fusion features are weighted according to the first fusion weight to obtain the weighted attention fusion features; the current feature map is weighted according to the second fusion weight to obtain the weighted current feature map; the sum of the weighted attention fusion features and the weighted current feature map is calculated to obtain the target fusion feature. For example, the calculation of the target fusion feature satisfies the following formula:

[0074]

[0075] in, Indicates target fusion features, Indicates the first fusion weight. Indicates the initial attention fusion features, Represents the current value vector. Indicates attention fusion features, Indicates the second fusion weight. This represents the current feature map.

[0076] As can be seen, the image feature fusion method in this embodiment responds to the current target vector including the current key vector by performing a first attention feature fusion process on the temporal fusion features and the current query vector to obtain the initial attention fusion features; then, a second attention feature fusion process is performed on the initial attention fusion features and the current value vector to obtain the target fusion features. This fully integrates the current feature map and the historical feature map, making the temporal information of the target fusion features richer. Furthermore, it can integrate the key-value states of the historical feature map to store historically useful information, requiring only the storage and updating of a small number of features, reducing the occupation of online storage space and improving computational efficiency.

[0077] As another example, when the current target vector includes the current query vector and the current key vector, the temporal fusion features include the temporal fusion features of the query vector and the temporal fusion features of the key vector; the temporal fusion features of the query vector and the temporal fusion features of the key vector are subjected to a first attention feature fusion process to obtain the initial attention fusion features; the initial attention fusion features and the current value vector are subjected to a second attention feature fusion process to obtain the target fusion features. Specifically, based on the temporal fusion features of the query vector, at least one feature point and its corresponding feature point weight are sampled from the temporal fusion features of the key vector; the products between the temporal fusion features of the query vector and each feature point and its corresponding feature point weight are obtained respectively; the products corresponding to each feature point are summed to obtain the sum of feature points; the sum of feature points is processed using an activation function to obtain the initial attention fusion features; the product of the initial attention fusion features and the current value vector is used as the attention fusion features; the first fusion weight of the attention fusion features and the second fusion weight of the current feature map are determined according to the current feature map and the attention fusion features; the attention fusion features and the current feature map are fused using the first fusion weight of the attention fusion features and the second fusion weight of the current feature map to obtain the target fusion features.

[0078] As another example, when the current target vector includes the current query vector, the current key vector, and the current value vector, the temporal fusion features include the temporal fusion features of the query vector, the temporal fusion features of the key vector, and the temporal fusion features of the value vector. The temporal fusion features of the query vector and the temporal fusion features of the key vector are subjected to a first attention feature fusion process to obtain the initial attention fusion features. The initial attention fusion features and the temporal fusion features of the value vector are subjected to a second attention feature fusion process to obtain the target fusion features. Specifically, based on the temporal fusion features of the query vector, at least one feature point and its corresponding feature point weight are sampled from the temporal fusion features of the key vector; the products between the temporal fusion features of the query vector and each feature point and its corresponding feature point weight are obtained respectively; the products corresponding to each feature point are summed to obtain the sum of feature points; the sum of feature points is processed using an activation function to obtain the initial attention fusion features; the product of the initial attention fusion features and the temporal fusion features of the value vector is used as the attention fusion features; the first fusion weight of the attention fusion features and the second fusion weight of the current feature map are determined according to the current feature map and the attention fusion features; the attention fusion features and the current feature map are fused using the first fusion weight of the attention fusion features and the second fusion weight of the current feature map to obtain the target fusion features.

[0079] The current target vector can also be any combination of the current query vector, the current key vector, and the current value vector. The specific fusion methods can be referred to the above, and will not be listed one by one here.

[0080] To illustrate in detail the image feature fusion method in the embodiments of this application, Figure 5 The flowchart shown below provides further explanation, as detailed below:

[0081] First, it should be noted that this embodiment uses the current target vector as the current key vector and the historical target vector as the historical key vector for illustration.

[0082] Get the current feature map and n historical feature maps The current feature map is processed by the operator respectively. , and This transforms the current query vector, current key vector, and current value vector. For example, the Conv2d operator can be used for vector transformation. The dimension of the current feature map can be... The dimensions of the transformed current query vector, current key vector, and current value vector can be [dimensions]. The n historical feature maps are processed by the operator. Converted into n historical key vectors, the dimension of the historical feature map can be... The dimension of the historical key vector can be .

[0083] To enhance the model's time-sensitivity, time-position encoding can be applied to the current query vector, current key vector, current value vector, and historical key vector, respectively. A learnable time-position encoding is added based on the time corresponding to the vector. As shown in the following formula:

[0084]

[0085] in, This represents the current query vector after time and location encoding. Indicates the current feature map, This represents the current key vector after time-location encoding. This represents the historical key vector encoded by time position. Represents historical features. This represents the current value vector after time and location encoding.

[0086] Then, the timing state fusion submodule is used to... and The first concatenated feature is obtained by performing concatenation along the channel dimension, with dimension [missing information]. Perform a one-dimensional convolution on the first concatenated feature to obtain the temporal fusion feature, which can be represented as follows: .

[0087] After obtaining the temporal fusion features The first attention feature fusion submodule is used to fuse temporal fusion features. and For example, a deformable attention mechanism can be used for fusion, sampling only m key feature points (m≤min{H, W}) from the temporal fusion features for the first attention feature fusion. Specifically, based on Perform parameter prediction to obtain the sampling coordinates and weights of feature points; The feature points and their corresponding weights are fused to obtain the initial attention fusion feature, which can be represented as: .

[0088] Finally, a fusion weight can be introduced to reduce the impact of historical noise. The initial attention fusion features and the current value vector are fused using a second attention feature fusion submodule to obtain the target fusion features. Specifically, the fusion weight is... and The product of the two features is used as the attention fusion feature; the attention fusion feature and the current feature map are concatenated to obtain the second concatenated feature; the second concatenated feature is subjected to one-dimensional convolution to obtain the second concatenated feature after one-dimensional convolution; the second concatenated feature is processed using an activation function to obtain the first fusion weight; the difference between the preset value and the first fusion weight is determined as the second fusion weight; the attention fusion feature is weighted according to the first fusion weight to obtain the weighted attention fusion feature; the current feature map is weighted according to the second fusion weight to obtain the weighted current feature map; the sum of the weighted attention fusion feature and the weighted current feature map is determined as the target fusion feature.

[0089] Furthermore, in some application scenarios, the temporal feature fusion module can be applied to object detection. Specifically, the acquired current image is input into an object detection network, which includes a feature extraction module, a temporal feature fusion module, and a detection module. The temporal feature fusion module is the one described above. The feature extraction module determines the current feature map of the current image. The temporal feature fusion module selects at least one of the current query vector, current key vector, and current value vector from the acquired current feature map as the current target vector. The current target vector is temporally fused with the historical target vectors corresponding to at least one historical feature map to obtain the temporal fused feature. The temporal fused feature is then fused with other vectors in the current feature map using attention features to obtain the target fused feature. The other vectors are the vectors in the current feature map other than the current target vector. The detection module performs object detection on the target fused feature to obtain the object detection result.

[0090] In the field of object detection, traditional object detection network structures include a feature extraction module and a detection module. The feature extraction module may include a backbone network, or it may include a backbone network and a neck structure. For example, the current image is sequentially fed into the backbone network, the neck structure, and the detection module for processing to obtain the final object detection result. In this case, the quality of a single frame image directly affects the object detection performance. When the target is moving, there may be temporary blurring or blurring of the target or background in a single frame image, thereby reducing the accuracy of object detection.

[0091] Based on this, this embodiment inserts a temporal feature fusion module between the feature extraction module and the detection module to enhance the performance of the target detection network in video target scenes. As an example, please refer to... Figure 6 , Figure 6 This is a schematic diagram of an exemplary embodiment of the target detection network shown in this application. When the feature extraction module includes a backbone network and a neck structure, multi-scale feature extraction is performed on the current image through the backbone network to obtain multi-scale image features; deep semantic feature extraction is performed on the multi-scale image features through the neck structure to obtain the current feature map; the current feature map is input to the temporal feature fusion module for feature fusion processing with historical feature maps to obtain target fusion features, and the output target fusion features have the same dimension as the input current feature map; the target fusion features are input to the detection module for target detection to obtain the target detection result. The target detection result includes, but is not limited to, detection box coordinates, target category, and confidence score.

[0092] Understandably, the transformer-based temporal feature fusion module is highly adaptable and applicable to most target detection networks based on the backbone-head architecture. The output features of this temporal feature fusion module maintain the same dimensionality as the input features, so this module can be used as a plug-in module to process the input features of the detection head.

[0093] Please see Figure 7 , Figure 7This is a schematic diagram of an exemplary embodiment of the image feature fusion apparatus shown in this application. The image feature fusion apparatus 700 includes a selection module 710, a first fusion module 720, and a second fusion module 730. The selection module 710 is used to select at least one as the current target vector from the current query vector, the current key vector, and the current value vector of the acquired current feature map. The first fusion module 720 is used to perform temporal fusion processing on the current target vector and the historical target vector corresponding to at least one historical feature map to obtain temporal fusion features. The second fusion module 730 is used to perform attention feature fusion processing on the temporal fusion features and other vectors in the current feature map to obtain target fusion features, wherein the other vectors are vectors in the current feature map other than the current target vector.

[0094] In the above scheme, the image feature fusion device selects at least one from the current query vector, current key vector, and current value vector of the acquired current feature map as the current target vector; performs temporal fusion processing on the current target vector and the historical target vector corresponding to at least one historical feature map to obtain temporal fusion features; and performs attention feature fusion processing on the temporal fusion features and other vectors in the current feature map to obtain target fusion features, where the other vectors are vectors in the current feature map other than the current target vector. This scheme, by fusing the current feature map and historical feature maps through an attention mechanism, pays more attention to relevant and effective information in both historical and current feature maps, thereby improving the accuracy of the target fusion features.

[0095] The functions of each module can be found in the implementation examples of the image feature fusion method, and will not be repeated here.

[0096] To implement the image feature fusion method and / or target detection method of the above embodiments, this application proposes another electronic device, please refer to [link to details]. Figure 8 , Figure 8 This is a schematic diagram of the structure of an embodiment of the electronic device provided in this application.

[0097] Electronic device 800 includes memory 810 and processor 820, wherein memory 810 and processor 820 are coupled together.

[0098] The memory 810 is used to store program data, and the processor 820 is used to execute the program data to implement the image feature fusion method and / or target detection method of the above embodiments.

[0099] In this embodiment, processor 820 can also be referred to as a CPU (Central Processing Unit). Processor 820 may be an integrated circuit chip with signal processing capabilities. Processor 820 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor, or processor 820 can be any conventional processor.

[0100] This application also provides a computer-readable storage medium, such as Figure 9 As shown, the computer-readable storage medium 900 is used to store program data 910, which, when executed by a processor, is used to implement the image feature fusion method and / or target detection method as described in the method embodiments of this application.

[0101] The methods involved in the image feature fusion method and / or target detection method embodiments of this application, when implemented as software functional units and sold or used as independent products, can be stored in a device, such as a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0102] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A target detection method, characterized in that, The target detection method includes: The acquired current image is input into a target detection network, which includes a feature extraction module, a temporal feature fusion module, and a detection module. The current feature map of the current image is determined by the feature extraction module; The temporal feature fusion module selects at least one of the current query vector, current key vector, and current value vector of the current feature map as the current target vector; the current target vector is temporally fused with at least one historical target vector corresponding to a historical feature map to obtain temporal fusion features, wherein the number of historical target vectors is proportional to the movement speed of the detected target; the temporal fusion features are then fused with other vectors in the current feature map using attention features to obtain target fusion features, wherein the other vectors are vectors in the current feature map other than the current target vector; The target detection result is obtained by performing target detection on the target fusion features through the detection module. The step of performing attention feature fusion processing on the temporal fusion features and other vectors in the current feature map to obtain the target fusion features includes: In response to the current target vector including the current key vector, the temporal fusion feature and the current query vector are subjected to the first attention feature fusion processing to obtain the initial attention fusion feature; The initial attention fusion features and the current value vector are subjected to a second attention feature fusion process to obtain the target fusion features.

2. The target detection method according to claim 1, characterized in that, The step of performing temporal fusion processing on the current target vector and the historical target vector corresponding to at least one historical feature map to obtain temporal fusion features includes: The current target vector and the historical target vector are respectively time-position encoded to obtain the time-position encoded current target vector and the time-position encoded historical target vector; The current target vector encoded by time position and the historical target vector encoded by time position are concatenated to obtain the first concatenation feature; The first spliced ​​feature is subjected to one-dimensional convolution to obtain the temporal fusion feature.

3. The target detection method according to claim 1, characterized in that, The step of performing a first attention feature fusion process on the temporal fusion features and the current query vector to obtain initial attention fusion features includes: Based on the current query vector, at least one feature point and the corresponding feature point weight are sampled from the temporal fusion features; The initial attention fusion feature is determined based on the current query vector, each feature point, and the corresponding feature point weights.

4. The target detection method according to claim 3, characterized in that, The step of determining the initial attention fusion feature based on the current query vector, each feature point, and the corresponding feature point weights includes: Obtain the product between the current query vector and each feature point, and the feature point weights corresponding to each feature point; The sum of the products corresponding to each feature point is obtained by summing the products of the feature points. The sum of the feature points is processed using an activation function to obtain the initial attention fusion feature.

5. The target detection method according to claim 1, characterized in that, The step of performing a second attention feature fusion process on the initial attention fusion features and the current value vector to obtain the target fusion features includes: The product of the initial attention fusion feature and the current value vector is used as the attention fusion feature; The first fusion weight of the attention fusion feature and the second fusion weight of the current feature map are determined based on the current feature map and the attention fusion feature; The attention fusion feature and the current feature map are fused using the first fusion weight of the attention fusion feature and the second fusion weight of the current feature map to obtain the target fusion feature.

6. The target detection method according to claim 5, characterized in that, The step of determining the first fusion weight of the attention fusion feature and the second fusion weight of the current feature map based on the current feature map and the attention fusion feature includes: The current feature map and the attention fusion feature are concatenated to obtain a second concatenated feature; The second spliced ​​feature is subjected to one-dimensional convolution processing to obtain the second spliced ​​feature after one-dimensional convolution processing; The second concatenated feature after the one-dimensional convolution is processed using an activation function to obtain the first fusion weight of the attention fusion feature; The difference between the preset value and the first fusion weight is determined as the second fusion weight of the current feature map.

7. An electronic device, characterized in that, include: A memory and a processor, wherein the memory stores program instructions, and the processor retrieves the program instructions from the memory to execute the target detection method as described in any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, include: The system stores program data, which, when executed by a processor, is used to implement the target detection method as described in any one of claims 1-6.

Images