Night target tracking method, system and device based on multidimensional space attention in intelligent driving scene and computer storage medium

Abstract

The invention discloses a nighttime target tracking method, system and device based on multidimensional space attention in an intelligent driving scene and a computer storage medium, and the method comprises the steps: carrying out the coding of current image data obtained through a camera through a four-layer down-sampling structure, and inputting the image features to generate a feature map; a spatial channel attention module based on Transform is used to establish a dependency relationship between pixels of the feature map; repeatedly extracting image features of the sampling stages from the first stage to the third stage through a multi-dimensional space attention module, and generating a global information feature map; splicing the global information feature map and the feature map in the up-sampling stage, and synchronously carrying out up-sampling to generate a brightened feature map; and outputting the bright feature image after synchronous up-sampling to a downstream tracker, and generating a final night target detection and tracking result. According to the invention, the problems of low target tracking accuracy and poor robustness at night or under a low illumination condition in the prior art can be solved.

Description

Technical Field

[0001] The present invention relates to the field of target tracking technology, and in particular to a nighttime target tracking method, system, device and computer storage medium based on multi-dimensional spatial attention in an intelligent driving scenario. Background Art

[0002] With the rapid development of artificial intelligence and computer vision technology, target tracking technology has been widely used in the fields of autonomous driving, security monitoring, drone navigation, etc. However, traditional target tracking methods perform poorly at night or in low light conditions. The main reason is that the contrast between the target and the background is low and the noise is large, which leads to a significant decrease in the accuracy of target detection and tracking. Existing night target tracking methods usually rely on the following technologies: infrared imaging technology, enhanced imaging technology, and deep learning technology. Although the above technologies have solved some problems of night target tracking to a certain extent, there are still many shortcomings.

[0003] In patent CN118334084A, target detection and tracking are achieved by detecting the thermal radiation of objects. Although infrared imaging can work under low light conditions, its resolution is low and is easily affected by changes in ambient temperature. In the published patent CN117974716A, a nighttime drone target tracking method integrating a hybrid attention mechanism is used. This method uses a night vision image enhancement model to enhance each night image frame in the night vision video sequence of the drone, and then performs image target tracking and recognition to improve the target tracking accuracy for night images. Although this method uses an image enhancement algorithm to improve image quality and improve target visibility, this method requires a complex preprocessing process and has limited effect in a strong noise environment. In recent years, deep learning methods based on convolutional neural networks (CNNs) and recurrent neural networks (RNNs) have made significant progress in the field of target tracking, but in nighttime environments, these methods still face challenges such as insufficient data and difficult training, and deep learning methods require a large amount of labeled data and computing resources. Summary of the invention

[0004] The purpose of the present invention is to provide a night target tracking method, system, device and computer storage medium based on multi-dimensional spatial attention in an intelligent driving scenario, which can better capture the difference between the target and the background and improve the robustness and accuracy of night target tracking.

[0005] To achieve the above object, the technical solution adopted by the present invention is:

[0006] A nighttime target tracking method based on multi-dimensional spatial attention in an intelligent driving scenario comprises the following steps:

[0007] The current image data obtained by the camera is encoded into the input image features through a four-layer downsampling structure to generate a feature map;

[0008] A Transformer-based spatial channel attention module is used to establish the dependency relationship between pixels of the feature map. When establishing the dependency relationship between pixels, the feature map obtained after downsampling in the fourth stage is input into the Transformer's spatial and channel attention, and evenly divided into non-overlapping windows of fixed size for self-attention calculation;

[0009] The multi-dimensional spatial attention module repeatedly extracts image features from the first stage to the third stage of downsampling to obtain the global information of the image and generate a global information feature map. The model understands the importance of each area in the image.

[0010] The global information feature map is concatenated with the feature map in the upsampling stage, and upsampling is performed synchronously to generate a brightening feature map;

[0011] The brightened feature image after synchronous upsampling is output to a downstream tracker to generate a final nighttime target detection and tracking result.

[0012] Furthermore, when inputting image features, multiple downsampling encoders are used to generate feature maps. Each encoder has two 3×3 convolutional layers and downsampling layers. After the first stage of downsampling, the feature map is obtained, and after each stage of downsampling encoder, the feature map channel is doubled.

[0013] Furthermore, when acquiring the global information of the image, the simple skip links in the U-Net structure are replaced by a four-branch multi-dimensional spatial attention mechanism, and the contextual information in the shallow features is used to enhance the image details.

[0014] Furthermore, when splicing feature images, after the Transformer module, the feature images of the first four downsampling stages are spliced ​​with the upsampling decoders of the corresponding four stages. The Tanh activation function of the last layer outputs an illumination curve mapping and a noise curve mapping, and the enhanced image is obtained through robust curve projection.

[0015] A nighttime target tracking system based on multi-dimensional spatial attention in an intelligent driving scenario, comprising:

[0016] A first generation module is used to encode the input image features of the current image data obtained by the camera through a four-layer downsampling structure to generate a feature map;

[0017] A processing module is used to establish the dependency relationship between the pixels of the feature map by using the Transformer-based spatial channel attention module. When establishing the dependency relationship between the pixels, the feature map obtained after downsampling in the fourth stage is input into the Transformer's spatial and channel attention, and is evenly divided into non-overlapping windows of a fixed size for self-attention calculation.

[0018] The second generation module is used to repeatedly extract the image features from the first stage to the third stage of downsampling through the multi-dimensional spatial attention module to obtain the global information of the image to generate a global information feature map. The model understands the importance of each area in the image;

[0019] A third generation module is used to splice the global information feature map with the feature map in the upsampling stage, and perform upsampling synchronously to generate a brightening feature map;

[0020] The output module is used to output the brightened feature image after synchronous upsampling to the downstream tracker to generate the final night target detection and tracking results.

[0021] Furthermore, when inputting image features, the first generation module uses multiple downsampling encoders to generate feature maps, each encoder has two 3×3 convolutional layers and downsampling layers. After the first stage of downsampling, the feature map is obtained, and after each stage of downsampling encoder, the feature map channel is doubled.

[0022] Furthermore, when acquiring the global information of the image, the second generation module replaces the simple jump links in the U-Net structure with a four-branch multi-dimensional spatial attention mechanism, and utilizes the contextual information in the shallow features to enhance the image details.

[0023] Furthermore, when the third generation module splices the feature images, after the Transformer module, the feature images of the first four downsampling stages are spliced ​​with the upsampling decoders of the corresponding four stages, and an illumination curve mapping and a noise curve mapping are output through the Tanh activation function of the last layer, and an enhanced image is obtained through robust curve projection.

[0024] An electronic device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the nighttime target tracking method based on multi-dimensional spatial attention in the intelligent driving scenario as described above.

[0025] A computer-readable storage medium stores a computer program, which is executed by a processor to implement the nighttime target tracking method based on multi-dimensional spatial attention in the intelligent driving scenario as described above.

[0026] Beneficial effects:

[0027] The present invention can solve the problems of low accuracy and poor robustness of target tracking in the prior art at night or under low light conditions. The present invention introduces a multi-dimensional spatial attention mechanism and combines a deep learning algorithm to effectively detect and track targets in a night environment based on the SCT model, while not affecting the real-time performance of the tracker. It can also effectively improve the tracking accuracy and success rate of mainstream trackers at night. The method can adaptively focus on the target area and improve the visibility of the target in a complex background, thereby significantly improving the performance and accuracy of night target tracking. The present invention can serve as the basis for advanced driver assistance functions of automobiles to achieve a nighttime driving assistance system with better performance and safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The specific implementation modes of the present invention are further described in detail below in conjunction with the accompanying drawings.

[0029] Figure 1 It is a flowchart of a night target tracking method based on multi-dimensional spatial attention in an intelligent driving scenario according to an embodiment of the present application;

[0030] Figure 2 is another flow chart of a nighttime target tracking method based on multi-dimensional spatial attention in an intelligent driving scenario provided according to an embodiment of the present application;

[0031] Figure 3 is a schematic diagram of the structure of a multidimensional spatial attention model provided according to an embodiment of the present application;

[0032] Figure 4 This is a comparison chart of the success rate and accuracy of a low-light enhancer provided according to an embodiment of the present application and a mainstream low-light enhancer when tracking at night using the online tracking algorithm SiamRPN++ on the UAVDark135 test dataset;

[0033] Figure 5 This is a visualization effect diagram of a low-light enhancer on the DarkTrack2021 dataset provided according to an embodiment of the present application;

[0034] Figure 6 This is an example diagram of a nighttime target tracking system based on multi-dimensional spatial attention in an intelligent driving scenario according to an embodiment of the present application.

[0035] Figure 7 It is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0036] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0037] Example

[0038] A nighttime target tracking method based on multi-dimensional spatial attention in intelligent driving scenarios, such as Figure 1 and Figure 2 As shown, the following steps are included:

[0039] In step S101, the current image data obtained by the camera is encoded into the input image features through a four-layer downsampling structure to generate a feature map;

[0040] Preferably, when inputting image features, multiple downsampling encoders are used to generate feature maps, each encoder has two 3×3 convolutional layers and a downsampling layer. After the first stage of downsampling, a feature map is obtained, and after each stage of downsampling encoder, the feature map channel is doubled.

[0041] In step S102, a Transformer-based spatial channel attention module is used to establish a dependency relationship between pixels of the feature map. When establishing the dependency relationship of each pixel, the feature map obtained after downsampling in the fourth stage is input into the Transformer's spatial and channel attention, and evenly divided into non-overlapping windows of fixed size for self-attention calculation;

[0042] In step S103, the image features of the first stage to the third stage downsampling stage are repeatedly extracted through the multi-dimensional spatial attention module to obtain the global information of the image to generate a global information feature map. The model understands the importance of each area in the image. When obtaining the global information of the image, a four-branch multi-dimensional spatial attention mechanism is designed to replace the simple jump link in the U-Net structure. Without increasing the model calculation degree too much, the contextual information in the shallow features is effectively used to strengthen the image details and improve the enhancement effect.

[0043] In step S104, the global information feature map is spliced ​​with the feature map of the upsampling stage, and upsampling is performed synchronously to generate a brightened feature map. When splicing the feature images, after the Transformer module, the feature images of the first four downsampling stages are spliced ​​with the upsampling decoders of the corresponding four stages. After the last layer of the Tanh activation function, an illumination curve mapping and a noise curve mapping are output, and an enhanced image is obtained through robust curve projection.

[0044] In step S105, the brightened feature image after synchronous upsampling is output to the downstream tracker to generate the final nighttime target detection and tracking results. In the feature extraction process, SCT uses the backbone network of the tracker to guide the iterative optimization of the model and outputs the brightened image.

[0045] Specifically, Figure 3 As shown in Figure 2, the working principle of the multidimensional spatial attention model is as follows:

[0046] Branch a rotates the feature image 90° counterclockwise along the H direction and inputs it into the squeezing mechanism, including global average pooling and standard deviation pooling, to further enhance the feature image representation;

[0047] In branch b, the aggregated feature map and the attention weight along the height direction are extracted in turn, and then the feature map obtained by the sigmoid activation function and the feature map after clockwise rotation are multiplied at the pixel level, and finally the clockwise rotation along the height direction is performed to further establish the dependency relationship between the heights;

[0048] In the c branch, similar to the traditional channel attention mechanism, it is mainly responsible for processing spatial information and capturing key features in the channel, so that the model pays more attention to the appearance information of the object when fusing low-light images. First, the original image is directly input into the compression and excitation changes to obtain the aggregated feature map and channel feature map in turn. Then the weight parameters are activated by the sigmoid function to obtain the channel weight. Finally, the feature weights of the original image are recalculated, and the final feature map is obtained through a simple identity mapping;

[0049] In the d branch, in order to enhance the model's perception of the global information of the image in a night environment, deep convolution is used to capture the spatial relationship between features while retaining the relationship between channels. First, the original image is input into a 5×5 deep convolution module. Then, a multi-scale structure is designed to enhance the ability of the convolution operation to capture spatial relationships. At the same time, considering the real-time problem of the tracking task, a three-branch structure is designed without increasing too much computational complexity. The first branch is a 1×7 and 7×1 deep convolution, and the second branch is a 1×11 and 11×1 deep convolution. The parameter size of the above convolution kernel effectively improves the model's overall perception of the image and the ability to extract multi-scale information. The third branch is an identity mapping. Finally, the feature responses on the three branches are added and fused, and 1×1 convolution is used for channel mixing to assist the a and b branches and produce high-quality spatial attention maps.

[0050] Finally, the output results of branches a, b, c, and d are added pixel by pixel and averaged to fully explore the information of objects in night images and use the detail information contained in early features to improve the problems of blurred and poor brightness of enhanced images.

[0051] like Figure 4As shown, the low-light enhancer that can execute a night target tracking method based on multi-dimensional spatial attention in an intelligent driving scenario provided in this embodiment has a higher success rate and accuracy than other mainstream low-light enhancers when using the online tracking algorithm SiamRPN++ for night tracking on the UAVDark135 test dataset.

[0052] When the image is enhanced by the low-light enhancement method in this embodiment, the pre-learned model parameters are used to adaptively adjust the regional brightness so that the overall image illumination is balanced, and the tracker can robustly track the target. When problems such as target occlusion and background clutter occur, the interference of unimportant areas is effectively suppressed when the enhancer is enabled, which indirectly enhances the tracking performance of the algorithm and enables it to better cope with the above challenges. Figure 5 As shown, a visualization diagram of the low-light enhancer MSA-SCT, which is capable of executing the low-light enhancement method in this embodiment, is displayed on the DarkTrack2021 dataset.

[0053] This embodiment also proposes a nighttime target tracking system 10 based on multi-dimensional spatial attention in an intelligent driving scenario. Figure 6 As shown, including:

[0054] The first generation module 100 is used to encode the input image features of the current image data obtained through the camera through a four-layer downsampling structure to generate a feature map. When the first generation module 100 inputs the image features, it uses multiple downsampling encoders to generate feature maps. Each encoder has two 3×3 convolutional layers and downsampling layers. After the first stage of downsampling, the feature map is obtained, and after each stage of downsampling encoder, the feature map channel is doubled.

[0055] The processing module 200 is used to establish the dependency relationship between the pixels of the feature map using the Transformer-based spatial channel attention module. When establishing the dependency relationship between the pixels, the feature map obtained after downsampling in the fourth stage is input into the Transformer's spatial and channel attention, and is evenly divided into non-overlapping windows of a fixed size for self-attention calculation.

[0056] The second generation module 300 is used to repeatedly extract image features from the first stage to the third stage downsampling stage through a multi-dimensional spatial attention module to obtain the global information of the image to generate a global information feature map. The model understands the importance of each area in the image. When obtaining the global information of the image, the second generation module 300 designs a four-branch multi-dimensional spatial attention mechanism to replace the simple jump link in the U-Net structure. Without increasing the model calculation degree too much, it effectively uses the contextual information in the shallow features to strengthen the image details and improve the enhancement effect.

[0057] The third generation module 400 is used to splice the global information feature map with the feature map of the upsampling stage, and synchronously perform upsampling to generate a brightened feature map. When the third generation module 400 splices the feature images, after the Transformer module, the feature images of the first four downsampling stages are spliced ​​with the upsampling decoders of the corresponding four stages, and an illumination curve mapping and a noise curve mapping are output through the Tanh activation function of the last layer, and an enhanced image is obtained through robust curve projection.

[0058] The output module 500 is used to output the brightened feature image after synchronous upsampling to a downstream tracker to generate a final nighttime target detection and tracking result.

[0059] An electronic device, such as Figure 7 As shown, it includes: a memory 701, a processor 702, and a computer program stored in the memory 701 and executable on the processor, and the processor 702 executes the program to implement a nighttime target tracking method based on multi-dimensional spatial attention in an intelligent driving scenario as provided in the above-mentioned embodiment.

[0060] The electronic device further includes a communication interface 703 for communication between the memory 701 and the processor 702. The memory 701 is used to store a computer program that can be run on the processor 702. The memory 701 may include a high-speed RAM memory, and may also include a nonvolatile memory, such as at least one disk memory.

[0061] If the memory 701, the processor 702 and the communication interface 703 are implemented independently, the communication interface 703, the memory 701 and the processor 702 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0062] A computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement a nighttime target tracking method based on multidimensional spatial attention in an intelligent driving scenario as provided in the above-mentioned embodiment. Wherein, the computer-readable medium can be any device that can contain, store, communicate, propagate or transmit a program for use in an instruction execution system, device or equipment or in combination with these instruction execution systems, devices or equipment. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection portion (electronic device) with one or N wirings, a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting or processing in other suitable ways as necessary, and then stored in a computer memory.

[0063] Although the embodiments of the present invention are described in the specification, these embodiments are only for reference and should not limit the protection scope of the present invention. Various omissions, substitutions and changes within the scope of the present invention should be included in the protection scope of the present invention.

Claims

1. A nighttime target tracking method based on multi-dimensional spatial attention in an intelligent driving scenario, characterized in that: The following steps are involved: The current image data obtained by the camera is encoded into the input image features through a four-layer downsampling structure to generate a feature map; A Transformer-based spatial channel attention module is used to establish the dependency relationship between pixels of the feature map. When establishing the dependency relationship between pixels, the feature map obtained after downsampling in the fourth stage is input into the Transformer's spatial and channel attention, and evenly divided into non-overlapping windows of fixed size for self-attention calculation; The image features from the first stage to the third stage of downsampling are repeatedly extracted through the multi-dimensional spatial attention module to obtain the global information of the image and generate a global information feature map; The global information feature map is concatenated with the feature map in the upsampling stage, and upsampling is performed synchronously to generate a brightening feature map; The brightened feature image after synchronous upsampling is output to a downstream tracker to generate a final nighttime target detection and tracking result.

2. The nighttime target tracking method based on multi-dimensional spatial attention in an intelligent driving scenario according to claim 1 is characterized in that: When inputting image features, multiple downsampling encoders are used to generate feature maps. Each encoder has two 3×3 convolutional layers and downsampling layers. After the first stage of downsampling, the feature map is obtained, and after each stage of downsampling encoder, the feature map channel is doubled.

3. The nighttime target tracking method based on multi-dimensional spatial attention in an intelligent driving scenario according to claim 1 is characterized in that: When acquiring the global information of the image, the simple skip links in the U-Net structure are replaced by a four-branch multi-dimensional spatial attention mechanism, and the contextual information in the shallow features is used to enhance the image details.

4. The nighttime target tracking method based on multi-dimensional spatial attention in an intelligent driving scenario according to claim 1 is characterized in that: When splicing feature images, after the Transformer module, the feature images of the first four downsampling stages are spliced ​​with the upsampling decoders of the corresponding four stages. The Tanh activation function of the last layer outputs an illumination curve mapping and a noise curve mapping, and the enhanced image is obtained through robust curve projection.

5. A nighttime target tracking system based on multi-dimensional spatial attention in an intelligent driving scenario, characterized in that: include: A first generation module is used to encode the input image features of the current image data obtained by the camera through a four-layer downsampling structure to generate a feature map; A processing module is used to establish the dependency relationship between the pixels of the feature map by using the Transformer-based spatial channel attention module. When establishing the dependency relationship between the pixels, the feature map obtained after downsampling in the fourth stage is input into the Transformer's spatial and channel attention, and is evenly divided into non-overlapping windows of a fixed size for self-attention calculation. The second generation module is used to repeatedly extract the image features of the first stage to the third stage of downsampling through the multi-dimensional spatial attention module to obtain the global information of the image to generate a global information feature map; A third generation module is used to splice the global information feature map with the feature map in the upsampling stage, and perform upsampling synchronously to generate a brightening feature map; The output module is used to output the brightened feature image after synchronous upsampling to the downstream tracker to generate the final night target detection and tracking results.

6. The nighttime target tracking system based on multi-dimensional spatial attention in the intelligent driving scenario according to claim 5 is characterized by: When inputting image features, the first generation module uses multiple downsampling encoders to generate feature maps, each encoder has two 3×3 convolutional layers and downsampling layers. After the first stage of downsampling, the feature map is obtained, and after each stage of downsampling encoder, the feature map channel is doubled.

7. The nighttime target tracking system based on multi-dimensional spatial attention in an intelligent driving scenario according to claim 5, characterized in that: When acquiring the global information of the image, the second generation module replaces the simple jump links in the U-Net structure with a four-branch multi-dimensional spatial attention mechanism, and uses the contextual information in the shallow features to enhance the image details.

8. The nighttime target tracking system based on multi-dimensional spatial attention in an intelligent driving scenario according to claim 5, characterized in that: When the third generation module splices the feature images, after the Transformer module, the feature images of the first four downsampling stages are spliced ​​with the upsampling decoders of the corresponding four stages, and an illumination curve mapping and a noise curve mapping are output through the Tanh activation function of the last layer, and an enhanced image is obtained through robust curve projection.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement a nighttime target tracking method based on multi-dimensional spatial attention in an intelligent driving scenario as described in any one of claims 1 to 4.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement a nighttime target tracking method based on multi-dimensional spatial attention in an intelligent driving scenario as described in any one of claims 1 to 4.

Images