A multi-target tracking method and system under low light conditions

Abstract

The present invention discloses a multi-target tracking method and system under low-light conditions, the method comprising: using a dual-camera system to collect paired low-light video data samples; performing multi-target tracking annotation on the low-light video data samples to obtain a low-light multi-target tracking data set; inputting the low-light multi-target tracking data set into a low-light multi-target tracking network model for network training to obtain network parameters; obtaining a trained low-light multi-target tracking network model based on the network parameters, and using the trained low-light multi-target tracking network model to perform low-light multi-target tracking on input real-time low-light video data to obtain a low-light multi-target tracking result. The present invention can effectively improve the performance of multi-target tracking in low-light environments without relying on additional imaging equipment.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-target tracking, and in particular to a multi-target tracking method and system under low-light conditions. Background Art

[0002] Multi-object tracking is an important computer vision task with a wide range of applications, such as autonomous driving, robotics, and security monitoring. The goal of multi-object tracking is to determine the locations of objects in a continuous video sequence and to associate the same objects between different video frames. Some recent multi-object tracking methods have studied multi-object tracking in specific application scenarios, but they all ignore the low-light scenes that are widely present in the real world.

[0003] Limited by current imaging technology, it is very difficult to obtain high-quality video data in low-light conditions. In order to avoid excessive motion blur, current cameras need to set the exposure time to a short time when capturing videos, usually tens of milliseconds. However, in low-light conditions, camera sensors cannot capture enough photoelectrons in such a short time. This limitation causes a significant degradation in image quality in low-light environments, accompanied by greater sensor noise. There are two challenges in multi-target tracking in low-light conditions. The first is to collect and annotate low-light video data. Due to the lower image quality, objects in low-light videos are more difficult to identify and annotate. The second challenge lies in low-light video target tracking. The most popular trackers currently consist of a detector, a motion-based association module, and an appearance-based association module, which rely on high-quality image input. Low-quality low-light images will cause performance degradation in both the detector and the appearance-based association module. A simple approach is to use an image enhancement module, but this introduces additional computation. In addition, images tuned for visual quality may be suboptimal for downstream tasks.

[0004] Traditional low-light enhancement methods are mainly based on histogram equalization and Retinex theory. With the development of deep learning, CNN and Transformer have been applied to low-light enhancement. Depending on the input, low-light enhancement can be divided into RAW-based and RGB-based methods. Since RAW data has more complete original sensor information, it generally has better restoration quality. According to the different data streams processed, there are methods for images and methods for videos. Image-based methods can be directly used for video low-light enhancement, but benefiting from the information of previous and next frames, video-based methods can have better results. Although these methods can restore images with good visual quality, these methods usually require a lot of computation and do not consider downstream tasks. Long and short exposure is a typical method for collecting paired low-light and normal-light images, but this method can only be used to collect low-light images of static scenes. In order to capture dynamic scenes, some researchers have designed a mechatronic system to obtain paired low-light data by repeating two movements. However, this mechatronic system cannot be used to collect videos of dynamic objects in the wild. Jiang et al. designed a dual-camera system that can simultaneously capture paired low-light and normal-light videos, which can be used to capture videos of dynamic scenes and dynamic objects and for multi-object tracking. These works explored methods for building low-light datasets and promoted the study of advanced visual tasks in dark scenes, such as LOD for object detection, SID for instance segmentation, and ExPose for human pose estimation. These datasets provide paired low-light data but only in image format, which cannot be extended for multi-target tracking under low-light conditions.

[0005] In order to track in low-light scenes, some methods explore the use of multimodal information for single target tracking, such as event cameras, depth cameras, and near-infrared cameras. Park et al. proposed the use of short-wave infrared (SWIR) images for multi-target tracking because of its advantages in robustness in low-light conditions. The common disadvantage of these methods is that they require additional hardware equipment and cannot be applied to widely used CMOS imaging systems. SORT uses a Kalman filter motion model and uses IoU for association. ByteTrack enhances tracking performance by considering low-confidence bounding boxes. OS-SORT enhances SORT by recovering lost targets. Recently, Transformer has also been explored for MOT. These methods have achieved high performance in many practical scenarios, and they do not consider handling low-light conditions. Therefore, there is an urgent need for a multi-target tracking method under low-light conditions based on CMOS imaging systems. Summary of the invention

[0006] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0007] To this end, the present invention proposes a multi-target tracking method under low-light conditions. Based on low-pass downsampling and degradation suppression learning strategies, the feature information that is invariant to degradation perturbations is directly learned from low-light videos to avoid the low-light enhancement link. This method can significantly improve the performance of multi-target tracking in low-light scenes with almost no additional computational effort.

[0008] Another object of the present invention is to provide a multi-target tracking system under low light conditions.

[0009] To achieve the above object, the present invention provides a multi-target tracking method under low light conditions, comprising:

[0010] Using a dual-camera system to collect paired low-light video data samples;

[0011] Performing multi-target tracking and annotation on the low-light video data samples to obtain a low-light multi-target tracking dataset;

[0012] Inputting the low-light multi-target tracking data set into the low-light multi-target tracking network model for network training to obtain network parameters;

[0013] A trained low-light multi-target tracking network model is obtained based on the network parameters, and the trained low-light multi-target tracking network model is used to perform low-light multi-target tracking on input real-time low-light video data to obtain a low-light multi-target tracking result.

[0014] The multi-target tracking method under low light conditions of the embodiment of the present invention may also have the following additional technical features:

[0015] In one embodiment of the present invention, the dual camera system comprises a beam splitter, a neutral density filter and a camera; the method of collecting paired low-light video data samples using the dual camera system comprises:

[0016] When imaging an object using a dual camera system, an original light beam is split into the same first light beam and second light beam by a beam splitter;

[0017] Acquire a normal light image based on the first light beam, and use the neutral density filter to weaken the light intensity of the second light beam to obtain a low light image;

[0018] Paired low-light video data samples including the normal-light image and the low-light image are collected using the camera.

[0019] In one embodiment of the present invention, performing multi-target tracking annotation on the low-light video data samples to obtain a low-light multi-target tracking dataset includes:

[0020] Annotating the attribute information of the normal light image to obtain an annotation result; wherein the attribute information includes a bounding box, an identifier, and a visibility state;

[0021] A low-light multi-target tracking dataset of the low-light image is obtained based on the annotation results.

[0022] In one embodiment of the present invention, the low-light multi-target tracking network model includes at least a backbone network, a detection head and an association module; inputting the low-light multi-target tracking data set into the low-light multi-target tracking network model for network training includes:

[0023] Inputting the low-light multi-target tracking dataset into a backbone network to extract perturbation-invariant features; wherein the perturbation-invariant features are the noise in the initial feature map trained by the network based on the smoothness constraint, and the perturbation-invariant features are extracted from the low-light image by the network based on the normal-light image;

[0024] Inputting the perturbation invariant features into a detection head to obtain object position information in each frame of image;

[0025] The object position information is input into the association module to extract the association information of the objects in different frame images, so as to train the low-light multi-target tracking network model based on the association information.

[0026] In one embodiment of the present invention, the training to obtain network parameters includes:

[0027] Acquire the initial feature map in the low-light multi-target tracking dataset, and input the initial feature map into the spatial low-pass convolution to extract low-frequency features;

[0028] Adaptively weighting the low-frequency features, and fusing the weighted features into the initial feature map to obtain low-frequency component features;

[0029] Global average pooling is used to extract channel descriptors of the initial feature map and the low-frequency component features, and weight values ​​are calculated based on the channel descriptors, so as to train network parameters of the low-light multi-target tracking network model based on the weight values.

[0030] To achieve the above object, the present invention further provides a multi-target tracking system under low light conditions, comprising:

[0031] A sample data acquisition module, for collecting paired low-light video data samples using a dual-camera system;

[0032] A target tracking and labeling module is used to perform multi-target tracking and labeling on the low-light video data samples to obtain a low-light multi-target tracking data set;

[0033] A target network training module, used for inputting the low-light multi-target tracking data set into the low-light multi-target tracking network model for network training to obtain network parameters;

[0034] The multi-target real-time tracking module is used to obtain a trained low-light multi-target tracking network model based on the network parameters, and use the trained low-light multi-target tracking network model to perform low-light multi-target tracking on the input real-time low-light video data to obtain a low-light multi-target tracking result.

[0035] The multi-target tracking method and system under low-light conditions of the embodiments of the present invention can directly learn the feature information that is invariant to degradation disturbances from the low-light video and avoid the low-light enhancement link, thereby significantly improving the performance of multi-target tracking in low-light scenes with almost no additional computational effort.

[0036] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0038] Figure 1 is a flow chart of a method for tracking multiple targets under low light conditions according to an embodiment of the present invention;

[0039] Figure 2 is a schematic diagram of a dual camera system according to an embodiment of the present invention;

[0040] Figure 3 is an architecture diagram of a multi-target tracking method under low light conditions according to an embodiment of the present invention;

[0041] Figure 4 is a schematic diagram of sample data of a low-light multi-target tracking dataset according to an embodiment of the present invention;

[0042] Figure 5 is a schematic diagram of the structure of a multi-target tracking system under low light conditions according to an embodiment of the present invention. DETAILED DESCRIPTION

[0043] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0044] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0045] The following describes a method and system for tracking multiple targets under low-light conditions according to an embodiment of the present invention with reference to the accompanying drawings.

[0046] Figure 1 4 is a flow chart of a multi-target tracking method under low light conditions according to an embodiment of the present invention.

[0047] like Figure 1 As shown, the method includes but is not limited to the following steps:

[0048] S1, collects paired low-light video data samples using a dual-camera system.

[0049] It is understandable that the present invention first performs data collection, using the constructed dual camera system to first collect low-light video data samples. The present invention chooses to construct a dual camera system to simultaneously collect low-light and normal-light video pairs.

[0050] In one embodiment of the present invention, a dual camera system is constructed and paired low-light-normal light video data is collected. Multi-target tracking requires dynamic scene and object videos. In order to collect low-light videos for multi-target tracking, the present invention constructs a dual camera system such as Figure 2 As shown, the system can capture pairs of low-light and high-light video frames simultaneously. The dual-camera system constructed by the present invention. Its main components include a beam splitter, a neutral density filter (ND filter) and two cameras. During imaging, the light will be split into two identical beams by the beam splitter, one of which will be weakened by the ND filter to obtain a low-light image, and the other beam will be used to obtain a normal-light image. Finally, the two cameras receive the light and complete the acquisition of the low-light image and the normal-light image.

[0051] The dual-camera system of the present invention is designed so that two cameras can simultaneously capture fully aligned low-light and normal-light video data. To ensure the time synchronization of video frames, a high-speed hardware interface is used to trigger camera exposure events. In addition, to avoid possible frame loss problems, the dual-camera system uses two independent hardware interfaces for video reading and writing, and is equipped with a high-speed solid-state drive. After precise calibration, the dual-camera system can capture paired low-light and normal-light video data of dynamic scenes and objects in real time.

[0052] Furthermore, in order to enable the camera to capture more scene information in low-light environments, the RAW format is used to store all video data. The exposure time of both cameras is set to 10ms and the frame rate is fixed to 20 to capture smooth video and avoid excessive motion blur. The optimal digital gain level is determined for the normal light camera to obtain the highest quality well-lit images. The gain of the low-light camera is always set to the maximum value to simulate the low-light shooting settings in real scenes. For real night videos, an embodiment of the present invention removes the ND filter, directly connects the lens to the camera, and keeps all other camera settings unchanged. Pairs of low-light video data are then collected using the constructed dual-camera system.

[0053] The video collection of the present invention includes various urban outdoor scenes, including roads, overpasses, etc. The overpass scene is shot from a bird's-eye view, while all other scenes are shot from the perspective of pedestrians. In order to consider the impact of camera motion, arbitrary horizontal rotation and vertical random motion of the camera are introduced. Figure 4 Two sample video sequences from the LMOT dataset are shown.

[0054] S2, perform multi-target tracking and annotation on low-light video data samples to obtain a low-light multi-target tracking dataset.

[0055] Specifically, the dual-camera system of the present invention can also annotate low-light multi-target tracking datasets. The present invention collects a new low-light multi-target tracking dataset LMOT. It provides well-aligned low-light and normal-light video pairs, real low-light scene videos, and multi-target tracking annotations for all videos. It includes 32 videos and 815K labeled boxes. Among them, 6 videos were collected in real night low-light environments to verify the performance in real low-light environments. Real low-light videos are used to verify the performance of the model in real scenes. LMOT is collected in urban outdoor scenes. It can be known that MOT requires videos of dynamic scenes as input.

[0056] Specifically, multi-target tracking and annotation are performed on normal light videos. For partially visible objects, the embodiment of the present invention will annotate a whole box; for completely invisible objects, if they are still in the camera, the estimated bounding box will be annotated. Each object in a single video is uniquely numbered. After annotation, the low-light multi-target tracking dataset LMOT can be obtained.

[0057] It is understandable that multi-target tracking based on deep learning requires the target to be tracked in the video to be annotated for model training and evaluation of model performance. Due to the low brightness and poor image quality of low-light videos, it is difficult and time-consuming to annotate directly on low-light videos. Thanks to the fully aligned low-light and normal-light videos captured by the dual-camera system, the embodiment of the present invention can directly annotate on the normal-light video and simultaneously obtain the multi-target tracking annotation of the low-light video. This greatly reduces the difficulty of annotation and improves the quality of annotation. The embodiment of the present invention annotates six common urban moving objects, including cars, people, bicycles, motorcycles, buses and trucks. The annotation includes bounding boxes, logos and visibility states. For partially occluded objects, a complete object box is annotated. For fully occluded objects, the object position is estimated and annotated. In the entire video, each object has a unique ID. After completing the annotation, the embodiment of the present invention carefully checks and corrects all annotations to ensure the quality of the data set. After completing all data set collection and annotation, the embodiment of the present invention randomly divides the video into a training set, a validation set and a test set, which are composed of 11, 4 and 11 videos respectively. The embodiment of the present invention also provides a LMOT real dataset, which captures 6 videos in real-world night scenes.

[0058] It is understandable that compared with the existing multi-target tracking datasets, the low-light multi-target tracking dataset collected by the embodiment of the present invention has a considerable data size and provides RAW format image data, which brings more flexibility to the research of low-light multi-target tracking.

[0059] S3, inputting the low-light multi-target tracking dataset into the low-light multi-target tracking network model for network training to obtain network parameters.

[0060] Specifically, the present invention uses the low-light multi-target tracking dataset LMOT to train a multi-target tracking model and saves the obtained parameters.

[0061] In one embodiment of the present invention, the present invention uses the collected low-light multi-target tracking dataset LMOT to train the proposed low-light multi-target tracking network and save the learned network parameters. The embodiment of the present invention observed greater sensor noise in low-light images, showing some similarities with random noise attacks. In a model that is not specially designed, this will cause significant degradation of shallow and deep features and reduce network performance. In order to address this problem, the embodiment of the present invention makes improvements in three aspects: network structure, training strategy and data enhancement. An adaptive low-pass downsampling module is used to help the network extract key image semantic information from high-frequency noise perturbations. Paired low-light and normal light images are used to help the network learn perturbation invariant features from degraded and noise-perturbed images. In addition, simulated low-light images based on physical noise models are also used for training.

[0062] It is understandable that in low-light scenes, the camera can only capture a small number of photons in a single exposure. Therefore, potential sensor noise is highlighted and causes significant degradation of the final image. The embodiments of the present invention observed that directly feeding low-light images to the network without any special design leads to feature map degradation and significantly reduces the performance of the model. A straightforward solution is to apply a low-light enhancement technique that learns a mapping function from a low-light image to a clean normal-light image. Since this is a highly ill-posed problem, learning such a mapping function requires considerable computational and storage overhead. Although DNN-based methods have achieved excellent performance in low-light enhancement, images with enhanced visual quality may be suboptimal for downstream tasks.

[0063] In this work, the embodiments of the present invention bypass low-light enhancement and perform multi-target tracking in low-light images. The overall framework is as follows Figure 3 As shown. In order to improve the performance and robustness of the multi-target tracking model, the main idea of ​​the embodiment of the present invention is to obtain unchanged semantic information in the case of noise interference quality degradation. The present invention observes that when a low-light image is input to a neural network that has undergone any special processing, the shallow feature map of the neural network will also contain large noise due to the larger sensor noise, and the feature response to the object cannot be generated in the deep feature map. Therefore, the embodiment of the present invention proposes an adaptive low-pass downsampling (ALD) module to enhance the low-frequency components in the image and suppress the high-frequency noise components. The embodiment of the present invention also proposes a degradation suppression learning strategy (DSL), which helps the model suppress image noise interference by using paired low-light and normal light videos, and encourages deep features to respond to target objects.

[0064] The video images in the low-light multi-target tracking dataset are first fed into the backbone network to extract perturbation-invariant features. The extracted features are then fed into the detection head and association module to obtain the object location in each frame and the association of objects in different frames.

[0065] Preferably, the embodiment of the present invention adopts CSPDarknet as the backbone network and makes improvements thereon. The association module may use many currently widely used motion-based association modules, such as SORT, DanceTrack, and OC-SORT.

[0066] Among them, the downsampling operation reduces the feature size while retaining the most important information. The noise in the low-light image will cause high-frequency interference to the feature map, thereby misleading the preservation of the target information. In order to suppress the influence of high-frequency noise and enhance the low-frequency part of the feature map, the embodiment of the present invention introduces spatial low-pass convolution (SConv), which focuses on learning to extract low-frequency component features from the feature map. The softmax function is used to constrain the original convolution kernel to be low-pass, expressed as:

[0067]

[0068] Where K is the original convolution kernel, is a low-pass convolution kernel. The embodiment of the present invention uses a standard Gaussian kernel to initialize the spatial low-pass convolution kernel. The size of the convolution kernel is set to 5 to obtain more spatial information. Then, the obtained low-frequency features are adaptively weighted and fused into the initial feature map to enhance the low-frequency components. The embodiment of the present invention uses global average pooling to obtain channel descriptors of the initial feature map and low-frequency component features. These descriptors are then fed into the fully connected layer to calculate the weight values, thereby obtaining the network parameters of the low-light multi-target tracking network model through training based on the weight values.

[0069] Among them, if two pairs of low-light images and normal-light images are input into the same neural network, ideally the model should show the same feature response to them, that is, it should be robust to low-light images. However, the embodiments of the present invention found that when facing low-light images, the existing methods have shallow features full of noise and deep features cannot produce feature responses to objects. In order to address this problem, the idea of ​​the embodiments of the present invention is to add smoothing constraints to help the network learn to suppress noise in shallow network features, and use normal-light images to help the model learn to extract perturbation-invariant features from low-light images. The degradation suppression loss can be expressed as:

[0070]

[0071] Where L DS is the degradation loss, and denote normal light and low light image features respectively, and l denotes the lth feature map. In order to further suppress the noise in the shallow features, the embodiment of the present invention introduces an additional variational loss (TV), which is expressed as:

[0072]

[0073] Among them G row and G col Represent the first-order variational matrices in the row and column directions respectively. The variational loss adds additional spatial smoothness constraints to the feature map, helping the model learn invariant features under noise. An excellent model should be able to perform well on both low-light images and normal-light images, so both low-light images and normal-light images are used to train the model, through a loss function, expressed as:

[0074]

[0075] in and denote the detection losses of normal light and low light images respectively, α, β and γ are the weights of the loss function, and the embodiment of the present invention sets them to 1, 1 and 0.01 respectively.

[0076] The embodiment of the present invention uses YOLOX as a detector. The model is pre-trained on the COCO dataset and then trained for 24 epochs on the LMOT dataset of the embodiment of the present invention. The embodiment of the present invention applies data enhancement strategies, including random flipping, resized proportional jitter, and mosaic enhancement strategies. The embodiment of the present invention also uses a physics-based noise model to enhance RAW images. The embodiment of the present invention uses an SGD optimizer with weight decay 0.0001 and a cosine learning rate schedule, with an initial learning rate of 0.001 and gradually reduced to 0.0001.

[0077] S4, obtaining a trained low-light multi-target tracking network model based on the network parameters, and using the trained low-light multi-target tracking network model to perform low-light multi-target tracking on the input real-time low-light video data to obtain a low-light multi-target tracking result.

[0078] It is understandable that the present invention uses the learned network parameters to input the low-light video into the model for inference calculation and obtain the position and number of the object. In this stage, the network parameters are fixed and there is no need to learn and update the network parameters.

[0079] Specifically, an improved spatial low-pass downsampling module and degradation suppression learning strategy are used to learn to extract invariant features for target detection from low-light images with degraded image quality and greater noise. Using the object's motion information, objects are associated online and assigned object numbers, which are the corresponding low-light multi-target tracking results.

[0080] It is understandable that during the use phase, the low-light multi-target tracking network proposed in the present invention will be deployed to the server for use. According to the pre-trained network parameters, the input video sequence is sequentially sent to the network for inference operations, and all tracked targets and their corresponding numbers are output. In order to speed up the inference speed, half-precision is used to perform inference operations on the network during the deployment phase to obtain a higher data throughput rate.

[0081] Furthermore, after completing the low-light multi-target tracking test, the present invention evaluates the performance of multi-target tracking under low-light conditions, with the aim of quantitatively evaluating the practical application performance of the proposed low-light multi-target tracking method and system. The embodiment of the present invention recommends using HOTA as the main evaluation indicator to simultaneously evaluate the detection and association performance. The embodiment of the present invention also uses AssA and IDF1 to evaluate the association performance, and uses MOTA and Deta to evaluate the detection performance. There are two ways to combine the indicators of all classes into one score. One is to take the average of the metrics of the class values, and the other is to take the average of the detection values. In order to avoid the possible result deviation caused by certain categories with fewer samples (such as buses and trucks), the embodiment of the present invention performs a weighted average of the detection values ​​according to the number of categories to obtain the final indicator result.

[0082] The multi-target tracking method under low-light conditions in the embodiment of the present invention creatively introduces an adaptive low-pass downsampling module and a degradation suppression learning strategy to learn and extract perturbation-invariant features from low-light video images with degraded image quality and greater sensor noise, effectively solving the problems of performance degradation and poor results of current multi-target tracking methods under low-light conditions. It provides solutions for fields such as security monitoring, autonomous driving, and robotics. At the same time, the present invention provides a new large-scale data set and a new idea for studying computer vision tasks under low-light conditions.

[0083] In order to implement the above embodiment, Figure 5 As shown, this embodiment also provides a multi-target tracking system 10 under low light conditions, the system 10 includes a sample data acquisition module 100, a target tracking and labeling module 200, a target network training module 300 and a multi-target real-time tracking module 400;

[0084] The sample data acquisition module 100 is used to collect paired low-light video data samples using a dual-camera system;

[0085] The target tracking and labeling module 200 is used to perform multi-target tracking and labeling on low-light video data samples to obtain a low-light multi-target tracking data set;

[0086] The target network training module 300 is used to input the low-light multi-target tracking data set into the low-light multi-target tracking network model for network training to obtain network parameters;

[0087] The multi-target real-time tracking module 400 is used to obtain a trained low-light multi-target tracking network model based on network parameters, and use the trained low-light multi-target tracking network model to perform low-light multi-target tracking on the input real-time low-light video data to obtain a low-light multi-target tracking result.

[0088] Furthermore, the dual camera system includes a beam splitter, a neutral density filter and a camera; the sample data acquisition module 100 is also used for:

[0089] When imaging an object using a dual camera system, an original light beam is split into the same first light beam and second light beam by a beam splitter;

[0090] Acquire a normal light image based on the first light beam, and use a neutral density filter to weaken the light intensity of the second light beam to obtain a low light image;

[0091] A camera is used to collect pairs of low-light video data samples including a normal-light image and the low-light image.

[0092] Furthermore, the target tracking and labeling module 200 is also used for:

[0093] Annotating the attribute information of the normal light image to obtain an annotation result; wherein the attribute information includes a bounding box, an identifier, and a visibility state;

[0094] A low-light multi-target tracking dataset of the low-light image is obtained based on the annotation results.

[0095] Furthermore, the low-light multi-target tracking network model includes at least a backbone network, a detection head and an association module; the target network training module 300 is also used for:

[0096] Inputting the low-light multi-target tracking dataset into the backbone network to extract perturbation-invariant features; wherein the perturbation-invariant features are the perturbation-invariant features extracted from the low-light image by the network based on the normal-light image by suppressing noise in the initial feature map based on smoothness constraint training;

[0097] Inputting the perturbation invariant features into a detection head to obtain object position information in each frame of image;

[0098] The object position information is input into the association module to extract the association information of the objects in different frame images, so as to train the low-light multi-target tracking network model based on the association information.

[0099] Furthermore, the target network training module 300 is also used for:

[0100] Obtaining the initial feature map in the low-light multi-target tracking dataset, and inputting the initial feature map into the spatial low-pass convolution to extract low-frequency features;

[0101] Adaptively weight the low-frequency features and fuse the weighted features into the initial feature map to obtain the low-frequency component features;

[0102] Global average pooling is used to extract the initial feature map and the channel descriptor of the low-frequency component features, and weight values ​​are calculated based on the channel descriptors to train the network parameters of the low-light multi-target tracking network model based on the weight values.

[0103] The multi-target tracking system under low-light conditions in the embodiment of the present invention creatively introduces an adaptive low-pass downsampling module and a degradation suppression learning strategy to learn and extract perturbation-invariant features from low-light video images with degraded image quality and greater sensor noise, effectively solving the problems of performance degradation and poor results of current multi-target tracking methods under low-light conditions. It provides solutions for fields such as security monitoring, autonomous driving, and robotics. At the same time, the present invention provides a new large-scale data set and a new idea for studying computer vision tasks under low-light conditions.

[0104] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0105] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

Claims

1. A multi-target tracking method under low light conditions, characterized in that: The following steps are involved: Using a dual-camera system to collect paired low-light video data samples; Performing multi-target tracking and annotation on the low-light video data samples to obtain a low-light multi-target tracking dataset; Inputting the low-light multi-target tracking data set into the low-light multi-target tracking network model for network training to obtain network parameters; A trained low-light multi-target tracking network model is obtained based on the network parameters, and the trained low-light multi-target tracking network model is used to perform low-light multi-target tracking on input real-time low-light video data to obtain a low-light multi-target tracking result; The dual camera system comprises a beam splitter, a neutral density filter and a camera; the dual camera system is used to collect paired low-light video data samples, comprising: When imaging an object using a dual camera system, an original light beam is split into the same first light beam and second light beam by a beam splitter; Acquire a normal light image based on the first light beam, and use the neutral density filter to weaken the light intensity of the second light beam to obtain a low light image; collecting, using the camera, paired low-light video data samples including the normal-light image and the low-light image; Performing multi-target tracking and annotation on the low-light video data samples to obtain a low-light multi-target tracking dataset, including: Annotating the attribute information of the normal light image to obtain an annotation result; wherein the attribute information includes a bounding box, an identifier, and a visibility state; Obtaining a low-light multi-target tracking dataset of the low-light image based on the annotation results; The low-light multi-target tracking network model at least includes a backbone network, a detection head and an association module; inputting the low-light multi-target tracking data set into the low-light multi-target tracking network model for network training, including: Inputting the low-light multi-target tracking dataset into a backbone network to extract perturbation-invariant features; wherein the perturbation-invariant features are the noise in the initial feature map trained by the network based on the smoothness constraint, and the perturbation-invariant features are extracted from the low-light image by the network based on the normal-light image; Inputting the perturbation invariant features into a detection head to obtain object position information in each frame of image; Inputting the object position information into the association module to extract association information of objects in different frame images, so as to train a low-light multi-target tracking network model based on the association information; The training obtains network parameters, including: Obtaining the initial feature map in the low-light multi-target tracking dataset, and inputting the initial feature map into a spatial low-pass convolution to extract low-frequency features; Adaptively weighting the low-frequency features, and fusing the weighted features into the initial feature map to obtain low-frequency component features; Global average pooling is used to extract channel descriptors of the initial feature map and the low-frequency component features, and weight values ​​are calculated based on the channel descriptors, so as to train network parameters of the low-light multi-target tracking network model based on the weight values.

2. A multi-target tracking system under low light conditions, characterized in that: include: A sample data acquisition module, for collecting paired low-light video data samples using a dual-camera system; A target tracking and labeling module is used to perform multi-target tracking and labeling on the low-light video data samples to obtain a low-light multi-target tracking data set; A target network training module, used for inputting the low-light multi-target tracking data set into the low-light multi-target tracking network model for network training to obtain network parameters; A multi-target real-time tracking module, used to obtain a trained low-light multi-target tracking network model based on the network parameters, and use the trained low-light multi-target tracking network model to perform low-light multi-target tracking on input real-time low-light video data to obtain a low-light multi-target tracking result; The dual-camera system includes a beam splitter, a neutral density filter and a camera; the sample data acquisition module is also used for: When imaging an object using a dual camera system, an original light beam is split into the same first light beam and second light beam by a beam splitter; Acquire a normal light image based on the first light beam, and use the neutral density filter to weaken the light intensity of the second light beam to obtain a low light image; collecting, using the camera, paired low-light video data samples including the normal-light image and the low-light image; The target tracking and labeling module is also used for: Annotating the attribute information of the normal light image to obtain an annotation result; wherein the attribute information includes a bounding box, an identifier, and a visibility state; Obtaining a low-light multi-target tracking dataset of the low-light image based on the annotation results; The low-light multi-target tracking network model at least includes a backbone network, a detection head and an association module; the target network training module is also used for: Inputting the low-light multi-target tracking dataset into a backbone network to extract perturbation-invariant features; wherein the perturbation-invariant features are the perturbation-invariant features extracted from the low-light image by the network based on the normal-light image by suppressing noise in the initial feature map based on smoothness constraint training; Inputting the perturbation invariant features into a detection head to obtain object position information in each frame of image; Inputting the object position information into the association module to extract association information of objects in different frame images, so as to train a low-light multi-target tracking network model based on the association information; The target network training module is further used for: Obtaining the initial feature map in the low-light multi-target tracking dataset, and inputting the initial feature map into a spatial low-pass convolution to extract low-frequency features; Adaptively weighting the low-frequency features, and fusing the weighted features into the initial feature map to obtain low-frequency component features; Global average pooling is used to extract channel descriptors of the initial feature map and the low-frequency component features, and weight values ​​are calculated based on the channel descriptors, so as to train network parameters of the low-light multi-target tracking network model based on the weight values.

Images