Multi-modal visual tracking method and system based on monitoring scene

Abstract

The application discloses a kind of multi-modal visual tracking method and system based on monitoring scene, image processing technical field.The method comprises the following steps: obtaining visible light image dataset and infrared light image dataset;Visible light image dataset and infrared light image dataset are input into common feature extraction branch structure to extract features, and obtain multi-modal common feature;Visible light image dataset is input into visible light feature extraction branch structure to extract features, and obtain visible light unique feature;Infrared light image dataset is input into infrared light feature extraction branch structure to extract features, and obtain infrared light unique feature;Multi-modal common feature, visible light unique feature and infrared light unique feature are fused to obtain multi-modal fusion feature;According to multi-modal fusion feature, visual tracking is carried out on multiple targets, the visual perception under multiple different modes can be known, the accuracy of perception is improved, and the accuracy of visual tracking under monitoring scene is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a multi-modal visual tracking method and system based on a monitoring scene. BACKGROUND

[0002] In recent years, high-position video technology has developed rapidly. Through the installation of high-position video cameras at road sides, intersections and the like for data collection, a variety of tasks such as vehicle detection, license plate recognition, pedestrian detection and parking line detection can be achieved on the collected data, which is conducive to more precise and accurate management of roadside parking. Among them, the target tracking technology also has important significance. Through visual tracking of targets such as vehicles and pedestrians, pedestrian prediction and vehicle trajectory prediction can be performed, thereby positively promoting various aspects such as urban traffic management and driving safety.

[0003] However, the traditional target tracking method uses visible light video image data for visual tracking algorithm based on monitoring video images. However, in the face of light changes, rain and fog and the like, the quality of visible light video images deteriorates, resulting in low accuracy during target tracking. SUMMARY

[0004] The purpose of the present application is to solve the technical problem of low accuracy during target tracking caused by the traditional target tracking method using visible light video image data for visual tracking. To achieve the above purpose, the present application provides a multi-modal visual tracking method and system based on a monitoring scene.

[0005] The present application provides a multi-modal visual tracking method based on a monitoring scene, comprising:

[0006] obtaining a visible light image data set and an infrared light image data set;

[0007] inputting the visible light image data set and the infrared light image data set into a common feature extraction branch structure for feature extraction to obtain multi-modal common features;

[0008] inputting the visible light image data set into a visible light feature extraction branch structure for feature extraction to obtain visible light unique features;

[0009] inputting the infrared light image data set into an infrared light feature extraction branch structure for feature extraction to obtain infrared light unique features;

[0010] performing feature fusion on the multi-modal common features, the visible light unique features and the infrared light unique features to obtain multi-modal fusion features;

[0011] performing visual tracking on multiple targets according to the multi-modal fusion features.

[0012] In one embodiment, visually tracking multiple targets based on the multimodal fusion features includes:

[0013] Performing target tracking on the multimodal fusion features according to a target tracking network to obtain the target category and target position of each target in each frame image;

[0014] The intersection-and-union ratio of target positions in adjacent frame images is used as the adjacent frame target relationship matching index. The target position of each target in each frame image is tracked according to the online tracking algorithm to obtain the matching relationship of the same target in adjacent frame images.

[0015] In one embodiment, the visual tracking of multiple targets based on the multimodal fusion features further includes:

[0016] The target position of each target in the continuous frame images is predicted and tracked according to the Kalman filter algorithm to obtain a coherent target position change trajectory.

[0017] In one embodiment, the visual tracking of multiple targets based on the multimodal fusion features further includes:

[0018] The target positions of the targets in each frame image are subjected to target association according to the Hungarian algorithm to obtain the multi-target association relationship in adjacent frame images.

[0019] In one embodiment, after fusing the multimodal common features, the visible light unique features, and the infrared light unique features to obtain a multimodal fusion feature, and before visually tracking multiple targets based on the multimodal fusion feature, the method further includes:

[0020] Performing model training on the common feature extraction branch structure to obtain common feature extraction branch parameters, and fixing the common feature extraction branch parameters;

[0021] Performing model training on the visible light feature extraction branch structure to obtain visible light feature extraction branch parameters;

[0022] Performing model training on the infrared light feature extraction branch structure to obtain infrared light feature extraction branch parameters;

[0023] Among them, the convolution kernel size of the common feature extraction branch structure is larger than the convolution kernel size of the infrared light feature extraction branch structure, and the convolution kernel size of the infrared light feature extraction branch structure is larger than the convolution kernel size of the visible light feature extraction branch structure.

[0024] The present invention provides a multimodal visual tracking system based on a monitoring scene, comprising:

[0025] A data acquisition module is used to acquire a visible light image dataset and an infrared light image dataset;

[0026] a shared feature extraction module, configured to input the visible light image dataset and the infrared light image dataset into a shared feature extraction branch structure to perform feature extraction and obtain multimodal shared features;

[0027] A visible light feature extraction module is used to input the visible light image data set into the visible light feature extraction branch structure to perform feature extraction and obtain unique features of visible light;

[0028] an infrared light feature extraction module, configured to input the infrared light image data set into the infrared light feature extraction branch structure to perform feature extraction and obtain unique infrared light features;

[0029] a feature fusion module, configured to fuse the multimodal common features, the visible light unique features, and the infrared light unique features to obtain a multimodal fusion feature;

[0030] The visual tracking module is used to visually track multiple targets based on the multimodal fusion features.

[0031] In one embodiment, the visual tracking module includes:

[0032] A target information acquisition module is used to track the multimodal fusion features according to the target tracking network to obtain the target category and target position of each target in each frame image;

[0033] The same-target matching module is used to use the intersection-and-union ratio of the positions of the targets in adjacent frame images as the matching index of the target relationship between adjacent frames, track the target position of each target in each frame image according to the online tracking algorithm, and obtain the same-target matching relationship in adjacent frame images.

[0034] In one embodiment, the visual tracking module further comprises:

[0035] The coherent target trajectory acquisition module is used to predict and track the target position of each target in the continuous frame image according to the Kalman filter algorithm to obtain a coherent target position change trajectory.

[0036] In one embodiment, the visual tracking module further comprises:

[0037] The multi-target association module is used to perform target association on the target positions of each target in each frame image according to the Hungarian algorithm to obtain the multi-target association relationship in adjacent frame images.

[0038] In one embodiment, the system further comprises:

[0039] A common feature branch training module is configured to train a model of the common feature extraction branch structure, obtain common feature extraction branch parameters, and fix the common feature extraction branch parameters.

[0040] A visible light feature branch training module is configured to train a model of the visible light feature extraction branch structure, obtain visible light feature extraction branch parameters.

[0041] An infrared light feature branch training module is configured to train a model of the infrared light feature extraction branch structure, obtain infrared light feature extraction branch parameters.

[0042] The common feature extraction branch structure has a convolution kernel size greater than that of the infrared light feature extraction branch structure, and the infrared light feature extraction branch structure has a convolution kernel size greater than that of the visible light feature extraction branch structure.

[0043] In the above-mentioned multi-modal visual tracking method and system based on a monitoring scene, common features, visible light unique features, and infrared light unique features are extracted from each image in the visible light image dataset and the infrared light image dataset, and multi-modal common features, visible light unique features, and infrared light unique features are fused to obtain multi-modal fusion features. The multi-modal fusion features not only fuse image features obtained by an infrared camera, but also fuse image features obtained by a visible light camera, and can obtain more accurate image features in night or rain, fog, or light changes. The multi-modal fusion features fuse image features in multiple modalities, which can complement image features in each modality and make up for the defects of image features obtained in a single modality. Furthermore, the multi-modal visual tracking method based on a monitoring scene provided by the present application can obtain visual perception in multiple different modalities, improve the accuracy of perception, and improve the accuracy of visual tracking in a monitoring scene. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 is a step flowchart of the multi-modal visual tracking method based on a monitoring scene provided by the present application.

[0045] Figure 2 is a structural diagram of the multi-modal visual tracking system based on a monitoring scene provided by the present application. DETAILED DESCRIPTION

[0046] The technical solutions of the present application will be described in further detail below with reference to the drawings and examples.

[0047] Please refer to Figure 1 The present application provides a multi-modal visual tracking method based on a monitoring scene, which comprises:

[0048] S10, obtaining a visible light image dataset and an infrared light image dataset;

[0049] S20, inputting the visible light image dataset and the infrared light image dataset into a common feature extraction branch structure to perform feature extraction to obtain multimodal common features;

[0050] S30, inputting the visible light image data set into the visible light feature extraction branch structure to perform feature extraction to obtain unique features of visible light;

[0051] S40, inputting the infrared image data set into the infrared feature extraction branch structure to perform feature extraction to obtain unique infrared features;

[0052] S50, performing feature fusion on the multimodal common features, the visible light unique features, and the infrared light unique features to obtain a multimodal fusion feature;

[0053] S60, visually tracking multiple targets based on multimodal fusion features.

[0054] In this embodiment, in S10, a visible light and infrared light visual tracking dataset is constructed. Two sensing devices, a visible light camera and an infrared camera, are installed in a single device. The two cameras can be placed horizontally or vertically, for example. By adjusting the observation angles of the two cameras, the two cameras capture visible light video images and infrared video images of the same area, thereby capturing visible light video images and infrared video images that are synchronized in time and perception area. Multi-target annotation is performed on each video clip of the visible light modality and the infrared light modality, using rectangular boxes to describe the target location. The multi-target categories include, but are not limited to, vehicles, pedestrians, and non-motor vehicles.

[0055] In S20, the visible light video image and the infrared light video image are collected from different devices, and include different features and common features of the target. The common features of the target include features such as the shape of the target and the edge position of the target. The common features of the two modalities can be extracted through the common feature extraction branch structure. The input of the common feature extraction branch structure is the visible light image dataset and the infrared light image dataset under the two modalities. The common feature extraction branch structure includes multiple convolutional neural network modules for feature extraction. Each convolutional neural network module includes a convolution layer, a pooling layer, a normalization layer, and an activation function layer.

[0056] In S30, the visible light image dataset contains rich color information and more detailed texture features. The visible light feature extraction branch structure extracts unique features of visible light, enhancing the extraction of visible light data features. The visible light feature extraction branch structure includes multiple convolutional modules for feature extraction. Each convolutional module consists of a convolution layer, a pooling layer, a normalization layer, and an activation function layer.

[0057] In S40, the infrared light data can achieve better target feature acquisition in special weather such as night, rain and fog. The infrared light feature extraction branch structure can extract unique infrared light features. The infrared light feature extraction branch structure includes multiple convolution modules for feature extraction. Each convolution module includes a convolution layer, a pooling layer, a normalization layer, and an activation function layer.

[0058] In S50, the multi-modal common features, visible light unique features, and infrared light unique features are fused, and the features of the three branches are size-aligned. The output of the three branches is aligned by using methods such as deconvolution or bilinear interpolation. The features of the three branches after alignment are fused by pixel-by-pixel addition to obtain multi-modal fusion features under multi-modal data.

[0059] In S60, the multi-modal fusion features fuse the image features under the visible light mode, the image features under the infrared light mode, and the common features under the visible light mode and the infrared light mode. The features in the image are extracted from multiple different angles, and the feature states of each target in the image can be fully learned. Therefore, according to the multi-modal fusion features corresponding to each visible light image and infrared light image, multiple targets in the image can be visually tracked.

[0060] The multi-modal visual tracking method based on a monitoring scene provided by the present application extracts common features, visible light unique features, and infrared light unique features from each image in the visible light image dataset and the infrared light image dataset, and fuses the multi-modal common features, the visible light unique features, and the infrared light unique features to obtain multi-modal fusion features. The multi-modal fusion features not only fuse the image features obtained by the infrared camera, but also fuse the image features obtained by the visible light camera, which can obtain more accurate image features in night or rain and fog or light changes. The multi-modal fusion features fuse image features under multiple modalities, which can complement each other and make up for the defects of image features obtained under a single modality. Furthermore, the multi-modal visual tracking method based on a monitoring scene provided by the present application can obtain visual perception under multiple different modalities, improve the accuracy of perception, and improve the accuracy of visual tracking in a monitoring scene.

[0061] In one embodiment, S60, the multi-modal fusion features are used to visually track multiple targets, including:

[0062] S610, the multi-modal fusion features are tracked by a target tracking network to obtain the target class and target position of each target in each frame of image.

[0063] S620, the position intersection ratio between targets in adjacent frame images is the adjacent frame target relationship matching index, the target position of each target in each frame image is tracked according to the online tracking algorithm, and the same target matching relationship in adjacent frame images is obtained.

[0064] In this embodiment, the target tracking network is used for predicting the target position (which can be understood as a rectangular box) and the target category of the multi-modal fusion feature, and obtaining the category and position information of multiple targets in each frame of data. The target tracking network includes multiple convolution modules and multiple fully connected modules. Each convolution module includes a convolution layer and an activation function layer. Each fully connected module includes a fully connected layer, a random deactivation layer, and an activation function layer. The fully connected layer can connect each node to all nodes of the previous layer, and integrate the extracted features. The random deactivation layer is connected to the fully connected layer. The fully connected layer and the random deactivation layer can avoid overfitting of the model. The activation function layer can be a ReLU activation function.

[0065] The position intersection ratio between targets in adjacent frame images can be understood as the intersection ratio of the rectangular boxes between the targets in adjacent frame images, as a target relationship matching index of adjacent two frame images in the visual tracking process. The target position of each target in each frame image is tracked by the online tracking algorithm (Simple online and realtime tracking, SORT) with the adjacent frame target relationship matching index, and the same target matching relationship in adjacent frame images is obtained. The same target matching relationship can be understood as which targets in adjacent two frame images are the same target, and if they are the same target, it is considered that they have the same target matching relationship. For example, a target intersection ratio threshold is set, target 1 and target 2 are detected in the first frame image, and the target intersection ratio between target 1 and target 2 is obtained. Target 3 and target 4 are detected in the second frame image. By the online tracking algorithm, the target intersection ratio is used as the matching index, it can be known that target 1 in the first frame image and target 3 in the second frame image have the same target matching relationship, that is, target 1 in the first frame image and target 3 in the second frame image are the same target, and the matching association of the same target relationship in adjacent frame images is realized, thereby realizing the visual tracking of the same target in adjacent frame images.

[0066] In one embodiment, S60, the visual tracking of multiple targets according to the multi-modal fusion feature further includes:

[0067] S630, the target position of each target in consecutive frame images is tracked according to the Kalman filtering algorithm, and the continuous target position change trajectory is obtained.

[0068] In this embodiment, the Kalman filter algorithm uses the linear system state equation to smooth the target position of each target in the continuous frame images through two processes of prediction and update, obtains a coherent position change, and then obtains a coherent target position change trajectory in the continuous frame images.

[0069] In one embodiment, S60, visually tracking multiple targets based on the multimodal fusion features, further includes:

[0070] S640: Perform target association on the target positions of each target in each frame image according to the Hungarian algorithm to obtain a multi-target association relationship in adjacent frame images.

[0071] In this embodiment, the Hungarian algorithm is an algorithm for finding maximum matching in graph theory. By using the Hungarian algorithm to associate multiple detected targets, visual tracking of multiple targets is achieved. The multi-target association relationship can be understood as the association relationship between multiple targets in adjacent frame images. For example, if N targets are detected in the first frame and M targets are detected in the second frame, the Hungarian algorithm can determine the association relationship between the N targets in the first frame and the M targets in the second frame.

[0072] In one embodiment, after performing feature fusion on the multimodal common features, the visible light unique features, and the infrared light unique features to obtain the multimodal fusion features, and before visually tracking multiple targets based on the multimodal fusion features, at step S60, the method further includes:

[0073] S501, performing model training on the common feature extraction branch structure, obtaining common feature extraction branch parameters, and fixing the common feature extraction branch parameters;

[0074] S502, performing model training on the visible light feature extraction branch structure to obtain visible light feature extraction branch parameters;

[0075] S503, performing model training on the infrared light feature extraction branch structure to obtain infrared light feature extraction branch parameters;

[0076] Among them, the convolution kernel size of the common feature extraction branch structure is larger than the convolution kernel size of the infrared light feature extraction branch structure, and the convolution kernel size of the infrared light feature extraction branch structure is larger than the convolution kernel size of the visible light feature extraction branch structure.

[0077] In this embodiment, when model training is performed on the common feature extraction branch structure, the visible light feature extraction branch structure, and the infrared light feature extraction branch structure, the common feature extraction branch structure is first trained. After training is completed, the common feature extraction branch parameters are fixed. Then, model training is performed on the visible light feature extraction branch structure and the infrared light feature extraction branch structure separately to obtain model training parameters for the three branch structures.

[0078] The convolution kernel size of the common feature extraction branch structure is larger than that of the infrared light feature extraction branch structure, and the convolution kernel size of the infrared light feature extraction branch structure is larger than that of the visible light feature extraction branch structure. By setting different convolution kernel sizes among the three feature extraction branch structures and showing a gradually decreasing trend, the receptive field of the multimodal common features, visible light features and infrared light features during extraction can be increased, and the extraction of features such as texture can be strengthened. At the same time, it can also reduce the amount of parameters calculated in the overall visual tracking process, save multimodal visual tracking time, and thus improve the accuracy and tracking efficiency of the multimodal visual tracking method.

[0079] See Figure 2 The present invention provides a multimodal visual tracking system 100 based on a surveillance scene. The multimodal visual tracking system 100 based on a surveillance scene includes a data acquisition module 10, a shared feature extraction module 20, a visible light feature extraction module 30, an infrared feature extraction module 40, a feature fusion module 50, and a visual tracking module 60. The data acquisition module 10 is used to acquire a visible light image dataset and an infrared image dataset. The shared feature extraction module 20 is used to input the visible light image dataset and the infrared image dataset into a shared feature extraction branch structure for feature extraction to obtain multimodal shared features.

[0080] The visible light feature extraction module 30 is used to input the visible light image dataset into the visible light feature extraction branch structure for feature extraction, thereby obtaining unique visible light features. The infrared light feature extraction module 40 is used to input the infrared light image dataset into the infrared light feature extraction branch structure for feature extraction, thereby obtaining unique infrared light features. The feature fusion module 50 is used to fuse multimodal common features, unique visible light features, and unique infrared light features to obtain multimodal fusion features. The visual tracking module 60 is used to visually track multiple targets based on the multimodal fusion features.

[0081] In this embodiment, the description of the data acquisition module 10 can refer to the description of S10 in the above embodiment. The description of the common feature extraction module 20 can refer to the description of S20 in the above embodiment. The description of the visible light feature extraction module 30 can refer to the description of S30 in the above embodiment. The description of the infrared light feature extraction module 40 can refer to the description of S40 in the above embodiment. The description of the feature fusion module 50 can refer to the description of S50 in the above embodiment. The description of the visual tracking module 60 can refer to the description of S60 in the above embodiment.

[0082] In one embodiment, the visual tracking module 60 comprises a target information acquisition module and a same target matching module. The target information acquisition module is configured to perform target tracking on the multi-modal fusion features according to a target tracking network, and obtain target categories and target positions of each target in each frame of image. The same target matching module is configured to take the position intersection ratio between targets in adjacent frame images as an adjacent frame target relationship matching index, track the target positions of each target in each frame of image according to an online tracking algorithm, and obtain a same target matching relationship in adjacent frame images.

[0083] In this embodiment, the description of the target information acquisition module can refer to the description of S610 in the above embodiment. The description of the same target matching module can refer to the description of S620 in the above embodiment.

[0084] In one embodiment, the visual tracking module 60 further comprises a continuous target trajectory acquisition module. The continuous target trajectory acquisition module is configured to perform predictive tracking on the target positions of each target in continuous frame images according to a Kalman filtering algorithm, and obtain a continuous target position change trajectory.

[0085] In this embodiment, the description of the continuous target trajectory acquisition module can refer to the description of S630 in the above embodiment.

[0086] In one embodiment, the visual tracking module further comprises a multi-target association module. The multi-target association module is configured to perform target association on the target positions of each target in each frame of image according to a Hungarian algorithm, and obtain a multi-target association relationship in adjacent frame images.

[0087] In this embodiment, the description of the multi-target association module can refer to the description of S640 in the above embodiment.

[0088] In one embodiment, the multimodal visual tracking system 100 based on the monitoring scene further includes a common feature branch training module, a visible light feature branch training module, and an infrared light feature branch training module. The common feature branch training module is used to perform model training on the common feature extraction branch structure, obtain common feature extraction branch parameters, and fix the common feature extraction branch parameters. The visible light feature branch training module is used to perform model training on the visible light feature extraction branch structure, obtain visible light feature extraction branch parameters. The infrared light feature branch training module is used to perform model training on the infrared light feature extraction branch structure, obtain infrared light feature extraction branch parameters. Among them, the convolution kernel size of the common feature extraction branch structure is larger than the convolution kernel size of the infrared light feature extraction branch structure, and the convolution kernel size of the infrared light feature extraction branch structure is larger than the convolution kernel size of the visible light feature extraction branch structure.

[0089] In this embodiment, the description of the common feature branch training module can refer to the description of S501 in the above embodiment. The description of the visible light feature branch training module can refer to the description of S502 in the above embodiment. The description of the infrared light feature branch training module can refer to the description of S503 in the above embodiment.

[0090] In the various embodiments described above, the specific order or hierarchy of steps in the disclosed processes is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process can be rearranged without departing from the scope of protection of this disclosure. The accompanying method claims provide elements of the various steps in an exemplary order and are not intended to be limited to a specific order or hierarchy.

[0091] Those skilled in the art will also appreciate that the various illustrative logical blocks, modules, and steps listed in the embodiments of the present invention can be implemented by electronic hardware, computer software, or a combination of the two. To clearly demonstrate the interchangeability of hardware and software, the various illustrative components, modules, and steps described above have generally described their functions. Whether such functions are implemented by hardware or software depends on the specific application and the design requirements of the entire system. Those skilled in the art may use various methods to implement the described functions for each specific application, but such implementation should not be understood as exceeding the scope of protection of the embodiments of the present invention.

[0092] The various illustrative logical blocks, modules, and circuits described in connection with the embodiments disclosed herein can be implemented or performed with a general purpose processor, a digital signal processor, an application specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor can be a microprocessor, but, in the alternative, the general purpose processor can be any conventional processor, controller, microcontroller, or state machine. A processor can also be implemented as a combination of computing devices, e.g., a combination of a digital signal processor and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a digital signal processor core, or any other such configuration.

[0093] The steps of a method or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is tangible. A storage medium can be coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium can be integral to the processor. The processor and the storage medium can reside in an ASIC. The ASIC can reside in a user terminal. In the alternative, the processor and the storage medium can reside in different devices in a user terminal.

[0094] The specific implementation described above is illustrative for purposes of teaching the present application. The application should not be limited, however, to one or more specific shown implementations, one of ordinary skill in the art will recognize numerous other implementations as being within the scope of the application. Each of the conceptual aspects of the application as described above is intended to be combinable with other aspects of the application.

Claims

1. A multimodal visual tracking method based on a monitoring scene, characterized in that: include: Obtain visible light image datasets and infrared image datasets; Inputting the visible light image dataset and the infrared light image dataset into a common feature extraction branch structure to perform feature extraction to obtain multimodal common features; Inputting the visible light image data set into a visible light feature extraction branch structure to perform feature extraction to obtain unique features of visible light; Inputting the infrared image data set into the infrared feature extraction branch structure to perform feature extraction to obtain unique infrared features; Performing feature fusion on the multimodal common features, the visible light unique features, and the infrared light unique features to obtain a multimodal fusion feature; Performing visual tracking on multiple targets based on the multimodal fusion features; After fusing the multimodal common features, the visible light unique features, and the infrared light unique features to obtain a multimodal fusion feature, and before visually tracking multiple targets based on the multimodal fusion feature, the method further includes: Performing model training on the common feature extraction branch structure to obtain common feature extraction branch parameters, and fixing the common feature extraction branch parameters; Performing model training on the visible light feature extraction branch structure to obtain visible light feature extraction branch parameters; Performing model training on the infrared light feature extraction branch structure to obtain infrared light feature extraction branch parameters; Among them, the convolution kernel size of the common feature extraction branch structure is larger than the convolution kernel size of the infrared light feature extraction branch structure, and the convolution kernel size of the infrared light feature extraction branch structure is larger than the convolution kernel size of the visible light feature extraction branch structure.

2. The multimodal visual tracking method based on monitoring scene according to claim 1 is characterized in that: The visual tracking of multiple targets according to the multimodal fusion features includes: Performing target tracking on the multimodal fusion features according to a target tracking network to obtain the target category and target position of each target in each frame image; The intersection-and-union ratio of target positions in adjacent frame images is used as the adjacent frame target relationship matching index. The target position of each target in each frame image is tracked according to the online tracking algorithm to obtain the matching relationship of the same target in adjacent frame images.

3. The multimodal visual tracking method based on monitoring scene according to claim 2 is characterized in that: The visual tracking of multiple targets according to the multimodal fusion features further includes: The target position of each target in the continuous frame images is predicted and tracked according to the Kalman filter algorithm to obtain a coherent target position change trajectory.

4. The multimodal visual tracking method based on monitoring scene according to claim 3 is characterized in that: The visual tracking of multiple targets according to the multimodal fusion features further includes: The target positions of the targets in each frame image are subjected to target association according to the Hungarian algorithm to obtain the multi-target association relationship in adjacent frame images.

5. A multimodal visual tracking system based on monitoring scenes, characterized in that: include: A data acquisition module is used to acquire a visible light image dataset and an infrared light image dataset; a shared feature extraction module, configured to input the visible light image dataset and the infrared light image dataset into a shared feature extraction branch structure to perform feature extraction and obtain multimodal shared features; A visible light feature extraction module is used to input the visible light image data set into the visible light feature extraction branch structure to perform feature extraction and obtain unique features of visible light; an infrared light feature extraction module, configured to input the infrared light image data set into the infrared light feature extraction branch structure to perform feature extraction and obtain unique infrared light features; a feature fusion module, configured to fuse the multimodal common features, the visible light unique features, and the infrared light unique features to obtain a multimodal fusion feature; A visual tracking module, configured to perform visual tracking on multiple targets based on the multimodal fusion features; The system further comprises: A common feature branch training module is used to perform model training on the common feature extraction branch structure, obtain common feature extraction branch parameters, and fix the common feature extraction branch parameters; A visible light feature branch training module is used to perform model training on the visible light feature extraction branch structure to obtain visible light feature extraction branch parameters; An infrared light feature branch training module is used to perform model training on the infrared light feature extraction branch structure to obtain infrared light feature extraction branch parameters; Among them, the convolution kernel size of the common feature extraction branch structure is larger than the convolution kernel size of the infrared light feature extraction branch structure, and the convolution kernel size of the infrared light feature extraction branch structure is larger than the convolution kernel size of the visible light feature extraction branch structure.

6. The multimodal visual tracking system based on monitoring scene according to claim 5, characterized in that: The visual tracking module includes: A target information acquisition module is used to track the multimodal fusion features according to the target tracking network to obtain the target category and target position of each target in each frame image; The same-target matching module is used to use the intersection-and-union ratio of the positions of the targets in adjacent frame images as the matching index of the target relationship between adjacent frames, track the target position of each target in each frame image according to the online tracking algorithm, and obtain the same-target matching relationship in adjacent frame images.

7. The multimodal visual tracking system based on monitoring scenes according to claim 6, characterized in that: The visual tracking module also includes: The coherent target trajectory acquisition module is used to predict and track the target position of each target in the continuous frame image according to the Kalman filter algorithm to obtain a coherent target position change trajectory.

8. The multimodal visual tracking system based on monitoring scenes according to claim 7, characterized in that: The visual tracking module also includes: The multi-target association module is used to perform target association on the target positions of each target in each frame image according to the Hungarian algorithm to obtain the multi-target association relationship in adjacent frame images.

Images