An optical autonomous cooperative detection and identification method based on multi-modal data

Abstract

The application provides an optical autonomous cooperative detection and identification method based on multi-modal data, and belongs to the technical field of machine vision. The advantages of visible light, infrared and laser sensors are comprehensively utilized to overcome the limitations of single sensors in adverse weather, complex background or strong light interference. Two independent algorithm processes are used for synchronous visible light search and infrared search of visible light and infrared images to ensure that even if one sensor fails or performance decreases due to environmental conditions, the other sensor can still serve as an effective supplement, greatly improving the first capture probability and overall reliability of the system for the target. A slice-assisted super-reasoning YOLOv5 algorithm based on a sliding window is used to process the visible light image, effectively improving the detection accuracy of distant small targets or dense targets and overcoming the performance bottleneck of traditional methods when the resolution is insufficient. The infrared image is divided into weather and environment types, and the corresponding algorithms in the prior algorithm library are dynamically called.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of machine vision, and particularly relates to an optical autonomous cooperative detection and identification method based on multi-modal data. BACKGROUND

[0002] Optical detection equipment has obvious shortcomings in various aspects of performance. First, the detection distance of different optical equipment is limited, such as the detection distance of laser radar is less than 300 meters, which is difficult to meet the demand of long-distance detection. Second, the sensitivity of optical equipment is affected by various factors, for example, the dark current and noise of photomultiplier tube can limit its detection ability of low-intensity signals. In addition, the search range of optical equipment is also limited, the detection angle of some equipment is small, such as the detection angle of millimeter wave radar is only 10 ° -70 ° , which is difficult to cover a wide area. In complex background and bad weather conditions, the performance of optical equipment will be greatly reduced, for example, infrared detection equipment has errors in atmospheric transmission correction, and lacks target radiation true value as feedback of measurement accuracy, which makes it difficult to guarantee the detection accuracy in complex environment.

[0003] The intelligent identification ability of optical detection equipment also has many shortcomings. For target identification in complex scenes, optical equipment often has difficulty in dealing with background interference, target shape change and light change. In addition, optical equipment also has defects in target imaging characteristics, for example, the current target infrared radiation characteristic measurement in the target range can only obtain the inverted target radiation characteristic and gray change information, but lacks the prior information of target radiation characteristic change in the flight process, which cannot accurately identify the target. In terms of multi-target detection and tracking, the autonomous ability of optical equipment is insufficient, and it is difficult to realize cooperative work, and cannot effectively handle the complex situation of multiple targets. SUMMARY

[0004] The purpose of the present application is to overcome the above-mentioned shortcomings, and provide an optical autonomous cooperative detection and identification method based on multi-modal data. Based on deep learning algorithm, through real-time image data stream transmission, multiple target types including fixed-wing unmanned aerial vehicle and rotary-wing unmanned aerial vehicle are automatically detected and identified. Based on data in three different modalities of visible light, infrared and laser, through integrating various advanced algorithms and technologies, efficient detection and identification tasks can be effectively and accurately performed in complex and variable scenes.

[0005] In order to achieve the above-mentioned purpose, the technical scheme adopted by the present application is as follows:

[0006] The present application provides an optical autonomous cooperative detection and identification method based on multi-modal data, comprising the following steps:

[0007] Start the scanning and searching process, search and capture the target in the scene to be measured through the visible light sensor and the infrared sensor, and collect visible light image data and infrared image data;

[0008] For visible light image data, a YOLOv5 algorithm based on sliding window slicing assisted super reasoning is used to search for targets in the visible light image to obtain a visible light search result; visible light image data and infrared image data are used for collaborative analysis for infrared search, weather type and environment type are divided, and a prior algorithm library is used for target detection to obtain an infrared search result;

[0009] Based on the visible light search result and the infrared search result, it is determined whether the target is captured, if yes, the subsequent steps are executed, otherwise, the scanning and searching process is re-performed;

[0010] If the target is captured, it is determined whether the visible light search result and the infrared search result are the same target, if yes, the target is fused; otherwise, based on the scene where the different targets are located, an infrared detection algorithm is used to track the targets captured by the visible light search result and the infrared search result respectively;

[0011] After the target tracking is completed, the target distance information is obtained through the laser sensor and the target is accurately identified;

[0012] The prior algorithm library includes a deep learning model and a detection algorithm, and different prior algorithms are selected according to different environment type identification results and scene prior information.

[0013] In the step of starting the scanning and searching process, searching and capturing the target in the scene to be measured through the visible light sensor and the infrared sensor, and collecting visible light image data and infrared image data, a multi-frame confirmation strategy is used to capture image data of the same area at different time points, and whether there is a real target is confirmed by comparing the multi-frame data.

[0014] In the step of using visible light image data and infrared image data for collaborative analysis for infrared search, dividing weather type and environment type, and combining the prior algorithm library for target detection to obtain an infrared search result, the method of using visible light image data and infrared image data for collaborative analysis for infrared search is as follows:

[0015] Based on the obtained visible light image data, a weather identification and classification algorithm of MobileNetv3 is used to identify the weather type, which is divided into three categories of sunny, cloudy and night;

[0016] Based on the obtained infrared image data, a semantic segmentation algorithm of an I-U-Net model is used to segment the infrared image data, and the environment type is identified based on the segmented infrared image data;

[0017] Based on the identified weather type and environment type, the YOLOv8 algorithm framework is used to build a deep learning algorithm combined with a priori algorithm library for infrared search, and an infrared search result is obtained.

[0018] The deep learning model includes YOLOv5, and the detection algorithm includes ADDGD and ADMD.

[0019] Based on the obtained infrared image data, the I-U-Net model is used for semantic segmentation algorithm to segment the infrared image data, and the environment type is identified based on the segmented infrared image data, and the specific method is as follows:

[0020] The obtained infrared image data is input into the shrinkage path of the I-U-Net model, the image size is reduced by downsampling operation, the shallow feature information of the image is extracted, and the feature map is obtained.

[0021] The extracted feature map is input into the expansion path, the resolution of the feature map is increased by upsampling operation, and the deep feature information of the image is extracted.

[0022] The shallow feature information and the deep feature information are fused by using the jump connection structure to form the fused feature information.

[0023] The fused feature information is subjected to a binary classification operation to obtain an output image, the output image is divided into a background region and a target region, and a background detection result is obtained.

[0024] Based on the background detection result, the proportion of different reference objects in the whole image pixels is calculated, and the environment type is determined based on the background corresponding to the reference object with the largest proportion.

[0025] Based on the identified weather type and environment type, the YOLOv8 algorithm framework is used to build a deep learning algorithm combined with a priori algorithm library for infrared search, and an infrared search result is obtained, and the specific method is as follows:

[0026] A detection network containing an improved Neck part is constructed, and based on the constructed detection network, the infrared image data for determining the weather type and the environment type is subjected to target detection in combination with a priori algorithm library to obtain a preliminary detection result.

[0027] Based on the scene prior information, the preliminary detection result is subjected to false alarm elimination, the coincidence degree of the preliminary detection position of the target and the prior region is calculated through the prior position of the target appearing region, the target with a coincidence degree lower than a preset coincidence degree is regarded as a false alarm, and an infrared search result is obtained.

[0028] The detection network of the improved Neck part is composed of a feature pyramid network and a path aggregation network, and a detection module is used to replace the original cross-stage partial bottleneck network.

[0029] In the improved Neck part detection network, 4 times down-sampling feature map output is added to form a multi-scale detection structure containing 4 detection heads for outputting higher fine-grained feature maps to retain infrared small target features and generating preliminary detection results.

[0030] If the visible light search result and the infrared search result are the same target, the target is fused by using a heterogenous sensor coordinate mapping method based on deep learning.

[0031] If the visible light search result and the infrared search result are the same target, the target is fused by using a heterogenous sensor coordinate mapping method based on deep learning.

[0032] If the visible light search result and the infrared search result are not the same target, a candidate target set is constructed based on all captured targets.

[0033] The targets in the candidate target set and the fused target are accurately identified and tracked by using an infrared detection algorithm.

[0034] The targets in the candidate target set and the fused target are accurately identified and tracked by using an infrared detection algorithm.

[0035] The target scene is intelligently analyzed by using an infrared detection algorithm to determine the number of targets existing in the target scene.

[0036] If multiple targets are determined to exist, a target tracking algorithm combining a re-detection mechanism and nanoscale tracking technology is used to track the multiple targets in the image, and each target is assigned a unique ID for identification.

[0037] If a single target is determined to exist, an adaptive pipeline filtering algorithm is used to track the target.

[0038] When a target has a tracking breakpoint due to occlusion, environmental changes or detection noise, the re-detection mechanism will re-detect the target to find the lost target through an iterative process.

[0039] In both the multiple target and single target cases, a gate tracking algorithm is used to track the corresponding target. In the multiple target case, the user selects a target with a specified ID for tracking. In the single target case, the user issues a tracking instruction to track the single target.

[0040] Compared with the prior art, the present application has the following beneficial effects:

[0041] The application provides an optical autonomous cooperative detection and identification method based on multi-modal data, which comprehensively utilizes the advantages of visible light, infrared and laser sensors, and overcomes the limitations of single sensors in severe weather, complex background or strong light interference. The initialization process ensures that each data stream is synchronized and ready, laying a foundation for subsequent efficient processing; two independent algorithm processes are used for synchronous visible light search and infrared search, which ensures that even if one sensor fails or performance decreases due to environmental conditions, the other sensor can still serve as an effective supplement, greatly improving the first capture probability and overall reliability of the system for the target; the sliding window-based slicing auxiliary super reasoning YOLOv5 algorithm is used to process visible light images, effectively improving the detection accuracy of small targets or dense targets at a distance, and overcoming the performance bottleneck of traditional methods when the resolution is insufficient; visible light image data and infrared image data are used for cooperative analysis for infrared search, weather type and environment type are divided, and specific algorithms in the prior algorithm library are dynamically called. Compared with a single general algorithm, this adaptive detection strategy based on scene understanding can more fully utilize the infrared physical characteristics, significantly improving the detection accuracy and anti-false alarm capability in different thermal scenes. Through the organic combination of multi-source sensor cooperation, parallel processing, intelligent decision and adaptive tracking, the application ultimately achieves continuous, stable and accurate perception and monitoring of the target in all-weather and complex environments, effectively solving the problems of missed detection, false alarm and environmental interference of traditional single sensor systems, and greatly improving the intelligent level and task execution efficiency of the overall system.

[0042] Further, the visible light search results and the infrared search results are associated and analyzed. If it is determined that they are the same target, fusion processing is performed to generate more comprehensive and reliable target information than any single source. If not, based on the specific scene where the different targets are located, an infrared detection algorithm is used to ensure that even if immediate association is not possible, suspicious targets can be continuously monitored to avoid losing the target and create conditions for subsequent re-identification. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 The figure is a schematic diagram of the method of the application;

[0044] Figure 2 The figure is a hybrid intelligent target cooperative detection and tracking processing algorithm framework of the application;

[0045] Figure 3 The figure is a schematic diagram of the image collected by the system of the application;

[0046] Figure 4 The figure is the overall architecture of the I-U-Net model in the application;

[0047] Figure 5 The figure is a schematic diagram of the environment recognition result of the application;

[0048] Figure 6 Figure 1 is a schematic diagram of the overall structure of the Neck based on YOLOv8 optimization in the present application;

[0049] Figure 7 Figure 2 is a schematic diagram of the target recognition result in the present application. DETAILED DESCRIPTION

[0050] In order to further understand the content of the present application, the present application will be described in detail below in combination with the drawings and specific embodiments. It should be understood that the embodiments are only used to explain the present application and are not limiting.

[0051] As shown in Figures 1-2 Figure 1, a multi-modal data-based optical autonomous cooperative detection and recognition method, comprising the following steps:

[0052] S1: starting the scanning and searching process, searching and capturing the target in the scene to be measured through the visible light sensor and the infrared sensor, collecting visible light image data and infrared image data;

[0053] S2: for the visible light image data, using the sliding window-based slice-assisted super-inference YOLOv5 algorithm to search for the target in the visible light image, obtaining the visible light search result; using the visible light image data and the infrared image data to cooperatively analyze and search for the infrared, dividing the weather type and the environment type, and detecting the target through the prior algorithm library to obtain the infrared search result;

[0054] S3: determining whether the target is captured based on the visible light search result and the infrared search result, if yes, executing the subsequent steps, otherwise, re-performing the scanning and searching process;

[0055] S4: if the target is captured, determining whether the visible light search result and the infrared search result are the same target, if yes, performing fusion processing on the target; otherwise, based on the specific scene where the different targets are located, using the infrared detection algorithm to track the targets captured by the visible light search result and the infrared search result respectively;

[0056] S5: after the target tracking is completed, obtaining the target distance information through the laser sensor and accurately identifying the target.

[0057] Specifically, in S1, in order to improve the efficiency of data processing, the data stream initialization module prepares and configures data stream processing when starting, receives raw data from the visible light sensor, the infrared sensor and the laser sensor, and performs analysis and format conversion on the raw data to ensure the consistency of the data. After initialization, the scanning and searching process is started, the target in the scene to be measured is searched and captured through the visible light sensor and the infrared sensor, and the visible light image data and the infrared image data are collected.

[0058] The initialization is divided into three parts: device initialization, data flow initialization and model warm-up. Device initialization configures the hardware and software environment to ensure that it is ready to work before starting. Data flow initialization is responsible for time synchronization, data correction (such as denoising and environmental compensation), data storage and rate control of data flow to ensure stability under high load. At the same time, monitor the abnormal situation in the data flow to ensure the accuracy and reliability of the whole data processing process. Provide a solid foundation for subsequent data processing and analysis. Model warm-up avoids the delay problem of cold start by preloading relevant data, weights or other parameters.

[0059] When receiving the search instruction, start the scanning and searching process. The visible light sensor and the infrared sensor first perform horizontal scanning, i.e. moving from left to right or from right to left in the horizontal direction, which is suitable for scenes such as plains or open and spacious areas. Subsequently, the visible light sensor and the infrared sensor perform vertical scanning through the pitch maneuver to capture targets in different height ranges, which is suitable for scenes such as mountainous areas or building groups with large terrain undulations. An example of the image acquisition result is shown in Figure 3

[0060] Further, in order to improve the reliability of target capture, a multi-frame confirmation strategy is adopted during horizontal scanning and pitch maneuver. Image data of the same area is captured multiple times at different time points, and by comparing multiple frames of data, it is confirmed whether there is a real target, thereby effectively filtering out false positives caused by noise, light changes or accidental factors. When the search and capture of the target are completed, the scanning and searching process is ended.

[0061] Specifically, in S2, the visible light sensor is mainly used for daytime or well-lit scenes, while the infrared sensor can cope with night or low-light conditions. For the visible light search part, the YOLOv5 algorithm (You Only Look Once version 5, target detection algorithm) based on slicing aided hyper inference (SAHI) is used to search for targets in visible light images, obtaining visible light search results.

[0062] For the infrared search part, due to the influence of complex time and weather conditions on image characteristics, such as different light conditions under different weather conditions, the brightness and contrast information of targets and backgrounds in the image are different, which affects the performance of detection and recognition algorithms. Therefore, in the infrared search part, the scene in the image is deeply understood through the scene perception stage to divide the weather type and environment type.

[0063] The scene perception stage is composed of a weather perception module and an environment perception module. ​

[0064] 1) The weather perception module first receives visible light image data, determines the time and weather conditions of the current scene, including sunny, cloudy and night, through a weather identification and classification algorithm based on MobileNetv3 (Mobile Neural Network version 3), thereby providing basic feature information of the current scene to be measured. Based on the feature information of the scene to be measured provided by the weather perception module, the processing strategy can be dynamically adjusted to ensure stable and accurate detection and identification of the target even in adverse weather conditions.

[0065] 2) The environment perception module refines the scene through image segmentation algorithms and environment determination modules. The environment perception module uses a semantic segmentation algorithm based on the I-U-Net model to segment the infrared image data, obtaining a segmented image as input to the environment perception module. The environment perception module determines which typical environment the current image belongs to by calculating the proportion of typical objects in each environment in the segmented image, including plains, suburbs and mountains, corresponding to flat ground, buildings and mountains.

[0066] As shown in Figure 4 , the I-U-Net model (Iterative-U-shaped Network) is used for background type recognition. The I-U-Net model is an improved fully convolutional network model. It uses an encoder-decoder structure, where the encoder is responsible for feature extraction, and the decoder restores the size of the obtained features. The encoder contains 5 feature extraction modules and 4 pooling operations. Each encoder consists of two 3x3 convolution layers. The decoder consists of a deconvolution, a concatenation operation and two convolution layers. The number of channels is exactly the opposite of the number of encoders. To compress the feature map, the I-U-Net model uses a pooling operation for downsampling. Downsampling reduces the size of the features and reduces the complexity of the network. In contrast to the pooling operation, deconvolution aims to enlarge the features, while the concatenation operation is used to obtain more initial features.

[0067] The specific method of segmenting infrared image data and identifying environment types using the I-U-Net model is as follows:

[0068] During the training process of the I-U-Net model, the acquired infrared image data is first reduced using the downsampling in the shrinkage path of the I-U-Net model, thereby selecting features with a larger activation degree, ignoring features with a relatively small effect and redundant information, to extract shallow feature information of the image and obtain a feature map;

[0069] The expansion path then enlarges the extracted feature map to increase the resolution of the feature map, thereby obtaining deep feature information of the image.

[0070] The extracted feature information is fused by using a skip connection structure in a convolutional neural network model, i.e. connecting the shallow feature information and the deep feature information of the image to form fused feature information, thereby improving the detection and segmentation precision and realizing accurate positioning of the target;

[0071] The fused feature information is subjected to a binary classification operation to obtain an output image, the output image is divided into a background region and a target region, and a background detection result is obtained;

[0072] Based on the background detection result, the proportion of different reference objects in the whole image is calculated, and the environment type is determined based on the background corresponding to the reference object with the largest proportion.

[0073] As preferred in this embodiment, Table 1 shows the evaluation index results of environment recognition of mountainous, plain and suburban scenes. Among them, the average recognition probability of the three environment types of mountainous, plain and suburban is 96.76%, and the average recognition accuracy is 95.90%. The recognition result is shown in Figure 5 , Figure 5 A1 and A2 are suburban scenes, B1 and B2 are plain scenes, and C1 and C2 are mountainous scenes.

[0074] Table 1 Environment type recognition result

[0075]

[0076] 3) Based on the recognized weather type and environment type, the target is detected by using a priori algorithm library.

[0077] Firstly, in target detection, the application adopts the YOLOv8 algorithm (You Only Look Once version 8, target detection algorithm) framework to construct a deep learning algorithm combined with a prior algorithm library for target detection. Based on the constructed detection network combined with the prior algorithm library, the infrared image data of the determined weather type and environment type is detected to obtain a preliminary detection result. Different from the YOLOv8 algorithm, the network of the Neck (neck network) part is refined. In the original YOLOv8, the network of the Neck part is constructed and fused in the HEAD (head) module. In order to improve the detection ability of the algorithm for small infrared targets, the feature map of 4 times down-sampling is added, that is, when the input image is 416x416, the corresponding feature map size is 104x104, and the strategy of generating three AnchorBox (anchor box) in each grid is still adopted. Referring to the existing FPN (Feature Pyramid Network, feature pyramid network) and PAN (Path Aggregation Network, path aggregation network) structure, the Neck part is divided into FPN and PAN two parts, as shown in Figure 6 The original CSP Bottle Net (Cross-Stage Partial Bottleneck, cross-stage partial bottleneck network) is replaced by a more efficient Detection Block (detection module), which greatly reduces the model size without affecting the model accuracy. After the improved Neck part fusion, the expected 4 detection head detection structure is finally obtained. The detection head can output a higher granularity feature map, accurately retain the small target feature information on the infrared image data, and obtain a preliminary detection result.

[0078] Specifically,

[0079] The FPN starts from the deepest feature map P5, and after the Upsample (up-sampling) operation, it is enlarged to the same size as P4 and gradually up-sampled. The up-sampled P5 and P4 from the backbone network are Concat (depth dimension splicing), and the spliced features are fused by the Detection Block (detection module). The above operation is repeated to generate feature maps P3 and P2 for detecting smaller objects in turn.

[0080] The input of the PAN directly comes from the feature map output by the corresponding node of the FPN path. Taking the P2 feature map of the FPN as the starting point, it is first down-sampled by a Conv (convolution with a step of 2) to make its size the same as P3. The down-sampled feature and the P3 feature from the FPN path are Concat, and the spliced feature is also fused by the Detection Block. The above operation is repeated to enhance the features from bottom to top.

[0081] After the bidirectional fusion of FPN and PAN, the network obtains four enhanced feature maps of different scales, which are rich in strong semantic information and accurate positioning information. Each scale of feature map is sent to an independent detection head (Head1, Head2, Head3 and Head4).

[0082] Secondly, false alarms are removed based on the scene prior information. For example, based on the prior information that targets usually appear above the skyline, the detected abnormal position targets are considered as false alarms and removed based on the Segment Anythin algorithm. In addition, in mountainous scenes, the algorithm will exclude false detection of non-skyline positions according to the characteristics of the terrain. In plain scenes, the algorithm will use the flatness of the background to filter out abnormal signals far from the target area. Through detailed modeling of the prior information of different scene types, the accuracy of detection is improved, and the number of false alarms is greatly reduced, making the detection result more accurate. The target selection is realized by the coincidence degree of the preliminary detection position of the target and the prior region. Let the preliminary detection position of the target be , , the index of the prior position region , the coincidence degree of the target , and the specific definition is as follows:

[0083]

[0084] The targets below the preset coincidence degree are removed as false alarms to obtain the infrared search result.

[0085] Preferably, suitable methods in the prior algorithm library can be flexibly called according to the infrared environment type identification result and the scene prior information. The prior algorithm library includes deep learning models such as YOLOv5 and traditional detection algorithms such as ADDGD (AutoData Driven Gaussian Diffusion), ADMD (Absolute Directional Mean Difference Algorithm), AAGD (Asymmetry-Aware Gradient Descent) and the like to cope with scene requirements of different background complexities. For example, for simple scenes such as pure sky, the algorithm preferentially selects traditional detection algorithms for detection, and the calculation overhead of low-altitude algorithms; while in flat plain environment, deep learning models are preferentially selected to improve detection speed and accuracy. Through such dynamic selection, optimal target detection effect can be achieved in various scenes. The infrared search result is obtained by combining the detection results of multiple algorithms. After filtering by the previous scene prior knowledge, false alarm targets that do not meet the expected position are eliminated. This module not only collects the advantages of different algorithms, but also further reduces false detection through intelligent fusion in the post-processing stage, providing more accurate input for the subsequent tracking module.

[0086] Specifically, in S3, it is determined whether the target is captured based on the visible light search result and the infrared search result. If yes, S4 is performed, otherwise, S1 is returned to re-perform the scanning and searching process.

[0087] Specifically, in S4, if the target is captured, it is determined whether the visible light search result and the infrared search result are the same target. If yes, the target is fused; otherwise, based on the scene where the different targets are located, the infrared detection algorithm is used to track the targets captured by the visible light search result and the infrared search result, respectively. Specifically as follows:

[0088] 1) If the visible light search result and the infrared search result are the same target, the specific method of fusing the target is as follows:

[0089] Due to the different working principles and imaging characteristics of visible light sensors and infrared sensors, there are large differences in resolution, field of view, and imaging details between visible light and infrared light, which cannot be directly fused. To overcome this challenge, a heterogeneous sensor coordinate mapping method based on deep learning is adopted. The target pixel coordinates of the infrared image data, the zoom parameters of the visible light camera, and the distance from the center of the infrared sensor to the target are used to register the visible light image data and the infrared image data. Specifically, based on a supervised deep learning network, the relative position, angle, and scale difference between the visible light image data and the infrared image data are automatically analyzed and calculated, and an accurate mapping relationship is generated to precisely align the visible light image data and the infrared image data, ensuring that the fusion step can be processed in the same coordinate system.

[0090] 2) If not the same target, build a candidate target set based on all captured targets (including visible light search results and infrared search results captured targets);

[0091] 3) Use infrared detection algorithms to accurately identify and track targets in the candidate target set and the fused target, with the following specific methods:

[0092] Determine the number of targets and select the appropriate tracking algorithm. To ensure the stability and accuracy of tracking, the system uses the corresponding infrared detection algorithm in a specific scene to determine whether there are multiple targets in the target scene and adaptively adjusts different tracking strategies;

[0093] If there are multiple targets, the system uses a target tracking algorithm that combines a re-detection mechanism and NanoTrack (a lightweight tracking network) to track multiple targets in the image and assigns each target a unique ID for identification. The re-detection mechanism can re-detect targets in the case of occlusion or mutual intersection, and associate them with the original ID through appearance features, thereby solving the problem of ID discontinuity and ID hopping;

[0094] If it is a single target, use an adaptive pipeline filtering algorithm to track the target.

[0095] Further, when the target has a tracking breakpoint due to occlusion, environmental changes, or detection noise, the re-detection mechanism will re-enable the target detection module to detect the target, and through an iterative process, it will find the lost target. This mechanism ensures the continuity of target tracking, and even in a dynamic and complex environment, the algorithm can still maintain stable tracking of low, slow, and small targets.

[0096] Further, whether in the case of multiple targets or single target, the system can track the corresponding target through the wave gate tracking algorithm. For the case of multiple targets, the user needs to select the target with the specified ID, and for the case of single target, the user only needs to issue a tracking instruction.

[0097] The entire process above combines intelligent algorithms and human-computer interaction, which not only improves the accuracy of target tracking, but also flexibly adapts to changing scenes, enabling the system to effectively distinguish and lock each target in a multi-target environment, ensuring efficient and accurate tracking capabilities.

[0098] Specifically, in S5, after target tracking is completed, the target distance information is obtained through the laser sensor and the target is accurately identified. The identification result is as shown in FIG. 6. Figure 7 Figure 7 In the figure, a is an optoelectronic ball, b is a rotary-wing unmanned aerial vehicle, and c is a fixed-wing unmanned aerial vehicle, all of which belong to data acquisition equipment.

[0099] At this stage, it is first determined whether the target is within a given distance threshold. This is because in the case of a long distance, the target only occupies a few pixels in the image, lacking significant appearance and shape feature information, making it difficult to identify the type. By distance threshold determination, the target with a short distance can be screened for accurate identification, avoiding invalid calculation on the target with a long distance, thereby saving computing resources.

[0100] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application and not to limit it. Although the present application has been described in detail with reference to the above examples, those skilled in the art should understand that the specific embodiments of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the present application. Any modification or equivalent replacement without departing from the spirit and scope of the present application should be covered within the protection scope of the claims of the present application.​

Claims

1. An optical autonomous cooperative detection and recognition method based on multi-modal data, characterized in that, The method comprises the following steps: starting a scanning and searching process, searching and capturing a target in a scene to be measured by visible light sensors and infrared sensors, and collecting visible light image data and infrared image data; for the visible light image data, a YOLOv5 algorithm based on a sliding window slicing auxiliary super reasoning is used to search for a target in the visible light image to obtain a visible light search result; infrared search is performed by cooperative analysis of the visible light image data and the infrared image data, weather types and environment types are divided, and a priori algorithm library is combined for target detection to obtain an infrared search result; Specifically: based on the acquired visible light image data, a weather identification and classification algorithm of MobileNetv3 is used to identify the weather type, which is divided into three categories of sunny, cloudy and night; based on the acquired infrared image data, a semantic segmentation algorithm of an I-U-Net model is used to segment the infrared image data, and the environment type is identified based on the segmented infrared image data; based on the identified weather type and environment type, a YOLOv8 algorithm framework is used to construct a deep learning algorithm combined with a priori algorithm library for infrared search to obtain an infrared search result; based on the visible light search result and the infrared search result, it is determined whether the target is captured, if yes, the subsequent steps are executed, otherwise, the scanning and searching process is restarted; if the target is captured, it is determined whether the visible light search result and the infrared search result are the same target, if yes, the target is fused; otherwise, based on the scenes where the different targets are located, infrared detection algorithms are used to track the targets captured by the visible light search result and the infrared search result respectively; after the target tracking is completed, the target distance information is acquired by a laser sensor and the target is accurately identified; the priori algorithm library comprises a deep learning model and a detection algorithm, and different priori algorithms are selected according to different environment type identification results and scene priori information; the deep learning model comprises YOLOv5, and the detection algorithm comprises ADDGD and ADMD.

2. The method of claim 1, wherein, In the step of starting the scanning and searching process, a multi-frame confirmation strategy is used to capture image data of the same area at different time points, and whether there is a real target is confirmed by comparing the multi-frame data.

3. The method of claim 1, wherein, In the step of inputting the acquired infrared image data into the contraction path of the I-U-Net model, the image size is reduced by downsampling operation to extract shallow feature information of the image and obtain a feature map; the extracted feature map is input into the expansion path, the resolution of the feature map is increased by upsampling operation to extract deep feature information of the image; the shallow feature information and the deep feature information are fused by using a skip connection structure to form fused feature information; ​ The fused feature information is subjected to a binary classification operation to obtain an output image, the output image is divided into a background region and a target region, and a background detection result is obtained; Based on the background detection result, the proportion of different reference objects in the whole image pixels is calculated, and the environment type is determined based on the background corresponding to the reference object with the largest proportion.

4. The method of claim 3, wherein, Based on the identified weather type and environment type, a YOLOv8 algorithm framework is used to construct a deep learning algorithm combined with a priori algorithm library for target detection to obtain an infrared search result, and the specific method is as follows: A detection network containing an improved Neck part is constructed, and based on the constructed detection network, a priori algorithm library is used for target detection on infrared image data of the determined weather type and environment type to obtain a preliminary detection result; Based on scene prior information, false alarms in the preliminary detection result are removed, the coincidence degree of the preliminary detection position of the target and the prior region is calculated through the prior position of the target appearing region, the target with a coincidence degree lower than a preset coincidence degree is removed as a false alarm, and an infrared search result is obtained.

5. The method of claim 4, wherein, The detection network with the improved Neck part is composed of a feature pyramid network and a path aggregation network, and a detection module is used to replace the original cross-stage partial bottleneck network; In the improved Neck part detection network, 4 times down-sampling feature map output is added to form a multi-scale detection structure containing 4 detection heads, and the detection heads are used to output higher granularity feature maps to retain infrared small target features to generate a preliminary detection result.

6. The optical autonomous cooperative detection and recognition method based on multi-modal data according to claim 4, characterized in that, If the captured target is determined, it is determined whether the visible light search result and the infrared search result are the same target, and if so, the target is subjected to fusion processing; Otherwise, a candidate target set is constructed based on all captured targets, and an infrared detection algorithm is used to track the targets in the candidate target set and the fused target, and the specific method is as follows: If the visible light search result and the infrared search result are the same target, a deep learning-based heterogenous sensor coordinate mapping method is used to fuse the target; If they are not the same target, a candidate target set is constructed based on all captured targets; An infrared detection algorithm is used to accurately identify and track the targets in the candidate target set and the fused target.

7. The method of claim 6, wherein, The specific method of using the infrared detection algorithm to accurately identify and track the targets in the candidate target set and the fused target is as follows: The target scene is intelligently analyzed by the infrared detection algorithm to determine the number of targets existing in the target scene; If multiple targets are determined to exist, a target tracking algorithm combining a re-detection mechanism and a nanoscale tracking technology is used to track the multiple targets in the image, and each target is assigned a unique ID for identification; If a single target is determined to exist, an adaptive pipeline filtering algorithm is used to track the target.

8. The method of claim 7, wherein, When the target has a tracking breakpoint due to occlusion, environmental changes or detection noise, the re-detection mechanism will re-detect the target through an iterative process to find the lost target.

9. The method of claim 7, wherein, In the case of multiple targets or single target, the gate tracking algorithm is used to track the corresponding target; in the case of multiple targets, the user selects a target with a specified ID to track; in the case of single target, the user issues a tracking instruction to track the single target.

Images