Multi-modal multi-scale small target segmentation network training reasoning method and system
Through the multimodal multi-scale small-object segmentation network training inference method, combined with the registration and expansion processing of infrared and visible images, the eight-neighborhood clustering algorithm is used to solve the problems of low infrared detection accuracy and false target interference in a single frame, and realizes the Accurate detection and precise evaluation of long-distance targets.
Patent Information
- Application Number
- CN202410986044.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-12
AI Technical Summary
Single-frame infrared small target detection technology is difficult to effectively distinguish the moon 380,000 kilometers away, aircraft tens of kilometers away or small drones hundreds of meters away. The target detection accuracy is not high enough and is more disturbed by false targets.
The multimodal multi-scale small-objective segmentation network training inference method is adopted. By registering infrared images and visible light images, the trained segmentation network is input simultaneously, and the prediction mask diagram expansion process is performed, and the target is obtained using the eight-neighborhood clustering algorithm. Information, discard the target with the largest pixel area and improve detection accuracy.
Effectively reduce false target interference, improve target detection accuracy and accuracy of algorithm performance evaluation, and be able to distinguish the moon 380,000 kilometers away from the moon and aircraft tens of kilometers away from the airplane or small drone hundreds of meters away from the airplane.
Smart Images

Figure CN118941525A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of image processing and target detection, and specifically relates to a multi-modal multi-scale small target segmentation network training inference method and system. Background Art
[0002] With the development of deep learning, with the support of public infrared small target datasets, the data-driven infrared small target segmentation network has achieved good detection results. However, limited by the characteristics of infrared imaging principles, it is difficult to effectively distinguish the moon 380,000 kilometers away from an airplane tens of kilometers away or a small drone hundreds of meters away using only single-frame infrared small target detection technology. The target detection accuracy is not high enough. Summary of the invention
[0003] In order to solve the problem of how to effectively distinguish the moon 380,000 kilometers away from an airplane tens of kilometers away or a small drone hundreds of meters away using only single-frame infrared small target detection technology, the present invention proposes a multi-modal and multi-scale small target segmentation network training and inference method and system. This method can reduce the interference of false targets (the moon), improve the accuracy of target detection and the accuracy of algorithm performance evaluation.
[0004] The technical solution adopted by the present invention is as follows:
[0005] A multi-modal and multi-scale small object segmentation network training and reasoning method, comprising:
[0006] Step 1. Construct an infrared multi-scale small target training data set and train the segmentation network. Send the infrared image and visible light image containing small targets to the trained segmentation network for inference at the same time to obtain the corresponding predicted mask map;
[0007] Step 2: dilate the predicted mask map of the infrared image and the predicted mask map of the visible light image to obtain the expanded mask map;
[0008] Step 3. Take the union of the expanded infrared and visible light mask images as the new mask image;
[0009] Step 4. Obtain the pixel sizes of multiple targets on the new mask map through the eight-neighborhood clustering algorithm, and discard the target with the largest pixel area as the final target mask map;
[0010] Step 5. The eight-neighborhood clustering algorithm is further used on the final target mask map to obtain the number of all targets, anchor box information, aspect ratio, center coordinates and pixel size of different targets, and then mapped back to the original infrared image.
[0011] Furthermore, the step 1 specifically includes the following steps:
[0012] Step 1.1: Align the infrared image and the visible light image, and the registration point error is within 5 pixels;
[0013] Step 1.2: Preprocess the infrared multi-scale small target training data set and train the segmentation network;
[0014] Step 1.3: Send the registered infrared image and visible light image to the trained segmentation network model for inference to obtain the predicted mask map of the infrared image and the predicted mask map of the visible light image, which are:
[0015]
[0016]
[0017] In the formula, and They represent the saliency maps after the visible light image and infrared image are fed into the trained network model for prediction; Sigmoid(·) represents the probability map normalized to [0,1]; Th represents the confidence threshold; M VIS and M IR Represent the predicted mask maps of visible light image and infrared image respectively.
[0018] Furthermore, the preprocessing of the infrared multi-scale small target training data set specifically includes:
[0019] Step 1.2.1: Fuse the single-frame infrared small target dataset to construct a new dataset covering targets of different scales and types in multiple scenes;
[0020] Step 1.2.2: Randomly select 50% of the data samples from the new data set constructed in step 1.2.1 to perform grayscale inversion of the image;
[0021] Step 1.2.3: Based on step 1.2.2, flip, rotate, adjust brightness and contrast of the data set;
[0022] Step 1.2.4: Perform channel conversion on each image in the data set processed in step 1.2.3, that is, convert the RGB3 channels into a single-channel grayscale image;
[0023] Step 1.2.5: The images of different scales in the data set processed in step 1.2.4 are resized to a uniform scale using the bilinear interpolation method. The resized scale is consistent with the input requirements of the deep learning segmentation network.
[0024] Step 1.2.6: Normalize each image in the dataset processed in step 1.2.5 and map the pixels of the image to the range of [0,1].
[0025] Furthermore, the confidence threshold Th is set to 0.5.
[0026] Furthermore, when the segmentation network is trained, 50% of the data samples are randomly selected from the data set to perform grayscale inversion of the image.
[0027] Furthermore, the mask image after expansion in step 2 is:
[0028]
[0029]
[0030] In the formula, represents the expansion operation; B is the structural element, using a circular template with a radius of 3; and They are respectively represented as the predicted mask images of the visible light image and the infrared image after the dilation operation.
[0031] Furthermore, the new mask image in step 3 is:
[0032]
[0033] Furthermore, the step 4 obtains the pixel sizes of multiple targets by using the eight-neighborhood clustering algorithm on the new mask map, and discards the target with the largest pixel area as the final target mask map, which specifically includes:
[0034] Step 4.1: Obtain the pixel sizes of multiple targets through the eight-neighborhood clustering algorithm:
[0035]
[0036] In the formula, represents the pixel set of the eight-neighborhood cluster of the i-th target, and Represented on the infrared small target mask M Any two pixels in the area;
[0037] According to formula (6), the pixel size P of different targets i is obtained i :
[0038]
[0039] Step 4.2: Discard the target with the largest pixel area as the final target mask map, which is:
[0040] M' IR+VIS =M IR+VIS (max(P i )=0) (8).
[0041] Furthermore, the number of all targets, anchor box information, aspect ratios, and center coordinates of different targets obtained in step 5 are:
[0042]
[0043]
[0044]
[0045]
[0046] In the formula, N represents the set of all target numbers, B i Represents the anchor box information of different targets i, R i Indicates the aspect ratio, C i represents the center coordinates, represents the number of clusters with a pixel grayscale value of 255 after eight-neighborhood clustering, and {·} represents the set of cluster numbers; represents the coordinates of the upper left extreme point of the i-th target anchor box, Represents the coordinates of the lower right extreme point of the i-th target anchor box.
[0047] A multi-modal multi-scale small target segmentation network training and inference system for implementing the joint detection method includes a data acquisition unit, a network training unit, a deep segmentation network inference prediction unit, a mask map processing unit and a target information extraction unit, wherein:
[0048] The data acquisition unit is used to acquire the registered infrared image and visible light image;
[0049] The network training unit is used to construct an infrared multi-scale small target training data set and train the segmentation network;
[0050] The deep segmentation network inference prediction unit obtains a prediction mask image corresponding to the infrared image and the visible light image through the segmentation network;
[0051] The mask image processing unit is used to perform expansion processing on the predicted mask image of the infrared image and the predicted mask image of the visible light image, take the union of the expanded infrared and visible light mask images as a new mask image, obtain the pixel sizes of multiple targets on the new mask image through the eight-neighborhood clustering algorithm, and discard the target with the largest pixel area as the final target mask image;
[0052] The target information extraction unit is used to further obtain the number of all targets, anchor frame information, aspect ratio, center coordinates and pixel size of different targets on the final target mask map through an eight-neighborhood clustering algorithm, and map it back to the original infrared image.
[0053] Compared with the prior art, the beneficial effects of the present invention are: the present invention respectively segments the mask shape of the target from single-frame images or video stream information in visible light and infrared modal scenes, and expands the predicted mask map of the infrared image and the predicted mask map of the visible light image to solve the problem of target detection failure caused by registration error; the target is effectively segmented and the interference of false targets such as the moon is eliminated. The moon 380,000 kilometers away and the airplane tens of kilometers away or the small drone hundreds of meters away can be effectively distinguished through the joint reasoning detection of "infrared image" and "visible light image", thereby improving the accuracy of target detection and the accuracy of algorithm performance evaluation. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.
[0055] Figure 1 This is a general flow chart of Example 1 of the present invention.
[0056] Figure 2 is the registered infrared image and visible light image. Figure 2 (a) is a schematic diagram of an infrared image. Figure 2 (b) is a schematic diagram of a visible light image.
[0057] Figure 3 These are images of small infrared targets in two modes: black hot and white hot.
[0058] Figure 4 It is the predicted mask map after inference of infrared image and visible light image. Figure 4 (a) is the predicted mask image after infrared image inference. Figure 4 (b) is the predicted mask image after infrared image inference.
[0059] Figure 5 Predict mask maps for infrared and visible light images after inference. Figure 5 (a) is the predicted mask map of the infrared image after inference. Figure 5 (b) is the predicted mask map after dilation of the visible light image.
[0060] Figure 6 Union of dilated predicted masks for infrared and visible images after inference.
[0061] Figure 7 is the final predicted target mask map.
[0062] Figure 8Map the target detection information back to the original infrared image. DETAILED DESCRIPTION
[0063] The following is a detailed description of the embodiments of the present invention. Although the present invention will be described and illustrated in conjunction with some specific embodiments, it should be noted that the present invention is not limited to these embodiments. On the contrary, modifications or equivalent substitutions made to the present invention should all be included in the scope of the claims of the present invention.
[0064] In addition, in order to better illustrate the present invention, numerous specific details are given in the following specific embodiments. It will be understood by those skilled in the art that the present invention can also be implemented without these specific details.
[0065] Aiming at the problem of how to effectively distinguish the moon 380,000 kilometers away from an airplane tens of kilometers away or a small drone hundreds of meters away using only single-frame infrared small target detection technology, the present invention proposes a multi-modal and multi-scale small target segmentation network training and reasoning method and system, such as Figure 1 Specifically including:
[0066] Step 1. Construct an infrared multi-scale small target training dataset and train the segmentation network. Send the infrared image and visible light image containing small targets to the trained segmentation network for inference to obtain the corresponding predicted mask map, which includes:
[0067] Step 1.1: The input infrared image and visible light image are required to be registered images, but an error of 5 pixels is allowed at the registration point, such as Figure 2 As shown;
[0068] Step 1.2: Preprocess the infrared multi-scale small target training data set and train the segmentation network;
[0069] Step 1.3: Send the infrared image and visible light image within the allowable range of registration error to the trained network model for inference to obtain the predicted mask map of the infrared image and the predicted mask map of the visible light image, such as Figure 4 shown.
[0070]
[0071]
[0072] In the formula, and They represent the saliency maps after the visible light image and infrared image are fed into the trained network model for prediction; Sigmoid(·) represents the probability map normalized to [0,1]; Th represents the confidence threshold, which is 0.5; M VIS and M IRRepresent the predicted mask maps of visible light image and infrared image respectively.
[0073] In the above technical solution, the preprocessing of the infrared multi-scale small target training data set in step 1.2 specifically includes:
[0074] Step 1.2.1: Image data fusion. In order to meet the detection tasks of different types of targets (point source targets and small surface source targets, drones, aircraft, ships, etc.) of different scales in multiple scenes (sky, clouds, ground, sea, jungle, buildings) as much as possible, it is necessary to build a rich infrared small target dataset. Therefore, the existing public single-frame infrared small target dataset is fused to build a larger new dataset, so as to cover different types of targets of different scales in multiple scenes as much as possible.
[0075] Step 1.2.2: Image grayscale inversion. In order to meet the target detection tasks in multimodal scenes (infrared white-hot mode, black-hot mode, and visible light scenes) as much as possible, 50% of the data samples (single-frame infrared images containing small targets) are randomly selected from the fusion data set constructed in step 1.1 to perform image grayscale inversion, thereby ensuring that there are enough small target training sets in white-hot mode and small target training sets in black-hot mode. For small targets in visible light scenes, especially small targets in the background of the sky, the imaging effect is very close to that in the black-hot mode, so the small target training set in the black-hot mode can also be called the small target training set in visible light scenes, such as Figure 3 shown.
[0076] Step 1.2.3: Image data enhancement. In order to cover different light intensities, target brightness, background-target contrast and other information as much as possible, the data set is flipped, rotated, brightness adjusted, and contrast adjusted based on step 1.2, so as to further enrich the target feature information and background feature information in the data set, such as Figure 4 shown.
[0077] Step 1.2.4: Image channel conversion. In order to adapt to the infrared small target segmentation network, each image in the data set processed by steps 1.1 to 1.3 needs to be converted into a single-channel grayscale image.
[0078] Step 1.2.5: Image resizing. In order to load the dataset into the network for training, the images of different scales in the dataset need to be resized to a uniform scale using bilinear interpolation. The resized size should be consistent with the input requirements of the network design. Usually the input size of the infrared small target segmentation network is 256×256.
[0079] Step 1.2.6: Image standardization. In order to prevent the influence of outliers in the data set on model training and prevent the problem of gradient disappearance or explosion during model training, each image in the data set processed by steps 1.1 to 1.4 needs to be standardized and the pixels of the image are mapped to the range of [0,1].
[0080] The segmentation network is trained with the pytorch framework and obtained as a model in pth format; then, for engineering deployment, it can be converted into a general onnx intermediate format; finally, an adapted model format can be selected according to the hardware platform used, for example, tflite format, ncnn format, mnn format, etc.
[0081] Step 2: The predicted mask map of the infrared image and the predicted mask map of the visible light image are expanded to obtain the expanded mask map. This is to solve the problem of target detection failure caused by registration error. Therefore, the expansion method is used to enlarge the target pixels on the predicted mask map of the infrared image and the predicted mask map of the visible light image, respectively. Figure 5 shown.
[0082] In the above technical solution, the step 2 specifically includes the following steps:
[0083]
[0084]
[0085] In the formula, ⊕ represents the expansion operation; B is the structural element, using a circular template with a radius of 3; and They are respectively represented as the predicted mask images of the visible light image and the infrared image after the dilation operation.
[0086] Step 3: Take the union of the expanded infrared and visible light mask images as the new mask image, such as Figure 6 shown.
[0087] In the above technical solution, the step 3 specifically includes the following steps:
[0088]
[0089] Step 4: Obtain the pixel sizes of multiple targets through the eight-neighborhood clustering algorithm on the new mask map, and discard the target with the largest pixel area as the final target mask map, such as Figure 7 shown.
[0090] In the above technical solution, step 4 specifically includes the following steps:
[0091] Step 4.1: Obtain the pixel sizes of multiple targets through the eight-neighborhood clustering algorithm
[0092]
[0093] In the formula, represents the pixel set of the eight-neighborhood cluster of the i-th target, and Represented on the infrared small target mask M Any two pixels within the region.
[0094] According to formula (6), the pixel size P of different targets i can be obtained: i :
[0095]
[0096] Step 4.2: Discard the target with the largest pixel area as the final target mask:
[0097] M' IR+VIS =M IR+VIS (max(P i )=0) (8)
[0098] Step 5: On the final target mask map, the eight-neighborhood clustering algorithm is further used to obtain the number of all targets, anchor box information, aspect ratio, center coordinates and pixel size of different targets, and then mapped back to the original infrared image, as shown in Figure 8 shown.
[0099] In the above technical solution, the step 5 specifically includes the following steps:
[0100] In the final target mask map M' IR+VIS We further use formula (6) to obtain the number of all targets N and the anchor box information B of different targets i. i , aspect ratio R i , center coordinates C i and pixel size P i , and mapped back to the original infrared image.
[0101]
[0102] In the formula, represents the number of clusters with a pixel grayscale value of 255 after eight-neighborhood clustering, and {·} represents the set of cluster numbers.
[0103]
[0104]
[0105]
[0106] In the formula, represents the coordinates of the upper left extreme point of the i-th target anchor box, Represents the coordinates of the lower right extreme point of the i-th target anchor box.
[0107] Step 6: In order to evaluate the accuracy of the final target detection, three conditions must be met at the same time to be judged as correctly predicted, namely, condition 1: the Euclidean distance between the centroid of the real target and the centroid of the predicted target is less than the threshold 3; condition 2: the real target pixel overlaps with the predicted target pixel; condition 3: when the Euclidean distance is less than the threshold 3, there are multiple targets, and the minimum value of Hungarian matching combined with the maximum value of pixel-level intersection-over-union ratio is used as the judgment basis.
[0108] In the above technical solution, step 6 specifically includes the following steps:
[0109] Step 6.1: The Euclidean distance between the centroid of the real target and the centroid of the predicted target is less than the threshold 3. The centroids of the real target i and the predicted target j are obtained by formula (12), and the Euclidean distance is calculated.
[0110]
[0111] Where, d E represents the Euclidean distance between the true target mass center and the predicted target mass center; (x i ,y i ) represents the coordinates of the real target i; (x j ,y j ) represents the coordinates of the predicted target j.
[0112] Step 6.2: The real target pixel overlaps with the predicted target pixel. When judging whether the predicted target pixel overlaps with the real target pixel, the pixel-level IoU judgment method is used:
[0113]
[0114] Step 6.3: When the Euclidean distance is less than the threshold 3 and there are multiple targets, the minimum Hungarian matching value combined with the maximum pixel-level intersection-over-union ratio is used as the judgment basis.
[0115] Based on formula (11), the Hungarian matching strategy is introduced:
[0116] Hungarian E =min∑ i ∑ j D i,j X i,j (15)
[0117] Where D i,j Represents the Euclidean distance matrix between the centroid of the real target i and the centroid of the predicted target j. i,jrepresents the optimal allocation matrix, that is, the matching result between the real target i and the predicted target j obtained by the Hungarian matching strategy, that is, Hungariand E .
[0118] At the same time, considering equations (14) and (15), that is, satisfying IoU pixel Under the condition of ≥0, new restrictions are added:
[0119]
[0120] Finally, it is required that equations (14), (15) and (16) are satisfied simultaneously to determine that the target is correctly detected.
[0121]
[0122] The accuracy of the method of the present invention can be verified by the evaluation method based on target centroid Hungarian matching and pixel-level mask matching in step 6.
[0123] The present invention also provides a multi-modal multi-scale small target segmentation network training and inference system for implementing the joint detection method, comprising a data acquisition unit, a network training unit, a deep segmentation network inference prediction unit, a mask map processing unit and a target information extraction unit, wherein:
[0124] The data acquisition unit is used to acquire the registered infrared image and visible light image;
[0125] The network training unit is used to construct an infrared multi-scale small target training data set and train the segmentation network;
[0126] The deep segmentation network inference prediction unit obtains a prediction mask image corresponding to the infrared image and the visible light image through the segmentation network;
[0127] The mask image processing unit is used to perform expansion processing on the predicted mask image of the infrared image and the predicted mask image of the visible light image, take the union of the expanded infrared and visible light mask images as a new mask image, obtain the pixel sizes of multiple targets on the new mask image through the eight-neighborhood clustering algorithm, and discard the target with the largest pixel area as the final target mask image;
[0128] The target information extraction unit is used to further obtain the number of all targets, anchor frame information, aspect ratio, center coordinates and pixel size of different targets on the final target mask map through an eight-neighborhood clustering algorithm, and map it back to the original infrared image.
[0129] This embodiment uses the combined detection method of "infrared image" and "visible light image" to effectively distinguish the moon 380,000 kilometers away and an airplane tens of kilometers away or a small drone hundreds of meters away, and effectively evaluates the detection results, verifying the accuracy of the detection method of the present invention through the evaluation.
[0130] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements of the technical solutions of the embodiments of the present invention do not cause the essence of the corresponding technical solutions to deviate from the scope of this solution.
Claims
1. A multi-modal and multi-scale small object segmentation network training and reasoning method, characterized by: include: Step 1. Construct an infrared multi-scale small target training data set and train the segmentation network. Send the infrared image and visible light image containing small targets to the trained segmentation network for inference at the same time to obtain the corresponding predicted mask map; Step 2: dilate the predicted mask map of the infrared image and the predicted mask map of the visible light image to obtain the expanded mask map; Step 3. Take the union of the expanded infrared and visible light mask images as the new mask image; Step 4. Obtain the pixel sizes of multiple targets on the new mask map through the eight-neighborhood clustering algorithm, and discard the target with the largest pixel area as the final target mask map; Step 5. The eight-neighborhood clustering algorithm is further used on the final target mask map to obtain the number of all targets, anchor box information, aspect ratio, center coordinates and pixel size of different targets, and then mapped back to the original infrared image.
2. According to claim 1, a multi-modal multi-scale small target segmentation network training and reasoning method is characterized by: The step 1 specifically comprises the following steps: Step 1.1: Align the infrared image and the visible light image, and the registration point error is within 5 pixels; Step 1.2: Preprocess the infrared multi-scale small target training data set and train the segmentation network; Step 1.3: Send the registered infrared image and visible light image to the trained segmentation network model for inference to obtain the predicted mask map of the infrared image and the predicted mask map of the visible light image, which are: In the formula, and They represent the saliency maps after the visible light image and infrared image are fed into the trained network model for prediction; Sigmoid(·) represents the probability map normalized to [0,1]; Th represents the confidence threshold; M VIS and M IR Represent the predicted mask maps of visible light image and infrared image respectively.
3. The multi-modal multi-scale small target segmentation network training and reasoning method according to claim 2 is characterized by: The preprocessing of infrared multi-scale small target training data set specifically includes: Step 1.2.1: Fuse the single-frame infrared small target dataset to construct a new dataset covering targets of different scales and types in multiple scenes; Step 1.2.2: Randomly select 50% of the data samples from the new data set constructed in step 1.2.1 to perform grayscale inversion of the image; Step 1.2.3: Based on step 1.2.2, flip, rotate, adjust brightness and contrast of the data set; Step 1.2.4: Perform channel conversion on each image in the data set processed in step 1.2.3, that is, convert the RGB3 channels into a single-channel grayscale image; Step 1.2.5: The images of different scales in the data set processed in step 1.2.4 are resized to a uniform scale using the bilinear interpolation method. The resized scale is consistent with the input requirements of the deep learning segmentation network. Step 1.2.6: Normalize each image in the dataset processed in step 1.2.5 and map the pixels of the image to the range of [0,1].
4. The multi-modal multi-scale small target segmentation network training and reasoning method according to claim 2, characterized in that: The confidence threshold Th is set to 0.
5.
5. The multi-modal multi-scale small target segmentation network training and reasoning method according to claim 2, characterized in that: When the segmentation network is trained, 50% of the data samples are randomly selected from the data set to perform grayscale inversion of the image.
6. The multi-modal multi-scale small target segmentation network training and reasoning method according to claim 2, characterized in that: The mask image after expansion in step 2 is: In the formula, ⊕ represents the expansion operation; B is the structural element, using a circular template with a radius of 3; and They are respectively represented as the predicted mask images of the visible light image and the infrared image after the dilation operation.
7. The multi-modal multi-scale small target segmentation network training and reasoning method according to claim 6 is characterized by: The new mask image in step 3 is:
8. The multi-modal multi-scale small target segmentation network training and reasoning method according to claim 7, characterized in that: The step 4 obtains the pixel sizes of multiple targets by using the eight-neighborhood clustering algorithm on the new mask map, and discards the target with the largest pixel area as the final target mask map, which specifically includes: Step 4.1: Obtain the pixel sizes of multiple targets through the eight-neighborhood clustering algorithm: In the formula, represents the pixel set of the eight-neighborhood cluster of the i-th target, and Represented on the infrared small target mask M Any two pixels in the area; According to formula (6), the pixel size P of different targets i is obtained i : Step 4.2: Discard the target with the largest pixel area as the final target mask map, which is: M' IR+VIS =M IR+VIS (max(P i )=0) (8)。 9. The multi-modal multi-scale small target segmentation network training and reasoning method according to claim 8, characterized in that: The number of all targets, anchor box information, aspect ratios, and center coordinates of different targets obtained in step 5 are: In the formula, N represents the set of all target numbers, B i Represents the anchor box information of different targets i, R i Indicates the aspect ratio, C i represents the center coordinates, represents the number of clusters with a pixel grayscale value of 255 after eight-neighborhood clustering, and {·} represents the set of cluster numbers; represents the coordinates of the upper left extreme point of the i-th target anchor box, Represents the coordinates of the lower right extreme point of the i-th target anchor box.
10. A multi-modal multi-scale small target segmentation network training and reasoning system for implementing the joint detection method according to any one of claims 1 to 9, characterized in that: It includes a data acquisition unit, a network training unit, a deep segmentation network reasoning prediction unit, a mask map processing unit and a target information extraction unit, wherein: The data acquisition unit is used to acquire the registered infrared image and visible light image; The network training unit is used to construct an infrared multi-scale small target training data set and train the segmentation network; The deep segmentation network inference prediction unit obtains a prediction mask image corresponding to the infrared image and the visible light image through the segmentation network; The mask image processing unit is used to perform expansion processing on the predicted mask image of the infrared image and the predicted mask image of the visible light image, take the union of the expanded infrared and visible light mask images as a new mask image, obtain the pixel sizes of multiple targets on the new mask image through the eight-neighborhood clustering algorithm, and discard the target with the largest pixel area as the final target mask image; The target information extraction unit is used to further obtain the number of all targets, anchor frame information, aspect ratio, center coordinates and pixel size of different targets on the final target mask map through an eight-neighborhood clustering algorithm, and map it back to the original infrared image.
