A cross-modal adaptive small target detection optimization method

Abstract

The present invention discloses a cross-modal adaptive small target detection optimization method, comprising the following steps: labeling the bounding box and target mask area of ​​the target in infrared images and visible light images; calculating the statistical properties of the texture features of the infrared and visible light images, and constructing a training data set with the statistical properties of the texture features; building a target detection model and its cross-modal loss function; the cross-modal loss function includes a bounding box regression loss function and a target classification loss function; training the target detection model using the training data set, and dynamically optimizing the bounding box regression loss function using the size of the target mask area during training; and dynamically optimizing the target classification loss function using the statistical properties of the texture features according to a pre-set adjustment strategy. The present invention only requires a single target detection model to complete training and detection of infrared images, and consumes relatively little computing resources.

Description

Technical Field

[0001] The present invention relates to the field of target detection technology, and more particularly to a cross-modal adaptive small target detection optimization method. Background Art

[0002] With the continuous development and application of deep learning technology, object detection technology, as an important and indispensable branch, has always been a hot topic of research and discussion among experts, engineers, scholars and other industry professionals.

[0003] At present, most detection frameworks are trained, verified and tested separately on infrared images (IR) or visible light images (VI), thus evolving into a band-based detection technology. In addition, some detection architectures use a method of fusing infrared images with visible light images, which are then used for target training and detection to form a target detection architecture. Although these target detection methods can bring about mAP (mean Average Precision) improvements in detection, if a band-based detection method is adopted, multiple detection models will invisibly increase the use of hardware storage resources or the consumption of computing resources during the detection model deployment and detection stages. If the infrared image and the visible light image are fused in advance before being used for detection, the complexity of the target detection model training stage or the consumption of computing resources will also be increased.

[0004] While target detection methods are constantly emerging, small target detection remains a significant challenge for industry experts. Infrared imaging technology is widely used in target detection due to its robustness against harsh environments like wind, frost, rain, and snow, and its strong anti-interference capabilities. However, the infrared images it produces suffer from poor resolution and blurred visuals compared to visible images, creating challenges for small target detection.

[0005] Therefore, how to provide a cross-modal adaptive small target detection optimization method that only requires one target detection model to complete the training and detection of infrared images and consumes little computing resources is an urgent problem that technicians in this field need to solve. Summary of the Invention

[0006] In view of this, the present invention provides a cross-modal adaptive small target detection optimization method, which only requires one target detection model to complete the training and detection of infrared images, and consumes less computing resources.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] A cross-modal adaptive small target detection optimization method, comprising:

[0009] Annotate the bounding box and target mask area of ​​the target in infrared and visible light images;

[0010] Calculate the statistical properties of texture features of infrared images and visible light images, and construct a training data set with statistical properties of texture features;

[0011] Constructing a target detection model and its cross-modal loss function; the cross-modal loss function includes a bounding box regression loss function and a target classification loss function;

[0012] The target detection model is trained using the training data set. During the training process, the bounding box regression loss function is dynamically optimized using the size of the target mask area; and the target classification loss function is dynamically optimized according to a pre-set adjustment strategy using the statistical properties of the texture features.

[0013] Furthermore, in the above-mentioned cross-modal adaptive small target detection optimization method, the construction process of the training dataset is:

[0014] The infrared image and the visible light image are processed as grayscale images, randomly sampled and placed together, and the bounding box and target mask area of ​​the target in the image are annotated using a general image annotation tool;

[0015] Calculate the global gray-level co-occurrence matrix for each gray-level image;

[0016] The maximum grayscale value of the neighborhood that is 1.5 times the length and width of the bounding box is obtained, and the grayscale co-occurrence matrix of the target neighborhood is calculated based on the maximum grayscale value;

[0017] According to the statistical properties of the global gray-level co-occurrence matrix and the target neighborhood gray-level co-occurrence matrix of each gray-level image, corresponding annotation data is created for each gray-level image.

[0018] Furthermore, in the above-mentioned cross-modal adaptive small target detection optimization method, when calculating the target neighborhood grayscale co-occurrence matrix, if the maximum grayscale value is greater than 128, the target neighborhood grayscale co-occurrence matrix is ​​calculated according to grayscale 256; if the maximum grayscale value is less than 128, the target neighborhood grayscale co-occurrence matrix is ​​calculated according to grayscale 128.

[0019] Furthermore, in the above-mentioned cross-modal adaptive small target detection optimization method, the annotation data format created for each grayscale image is: [target category, bounding box center coordinate x, bounding box center coordinate y, bounding box width, bounding box height, global standard deviation, target neighborhood standard deviation, global homogeneity, target neighborhood homogeneity, global contrast, target neighborhood contrast, global entropy value, target neighborhood entropy value, global angular second moment, target neighborhood angular second moment, image type], where the image type is infrared image or visible light image.

[0020] Furthermore, in the above-mentioned cross-modal adaptive small target detection optimization method, the number of output channels of the target detection model is: (6+classes)*AN, where classes represents the number of target types detected, and AN is the number of anchor boxes or 1.

[0021] Furthermore, in the above-mentioned cross-modal adaptive small target detection optimization method, the frame regression loss function is: L1 = (1-λ size )*Loss box ; Among them, λ size Indicates the ratio of the target mask area to the entire image area, Loss box Represents the bounding box regression loss value.

[0022] Furthermore, in the above-mentioned cross-modal adaptive small target detection optimization method, the adjustment strategy of the target classification loss function includes:

[0023] The classification loss weight is adjusted using the mean of the statistical attributes of the texture features of each batch; the expression is:

[0024]

[0025] Among them, N represents the batch size, i represents the i-th sample in the batch, f(texture i ) represents the statistical attributes of texture features, Loss Classification Represents the target classification loss value;

[0026] or:

[0027] The dynamic adjustment classification loss weight is calculated for each input sample, and the expression is:

[0028]

[0029] in, Represents the classification loss value of the i-th sample.

[0030] Furthermore, in the above-mentioned cross-modal adaptive small target detection optimization method, the cross-modal loss function expression of the target detection model is:

[0031]

[0032] Among them, Loss box represents the bounding box regression loss function, (1-λ size ) represents the weight of the bounding box regression loss function; Loss Classification represents the target classification loss function, Represents the weight of the target classification loss function; represents the target category confidence loss; Indicates the data type loss of the current image.

[0033] Furthermore, in the above-mentioned cross-modal adaptive small target detection optimization method, the texture feature statistical attributes include: global texture feature statistical attributes and target neighborhood texture feature attributes; the adjustment strategy of the target classification loss function also includes:

[0034] When the number of training iterations is greater than a set threshold, randomly selecting one of the global texture feature statistical attributes to update the weight coefficient of the target classification loss function;

[0035] When the number of training iterations is less than a set threshold, one of the target neighborhood texture feature attributes is randomly selected to update the weight coefficient of the target classification loss function.

[0036] Furthermore, in the above-mentioned cross-modal adaptive small target detection optimization method, when the target detection model performs target detection, the target prediction probability is equal to the product of the target category probability value, the target confidence and the probability value of the data type to which the target belongs.

[0037] It can be seen from the above technical solution that compared with the prior art, the present invention discloses a cross-modal adaptive small target detection optimization method, which adaptively adjusts the loss weight of the detection model training according to the target size in the infrared image and the visible light image and the statistical attributes of the target texture features, wherein the target size is mainly used to adjust the border regression loss, and the target texture feature statistical attributes are mainly used to adjust the target classification loss; only one target detection model is used to complete the training and detection of infrared images and visible light images, instead of training multiple models in a band-based manner and deploying multiple models for detection; for small and weak targets in infrared images and visible light images, the loss weight during the detection model training is adaptively adjusted, thereby achieving the detection effect of cross-modal adaptive small and weak targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0039] Figure 1 Flowchart of the cross-modal adaptive small target detection optimization method provided by the present invention;

[0040] Figure 2 A schematic diagram of the construction process of the training data set provided by the present invention;

[0041] Figure 3 This is a schematic diagram of the structure of the target detection model provided by the present invention;

[0042] Figure 4 A schematic diagram of the construction of the cross-modal loss function provided by the present invention;

[0043] Figure 5 Schematic diagram of the detection process of the target detection model provided by the present invention;

[0044] Figure 6 Schematic diagram of the weight coefficient adjustment strategy for the target classification loss function provided by the present invention. DETAILED DESCRIPTION

[0045] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0046] like Figure 1 As shown, an embodiment of the present invention discloses a cross-modal adaptive small target detection optimization method, including:

[0047] S1, annotate the bounding box and target mask area of ​​the target in the infrared image and visible light image;

[0048] S2. Calculate the statistical attributes of texture features of infrared images and visible light images, and construct a training data set with statistical attributes of texture features;

[0049] S3. Build an object detection model and its cross-modal loss function; the cross-modal loss function includes a bounding box regression loss function and an object classification loss function;

[0050] S4. Use the training dataset to train the target detection model. During the training process, the bounding box regression loss function is dynamically optimized using the size of the target mask area; the target classification loss function is dynamically optimized using the statistical properties of the texture features according to a pre-set adjustment strategy.

[0051] In a specific embodiment, Figure 2 As shown in Figure 2, the construction process of the training data set is:

[0052] ① Process both infrared and visible light images as grayscale images, randomly sample them and place them together. Use general image annotation tools to mark the bounding boxes and target mask areas of the objects in the images to obtain the initial dataset.

[0053] ② Calculate the global gray-level co-occurrence matrix for each gray-level image;

[0054] Among them, the global gray-level co-occurrence matrix is ​​calculated, the grayscale is set to 256, the adjacent pixel distances are set to 2, 8, and 16 respectively, and the 8 direction angles are set to 0°, 45°, 90°, 135°, 180°, 225°, 270°, 315°, and 360° respectively. Here, 3*8=24 gray-level co-occurrence matrices need to be calculated.

[0055] ③ Obtain the maximum grayscale value of the neighborhood that is 1.5 times the length and width of the bounding box, and calculate the target neighborhood grayscale co-occurrence matrix based on the maximum grayscale value;

[0056] Specifically, the maximum grayscale value of a neighborhood that is 1.5 times the length and width of the bounding box is calculated. If the maximum grayscale value is greater than 128, the target neighborhood grayscale co-occurrence matrix is ​​calculated according to grayscale 256, where the adjacent pixel distances are 2, 8, and 16, respectively, and 8 direction angles are also used. If the maximum grayscale value is less than 128, the target neighborhood grayscale co-occurrence matrix is ​​calculated according to grayscale 128, where the adjacent pixel distances are 2 and 8, respectively, and 4 direction angles are used: 0°, 90°, 180°, 270°, and 360°.

[0057] ④ According to the statistical properties of the global grayscale co-occurrence matrix and the target neighborhood grayscale co-occurrence matrix of each grayscale image, corresponding annotation data are created for each grayscale image to obtain the final training data set.

[0058] According to the global gray-level co-occurrence matrix, the global contrast, target neighborhood contrast, global entropy, target neighborhood entropy, global angular second moment (ASM), target neighborhood angular second moment (ASM), global homogeneity, target neighborhood homogeneity, and global standard deviation (STD) are calculated;

[0059] Calculate the target neighborhood standard deviation based on the target neighborhood gray level co-occurrence matrix.

[0060] Specifically, the target neighborhood contrast (Contrast) calculation formula is as follows:

[0061]

[0062] In the formula, As represents the number of set direction angles (such as 4), Ds represents the number of selected adjacent pixel distances (such as 2, 8), and p i，j Represents the gray-level co-occurrence matrix element, gray_levels represents the grayscale, and Norm represents the maximum and minimum values ​​(Min-Max) normalization of the contrast between the two distance types and the four direction angles.

[0063] The formula for calculating the homogeneity of the target neighborhood is as follows:

[0064]

[0065] The target neighborhood entropy calculation formula is as follows:

[0066]

[0067] The calculation formula of the target neighborhood angular second moment (ASM) is as follows:

[0068]

[0069] In a specific embodiment, the annotation data format created for each grayscale image is: [target category, bounding box center coordinate x, bounding box center coordinate y, bounding box width, bounding box height, global standard deviation, target neighborhood standard deviation, global homogeneity, target neighborhood homogeneity, global contrast, target neighborhood contrast, global entropy value, target neighborhood entropy value, global angular second moment, target neighborhood angular second moment, image type], where the image type is infrared image or visible light image.

[0070] In one embodiment, Figure 3 As shown in the figure, the target detection model consists of a feature extraction network and a detection head. The number of output channels of the detection head is: (6 + classes) * AN, where 6 represents the number of output channels of the feature extraction network. The output channels respectively represent the detection coordinate information, the confidence of the detection box, and the image type (infrared or visible light) to which the detection box belongs. The detected coordinate information includes: the center coordinates (x, y) and the width and height (w, h) of the detection box, a total of 4 output channels. Classes represents the number of target types detected, and AN is the number of anchor boxes or 1. If the detector is anchor-based, AN is the set number of anchor boxes. If the detector is not anchor-free, AN is 1.

[0071] For example, the PascalVOC training dataset has 20 target categories and the number of anchor boxes is N=3, so the number of output channels is calculated as (6+20)*3=78. The COCO training dataset has 80 target categories and the number of anchor boxes is N=3, so the number of output channels is calculated as (6+80)*3=258.

[0072] In one embodiment, Figure 4 As shown, the border regression loss function is: L1 = (1-λ size )*Loss box ; Among them, λ size Indicates the ratio of the target mask area to the entire image area, Loss box Represents the bounding box regression loss value.

[0073] The bounding box regression loss function is used to measure the target size, which involves the coefficient λ size , this coefficient is used to explicitly adjust the bounding box regression loss function. Its calculation is defined as the ratio of the target mask area to the full image area, and its calculation is shown in the following formula:

[0074]

[0075] Among them, λ size Represents the coefficient used to measure the target size in bounding box regression, Represents the target mask area, A area Indicates the area of ​​the entire map.

[0076] In one embodiment, there are two strategies for adjusting the target classification loss function, one of which is:

[0077] The classification loss weight is adjusted using the mean of the statistical attributes of the texture features of each batch; the expression is:

[0078]

[0079] Among them, N represents the batch size, i represents the i-th sample in the batch, f(texture i ) represents the statistical attributes of texture features, Loss Classification Represents the target classification loss value;

[0080] Another is:

[0081] The dynamic adjustment classification loss weight is calculated for each input sample, and the expression is:

[0082]

[0083] in, Represents the classification loss value of the i-th sample.

[0084] In one embodiment, the cross-modal loss function of the target detection model is expressed as:

[0085]

[0086] Among them, Loss box represents the bounding box regression loss function, (1-λ size ) represents the weight of the bounding box regression loss function; Loss Classification represents the target classification loss function, Represents the weight of the target classification loss function; represents the target category confidence loss; Indicates the data type loss of the current image (infrared image or visible light image).

[0087] In one embodiment, Figure 5 As shown in the figure, when the target detection model performs target detection, the target prediction probability (target global confidence) is equal to the product of the target category probability value, the target confidence value, and the probability value of the target data type. When screening the prediction box, the classification probability threshold can be used to complete the preliminary prediction box selection. The calculation formula is:

[0088] Prob=prob classification *prob confidence *prob obj_type

[0089] Among them, prob classification The probability of the detected target belonging to the category, for example: the classification probability of target A is 0.1, and the classification probability of target B is 0.5; prob confidence The confidence level of the detected target indicates the probability that the detected target is the target. For example, the probability that target C is the target is 0.2; prob obj_type The type of data (infrared or visible light image) of the detected target.

[0090] In one embodiment, Figure 6 As shown in the figure, the texture feature statistical attributes include: global texture feature statistical attributes and target neighborhood texture feature attributes; before training, the grayscale texture feature statistical attributes are used to randomly initialize the target classification loss function weight coefficient f(texture i ). Using different statistical properties of texture features, we explicitly update the loss function f(texture i ). The strategy for adjusting the weight coefficient of the target classification loss function is as follows:

[0091] When the number of training iterations is greater than the set threshold, one of the global texture feature statistical attributes is randomly selected to update the weight coefficient of the target classification loss function; the global texture feature statistical attributes include: global standard deviation, global homogeneity, global contrast, global entropy and global ASM.

[0092] When the number of training iterations is less than the set threshold, one of the target neighborhood texture feature attributes is randomly selected to update the weight coefficient of the target classification loss function. The target neighborhood texture feature statistical attributes include: target neighborhood standard deviation, target neighborhood homogeneity, target neighborhood contrast, target neighborhood entropy value, and target neighborhood ASM.

[0093] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0094] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A cross-modal adaptive small target detection optimization method, characterized in that: include: Annotate the bounding box and target mask area of ​​the target in infrared and visible light images; Calculate the statistical properties of texture features of infrared images and visible light images, and construct a training data set with statistical properties of texture features; Constructing a target detection model and its cross-modal loss function; the cross-modal loss function includes a bounding box regression loss function and a target classification loss function; The target detection model is trained using the training data set. During the training process, the bounding box regression loss function is dynamically optimized using the size of the target mask area; and the target classification loss function is dynamically optimized according to a pre-set adjustment strategy using the statistical properties of the texture features.

2. A cross-modal adaptive small target detection optimization method according to claim 1, characterized in that: The construction process of the training data set is: The infrared image and the visible light image are processed as grayscale images, randomly sampled and placed together, and the bounding box and target mask area of ​​the target in the image are annotated using a general image annotation tool; Calculate the global gray-level co-occurrence matrix for each gray-level image; The maximum grayscale value of the neighborhood that is 1.5 times the length and width of the bounding box is obtained, and the grayscale co-occurrence matrix of the target neighborhood is calculated based on the maximum grayscale value; According to the statistical properties of the global gray-level co-occurrence matrix and the target neighborhood gray-level co-occurrence matrix of each gray-level image, corresponding annotation data is created for each gray-level image.

3. A cross-modal adaptive small target detection optimization method according to claim 2, characterized in that: When calculating the target neighborhood gray level co-occurrence matrix, if the maximum gray value is greater than 128, the target neighborhood gray level co-occurrence matrix is ​​calculated according to gray level 256; if the maximum gray value is less than 128, the target neighborhood gray level co-occurrence matrix is ​​calculated according to gray level 128.

4. A cross-modal adaptive small target detection optimization method according to claim 2, characterized in that: The annotation data format created for each grayscale image is: [target category, bounding box center coordinate x, bounding box center coordinate y, bounding box width, bounding box height, global standard deviation, target neighborhood standard deviation, global homogeneity, target neighborhood homogeneity, global contrast, target neighborhood contrast, global entropy, target neighborhood entropy, global angular second moment, target neighborhood angular second moment, image type], where the image type is infrared image or visible light image.

5. A cross-modal adaptive small target detection optimization method according to claim 1, characterized in that: The number of output channels of the target detection model is: (6+classes)*AN, where classes represents the number of target types detected and AN is the number of anchor boxes or 1.

6. A cross-modal adaptive small target detection optimization method according to claim 1, characterized in that: The border regression loss function is: L1 = (1-λ size )*Loss box ; Among them, λ size Indicates the ratio of the target mask area to the entire image area, Loss box Represents the bounding box regression loss value.

7. A cross-modal adaptive small target detection optimization method according to claim 6, characterized in that: The adjustment strategy of the target classification loss function includes: The classification loss weight is adjusted using the mean of the statistical attributes of the texture features of each batch; the expression is: Among them, N represents the batch size, i represents the i-th sample in the batch, f(texture i ) represents the statistical attributes of texture features, Loss Classification Represents the target classification loss value; or: The dynamic adjustment classification loss weight is calculated for each input sample, and the expression is: in, Represents the classification loss value of the i-th sample.

8. A cross-modal adaptive small target detection optimization method according to claim 7, characterized in that: The cross-modal loss function expression of the target detection model is: Among them, Loss box represents the bounding box regression loss function, (1-λ size ) represents the weight of the bounding box regression loss function; Loss Classification represents the target classification loss function, Represents the weight of the target classification loss function; represents the target category confidence loss; Indicates the data type loss of the current image.

9. A cross-modal adaptive small target detection optimization method according to claim 1, characterized in that: The texture feature statistical attributes include: global texture feature statistical attributes and target neighborhood texture feature attributes; the adjustment strategy of the target classification loss function also includes: When the number of training iterations is greater than a set threshold, randomly selecting one of the global texture feature statistical attributes to update the weight coefficient of the target classification loss function; When the number of training iterations is less than a set threshold, one of the target neighborhood texture feature attributes is randomly selected to update the weight coefficient of the target classification loss function.

10. A cross-modal adaptive small target detection optimization method according to claim 1, characterized in that: When the target detection model performs target detection, the target prediction probability is equal to the product of the probability value of the category to which the target belongs, the target confidence and the probability value of the data type to which the target belongs.

Images