A monocular infrared image target detection and depth estimation method and device

Abstract

The present invention discloses a method and device for monocular infrared image target detection and depth estimation. The method includes: obtaining a training data sample set and a training data sample label set; performing data preprocessing and data augmentation processing on the training data sample set to obtain a training data set; using a feature extraction network model to perform feature extraction on the training data set to obtain a feature map; processing the feature map to obtain a fused feature map; using the fused feature map to train a target detection and depth estimation network model to obtain a globally optimal target detection model and a globally optimal instance segmentation depth estimation model; using the globally optimal target detection model and the globally optimal instance segmentation depth estimation model to process the infrared image to be processed to obtain a target prediction result, an instance segmentation of the target, and a depth estimation result. The present invention improves the accuracy of infrared image depth estimation and realizes the instance segmentation of infrared image targets while estimating the depth.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision instance segmentation, and in particular, to a method and device for monocular infrared image target detection and depth estimation. Background Art

[0002] In recent years, machine vision has achieved a leap in both research and application with the development of deep learning. It has become possible for machine learning to estimate the depth of an object based on an image like the human eye. A depth map can provide the distance of an object, construct a three-dimensional image of the environment, and has begun to be applied in fields such as measurement, navigation, and autonomous driving. Currently, depth maps mainly come from depth cameras or radar detections, but limitations such as high cost and difficulty in carrying have become their drawbacks, so depth estimation based on images has begun to receive extensive attention. Currently, image depth estimation is divided into two types: binocular depth estimation and monocular depth estimation. Binocular depth estimation is based on the human eye for binocular parallax triangulation ranging, which can obtain relatively accurate depth information, but requires camera pairs, is troublesome to configure, and has high requirements for image shooting; there is also a problem of matching corresponding points in binocular images in binocular depth estimation, and currently features such as SIFT and SURF are used for matching; in practical applications (such as recognition tasks), there are mostly only single images, so it is not easy to apply in practice. In view of this, monocular depth estimation is more in line with the actual application scenario. Monocular depth estimation is further divided into depth estimation based on depth cues and depth estimation by deep learning. Since the deep learning method is not restricted by the scene and has good practicability, it has become the most concerned research. This method is similar to the learning ability of the human eye, using a large number of training images input into a defined model, using depth information as the supervision to train the model, and after training is completed, the desired depth information can be obtained by inputting the image to be measured. Currently, the achievements obtained in monocular depth estimation are mostly for depth estimation of visible light images, and infrared image depth estimation methods mostly come from the "transplantation" of visible light image depth estimation methods, without much consideration of infrared image depth features and depth cues; the background of infrared images contributes far less to depth estimation than infrared targets; at the same time, the lack of depth data for infrared images is also a drawback of infrared image depth estimation. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a method and device for monocular infrared image target detection and depth estimation, which can perform instance segmentation on the targets in a monocular infrared image through a deep learning method, and at the same time estimate the depth of the targets through the network. The characteristics of the present invention are that while completing target detection, classification, and positioning, target depth estimation is also achieved. In the present invention, the target is an imaging target in an infrared image; the depth refers to the distance of the target relative to the sensor (infrared device); instance segmentation will distinguish targets of different categories and different individuals of the same category; the present invention is a one-stage prediction network.

[0004] To solve the above technical problems, a monocular infrared image target detection and depth estimation method is disclosed in the first aspect of the embodiment of the present invention. The method includes:

[0005] S1. Obtain a training data sample set and a training data sample label set; the training data sample set includes infrared images and depth information corresponding to the pixels of the infrared images;

[0006] The training data sample label set includes the category of the target, the target mask, the target position rectangle, the target depth information, and the image geometric information;

[0007] S2. Perform data preprocessing and data augmentation processing on the training data sample set to obtain a training data set;

[0008] S3. Use a feature extraction network model to extract features from the training data set to obtain a feature map; the feature map includes a feature map downsampled by 1 / 2 times, a feature map downsampled by 1 / of 4 times, a feature map downsampled by 1 / 8 times, a feature map downsampled by 1 / 16 times, and a feature map downsampled by 1 / 32 times;

[0009] S4. Use a feature fusion model to process the feature map to obtain a fused feature map;

[0010] The feature fusion model includes a feature pyramid network and a path aggregation network;

[0011] The feature pyramid network is composed of a spatial pyramid pooling module, an upsampling by 1 / 16 times module, a first channel splicing module, a first convolutional layer, an upsampling by 1 / 8 times module, and a second channel splicing module connected in series;

[0012] The path aggregation network is composed of a second convolutional layer, a downsampling by 1 / 16 times module, a third channel splicing module, a third convolutional layer, a downsampling by 1 / 32 times module, and a fourth channel splicing module connected in series;

[0013] S5. Process the fused feature map to obtain a target classification prediction result and a localization box regression prediction result;

[0014] S6. Process the fused feature map to obtain an instance segmentation and depth estimation result of the target;

[0015] S7. Use the target classification prediction result and the localization box regression prediction result, the instance segmentation and depth estimation result of the target, and the training data sample label set to calculate the loss, and use the backpropagation gradient descent method to minimize the loss to obtain a globally optimal target detection model and a globally optimal instance segmentation depth estimation model;

[0016] S8. Obtain the infrared image to be processed, and process the infrared image to be processed by using the globally optimal object detection model and the globally optimal instance segmentation depth estimation model to obtain the target classification prediction result, the localization box regression prediction result, the instance segmentation of the target, and the depth estimation result.

[0017] As an optional implementation manner, in the first aspect of the embodiments of the present invention, the data preprocessing and data augmentation processing are performed on the training data sample set to obtain a training data set, including:

[0018] S21. Perform scaling processing and white edge filling processing on the training data sample set to obtain a preprocessed training data sample set;

[0019] S22. Randomly select M preprocessed training data samples from the preprocessed training data sample set, and perform λ-fold scaling processing on the M preprocessed training data samples and the corresponding target mask and target position rectangle to obtain M scaled preprocessed training data samples, where M is an integer greater than or equal to 1;

[0020] S23. Randomly select N preprocessed training data samples from the preprocessed training data sample set, and perform cropping processing on the N preprocessed training data samples and the corresponding target mask and target position rectangle to obtain N cropped preprocessed training data samples, where N is an integer greater than or equal to 1;

[0021] S24. Randomly extract P preprocessed training data samples from the preprocessed training data sample set, and perform brightness and contrast adjustment on the P preprocessed training data samples to obtain a first preprocessed training data sample set, where P is an integer greater than or equal to 1;

[0022] S25. Randomly extract Q preprocessed training data samples from the preprocessed training data sample set, and perform mirror processing on the head images and the corresponding target masks and target position rectangles in the Q preprocessed training data samples to obtain a mirror preprocessed training data sample set, where Q is an integer greater than or equal to 1;

[0023] S26. Randomly extract O preprocessed training data samples from the preprocessed training data sample set, and perform rotation processing on the head images and the corresponding target masks and target position rectangles in the O preprocessed training data samples to obtain a rotation preprocessed training data sample set, where O is an integer greater than or equal to 1;

[0024] S27. Add the M scaled preprocessed training data samples, the N cropped preprocessed training data samples, the first preprocessed training data sample set, the mirror preprocessed training data sample set, and the rotation preprocessed training data sample set to the preprocessed training data sample set to obtain a training data set.

[0025] As an alternative implementation, in the first aspect of the embodiments of the present invention, the feature extraction network model is composed of a first downsampling module, a second downsampling module, a third downsampling module, a fourth downsampling module, and a fifth downsampling module connected in series, and each time the width and height of the image are downsampled by one-half;

[0026] Each downsampling module includes a convolutional downsampling module, an atrous convolutional downsampling module, a residual module, and a channel splicing module;

[0027] The convolutional downsampling module, the atrous convolutional downsampling module, and the residual module are in a parallel relationship;

[0028] The channel splicing module fuses the outputs of the convolutional downsampling module, the atrous convolutional downsampling module, and the residual module to obtain a fused output result;

[0029] The convolutional downsampling module is composed of a convolutional layer, batch normalization, and an activation function connected in series;

[0030] The atrous convolutional downsampling module is composed of an atrous convolutional layer, batch normalization, and an activation function connected in series;

[0031] The residual module is composed of two convolutional modules connected in series, and the intermediate output is short-circuited to the output;

[0032] The atrous convolutional downsampling module is used to increase the receptive field of the convolutional downsampling feature map;

[0033] The residual module is used to prevent gradient disappearance during the training process.

[0034] As an alternative implementation, in the first aspect of the embodiments of the present invention, the use of the feature fusion model to process the feature map to obtain a fused feature map includes:

[0035] S41, input the downsampled 1 / 32 - fold feature map P 32 into the spatial pyramid pooling module of the feature pyramid network to obtain a feature fusion map P1;

[0036] S42, upsample the feature fusion map P1 by 1 / 16 - fold to obtain an upsampled 1 / 16 - fold feature map;

[0037] S43, perform channel splicing on the upsampled 1 / 16 - fold feature map and the downsampled 1 / 16 - fold feature map P 16 and perform convolutional fusion to obtain a first feature fusion map P 16+ ;

[0038] S43, the first feature fusion map P 16+Upsample by 1 / 8 times to obtain an upsampled 1 / 8 - times feature map;

[0039] S44, perform channel concatenation on the upsampled 1 / 8 - times feature map and the downsampled 1 / 8 - times feature map P8 and perform convolution fusion to obtain a second feature fusion map P 8+ ;

[0040] S45, perform channel concatenation on the second feature fusion map P 8+ and the downsampled 1 / 8 - times feature map P8 and perform convolution fusion to obtain a fused and sampled 1 / 8 - times feature map P 8* ;

[0041] S46, downsample the second feature fusion map P 8+ by 1 / 16 times to obtain a downsampled 1 / 16 - times feature map;

[0042] S47, perform channel concatenation on the downsampled 1 / 16 - times feature map and the first feature fusion map P 16+ and perform convolution fusion to obtain a fused and sampled 1 / 16 - times feature map P 16* ;

[0043] S48, downsample the fused and sampled 1 / 16 - times feature map P 16* by 1 / 32 times to obtain a downsampled 1 / 32 - times feature map;

[0044] S49, perform channel concatenation on the downsampled 1 / 32 - times feature map and the feature fusion map P1 and perform convolution fusion to obtain a fused and sampled 1 / 32 - times feature map P 32* ;

[0045] The fused and sampled 1 / 8 - times feature map P 8* , the fused and sampled 1 / 16 - times feature map P 16* and the fused and sampled 1 / 32 - times feature map P 32* constitute a fused feature map.

[0046] As an optional implementation manner, in the first aspect of the embodiments of the present invention, the processing the fused feature map to obtain a target classification prediction result and a localization box regression prediction result includes:

[0047] Use a target detection network loss function to process the fused feature map to obtain a target classification prediction result and a localization box regression prediction result;

[0048] The target detection network loss function is:

[0049] Loss = Loss clc + Loss conf + Loss loc

[0050] Among them, Loss clc is the classification loss, and the multi-class binary cross-entropy loss is adopted. Loss conf is the confidence loss, and the CIOU method is adopted. In the CIOU method, IoU(BP, BGT) is used as the confidence, where BP is the predicted target box and BGT is the target box of the true label; Loss loc is the localization regression loss;

[0051] The above-mentioned localization regression loss Loss loc is:

[0052] Loss clc =(x - x*) 2 +(y - y*) 2 +(w - w*) 2 +(h - h*) 2

[0053] Among them, x, y, w, h are the coordinate information of the predicted box, and x*, y*, w*, h* are the coordinate information of the true label box.

[0054] As an optional implementation manner, in the first aspect of the embodiments of the present invention, the processing of the fused feature map to obtain the instance segmentation and depth estimation results of the target includes:

[0055] S61. According to the target classification prediction result and the localization box regression prediction result, process the fused sampled 1 / 8 - fold feature map P8*, the fused sampled 1 / 16 - fold feature map P16*, and the fused sampled 1 / 32 - fold feature map P32* to obtain the first feature map, the second feature map, and the third feature map with a downsampling of 1 / 8;

[0056] S62. Use the instance segmentation and depth estimation loss function to process the first feature map, the second feature map, and the third feature map with a downsampling of 1 / 8 to obtain the instance segmentation and depth estimation results of the target.

[0057] As an optional implementation manner, in the first aspect of the embodiments of the present invention, the instance segmentation and depth estimation loss function is:

[0058] Loss1 = Loss mask + Loss death + Loss geom

[0059] Among them, Loss mask is the instance segmentation loss, and the binary cross - entropy loss is adopted for each pixel. Loss death is the depth estimation loss, and the MSE loss function is adopted for the depth value. Loss geomFor the geometric loss, the Smooth-L1 loss function is adopted.

[0060] In a second aspect of the embodiments of the present invention, a monocular infrared image target detection and depth estimation device is disclosed. The device includes:

[0061] A data acquisition module, configured to acquire a training data sample set and a training data sample label set; the training data sample set includes infrared images and depth information corresponding to the pixels of the infrared images;

[0062] The training data sample label set includes the category of the target, the target mask, the target position rectangle, the target depth information, and the image geometric information;

[0063] A preprocessing module, configured to perform data preprocessing and data augmentation processing on the training data sample set to obtain a training data set;

[0064] A feature extraction module, configured to use a feature extraction network model to extract features from the training data set to obtain a feature map; the feature map includes a feature map downsampled by 1 / 2 times, a feature map downsampled by 1 / 4 times, a feature map downsampled by 1 / 8 times, a feature map downsampled by 1 / 16 times, and a feature map downsampled by 1 / 32 times;

[0065] A feature fusion module, configured to use a feature fusion model to process the feature map to obtain a fused feature map;

[0066] The feature fusion model includes a feature pyramid network and a path aggregation network;

[0067] The feature pyramid network is composed of a spatial pyramid pooling module, an upsampling by 1 / 16 times module, a first channel splicing module, a first convolutional layer, an upsampling by 1 / 8 times module, and a second channel splicing module connected in series;

[0068] The path aggregation network is composed of a second convolutional layer, a downsampling by 1 / 16 times module, a third channel splicing module, a third convolutional layer, a downsampling by 1 / 32 times module, and a fourth channel splicing module connected in series;

[0069] A prediction module, configured to process the fused feature map to obtain a target classification prediction result and a localization box regression prediction result;

[0070] An estimation module, configured to process the fused feature map to obtain an instance segmentation and depth estimation result of the target;

[0071] A training module, configured to calculate a loss by using the target classification prediction result, the localization box regression prediction result, the instance segmentation and depth estimation results of the target, and the training data sample label set, and minimize the loss by using the backpropagation gradient descent method to obtain a globally optimal object detection model and a globally optimal instance segmentation depth estimation model;

[0072] An object detection and depth estimation module, configured to obtain an infrared image to be processed, and process the infrared image to be processed by using the globally optimal object detection model and the globally optimal instance segmentation depth estimation model to obtain a target classification prediction result, a localization box regression prediction result, an instance segmentation and depth estimation result of the target.

[0073] As an optional implementation manner, in the second aspect of the embodiments of the present invention, the data preprocessing and data augmentation processing of the training data sample set to obtain a training data set includes:

[0074] S21, performing a scaling process and a white edge filling process on the training data sample set to obtain a preprocessed training data sample set;

[0075] S22, randomly selecting M preprocessed training data samples from the preprocessed training data sample set, and performing a λ-fold scaling process on the M preprocessed training data samples and the corresponding target masks and target position rectangles to obtain M scaled preprocessed training data samples, where M is an integer greater than or equal to 1;

[0076] S23, randomly selecting N preprocessed training data samples from the preprocessed training data sample set, and performing a cropping process on the N preprocessed training data samples and the corresponding target masks and target position rectangles to obtain N cropped preprocessed training data samples, where N is an integer greater than or equal to 1;

[0077] S24, randomly extracting P preprocessed training data samples from the preprocessed training data sample set, and adjusting the brightness and contrast of the P preprocessed training data samples to obtain a first preprocessed training data sample set, where P is an integer greater than or equal to 1;

[0078] S25, randomly extracting Q preprocessed training data samples from the preprocessed training data sample set, and performing a mirroring process on the head images and the corresponding target masks and target position rectangles in the Q preprocessed training data samples to obtain a mirrored preprocessed training data sample set, where Q is an integer greater than or equal to 1;

[0079] S26. Randomly select O preprocessed training data samples from the preprocessed training data sample set, and perform rotation processing on the head images, corresponding target masks, and target position rectangular frames in the O preprocessed training data samples to obtain a rotated preprocessed training data sample set, where O is an integer greater than or equal to 1;

[0080] S27. Add the M downscaled preprocessed training data samples, N cropped preprocessed training data samples, the first preprocessed training data sample set, the mirrored preprocessed training data sample set, and the rotated preprocessed training data sample set to the preprocessed training data sample set to obtain a training data set.

[0081] As an optional implementation manner, in the second aspect of the embodiments of the present invention, the feature extraction network model is composed of a first downsampling module, a second downsampling module, a third downsampling module, a fourth downsampling module, and a fifth downsampling module connected in series, and each time the width and height of the image are downsampled by one-half;

[0082] Each downsampling module includes a convolutional downsampling module, an atrous convolutional downsampling module, a residual module, and a channel splicing module;

[0083] The convolutional downsampling module, the atrous convolutional downsampling module, and the residual module are in a parallel relationship;

[0084] The channel splicing module fuses the outputs of the convolutional downsampling module, the atrous convolutional downsampling module, and the residual module to obtain a fused output result;

[0085] The convolutional downsampling module is composed of a convolutional layer, batch normalization, and an activation function connected in series;

[0086] The atrous convolutional downsampling module is composed of an atrous convolutional layer, batch normalization, and an activation function connected in series;

[0087] The residual module is composed of two convolutional modules connected in series, and the intermediate output is short-circuited to the output;

[0088] The atrous convolutional downsampling module is used to increase the receptive field of the convolutional downsampling feature map;

[0089] The residual module is used to prevent gradient disappearance during the training process.

[0090] As an optional implementation manner, in the second aspect of the embodiments of the present invention, the processing of the feature map by using the feature fusion model to obtain a fused feature map includes:

[0091] S41. Input the downsampled 1 / 32 - fold feature map P 32 into the spatial pyramid pooling module of the feature pyramid network to obtain a feature fusion map P1;

[0092] S42, upsampling the feature fusion map P1 by 1 / 16 to obtain an upsampled 1 / 16 feature map;

[0093] S43, the up-sampled 1 / 16 times feature map and the down-sampled 1 / 16 times feature map P 16 Perform channel splicing and convolution fusion to obtain the first feature fusion map P 16+ ;

[0094] S43, the first feature fusion graph P 16+ Upsample by 1 / 8 to obtain a 1 / 8 times upsampled feature map;

[0095] S44, channel splicing and convolution fusion are performed on the up-sampled 1 / 8 times feature map and the down-sampled 1 / 8 times feature map P8 to obtain a second feature fusion map P 8+ ;

[0096] S45, fusion graph P of the second feature 8+ The channel is spliced with the downsampled 1 / 8 times feature map P8 and convolution fusion is performed to obtain the fused sampling 1 / 8 times feature map P 8* ;

[0097] S46, fusion graph P of the second feature 8+ Downsample by 1 / 16 to obtain a downsampled 1 / 16 feature map;

[0098] S47, the downsampled 1 / 16 times feature map is combined with the first feature fusion map P 16+ Perform channel splicing and convolution fusion to obtain the fused sampling 1 / 16 times feature map P 16* ;

[0099] S48, the fusion sampling 1 / 16 times the feature map P 16* Downsample by 1 / 32 to obtain a downsampled 1 / 32 feature map;

[0100] S49, the downsampled 1 / 32 times feature map and the feature fusion map P1 are channel-joined and convolutionally fused to obtain a fused 1 / 32 times sampled feature map P 32* ;

[0101] The fusion sampling is 1 / 8 times the feature map P 8* , fusion sampling 1 / 16 times feature map P 16* And fusion sampling 1 / 32 times feature map P 32* Construct a fusion feature map.

[0102] As an optional implementation, in the second aspect of the embodiment of the present invention, processing the fused feature map to obtain a target classification prediction result and a positioning box regression prediction result includes:

[0103] Process the fused feature map by using the loss function of the object detection network to obtain the object classification prediction result and the bounding box regression prediction result;

[0104] The loss function of the object detection network is as follows:

[0105] Loss = Loss clc + Loss conf + Loss loc

[0106] where Loss clc is the classification loss, and the multi-class binary cross-entropy loss is adopted. Loss conf is the confidence loss, and the CIOU method is adopted. In the CIOU method, IoU(BP, BGT) is used as the confidence, where BP is the predicted object bounding box and BGT is the object bounding box of the true label; Loss loc is the localization regression loss;

[0107] The localization regression loss Loss loc is as follows:

[0108] Loss clc = (x - x*) 2 + (y - y*) 2 + (w - w*) 2 + (h - h*) 2

[0109] where x, y, w, h are the coordinate information of the predicted bounding box, and x*, y*, w*, h* are the coordinate information of the true label bounding box.

[0110] As an alternative implementation manner, in the second aspect of the embodiments of the present invention, the processing of the fused feature map to obtain the instance segmentation and depth estimation results of the object includes:

[0111] S61. Process the fused downsampled 1 / 8 - fold feature map P8*, the fused downsampled 1 / 16 - fold feature map P16*, and the fused downsampled 1 / 32 - fold feature map P32* according to the object classification prediction result and the bounding box regression prediction result to obtain the first feature map, the second feature map, and the third feature map that are downsampled 1 / 8 - fold;

[0112] S62. Process the first feature map, the second feature map, and the third feature map that are downsampled 1 / 8 - fold by using the instance segmentation and depth estimation loss function to obtain the instance segmentation and depth estimation results of the object.

[0113] As an alternative implementation manner, in the second aspect of the embodiments of the present invention, the instance segmentation and depth estimation loss function is:

[0114] Loss1 = Loss mask + Loss death + Loss geom

[0115] Among them, Loss mask is the instance segmentation loss, and binary cross-entropy loss is adopted for each pixel. Loss death is the depth estimation loss, and the MSE loss function is adopted for the depth value. Loss geom is the geometric loss, and the Smooth-L1 loss function is adopted.

[0116] The third aspect of the present invention discloses another monocular infrared image target detection and depth estimation device, and the device includes:

[0117] A memory storing executable program code;

[0118] A processor coupled to the memory;

[0119] The processor calls the executable program code stored in the memory and executes some or all of the steps in the monocular infrared image target detection and depth estimation method disclosed in the first aspect of the embodiments of the present invention.

[0120] The fourth aspect of the present invention discloses a computer storage medium, and the computer storage medium stores computer instructions, which are used to execute some or all of the steps in the monocular infrared image target detection and depth estimation method disclosed in the first aspect of the embodiments of the present invention when called.

[0121] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0122] (1) According to the characteristics of infrared images and the characteristics of infrared targets in infrared images, the present invention specifically sets the instance segmentation and depth estimation of targets. Through various information such as the texture, geometry, and edges of different targets, the relative information between targets, and environmental information, the depth of the target is jointly estimated, improving the accuracy of depth estimation of infrared images and realizing the instance segmentation of infrared image targets while estimating the depth.

[0123] (2) The present invention can more accurately reflect the pixel information of the target, so as to more accurately apply the geometric features of the target to improve the accuracy of depth estimation.

[0124] (3) The object of attention in depth estimation of the present invention is the infrared target. Since the background generally contributes relatively little to the depth in infrared images, shifting the attention of network learning to the target can better learn the learnable content and improve the accuracy of depth estimation. Description of the Drawings

[0125] To more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the accompanying drawings required for the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0126] Figure 1 is a schematic flowchart of a monocular infrared image target detection and depth estimation method disclosed in an embodiment of the present invention;

[0127] Figure 2 is a schematic diagram of a target detection and depth estimation network model disclosed in an embodiment of the present invention;

[0128] Figure 3 is a schematic diagram of an image and Grid Cell division disclosed in an embodiment of the present invention;

[0129] Figure 4 is a schematic diagram of the composition of a target detection and depth estimation network disclosed in an embodiment of the present invention;

[0130] Figure 5 is a schematic diagram of the training process of a target detection and depth estimation network model disclosed in an embodiment of the present invention;

[0131] Figure 6 is a schematic diagram of the input and output of a target detection and depth estimation network model disclosed in an embodiment of the present invention;

[0132] Figure 7 is the instance segmentation and target depth estimation result disclosed in an embodiment of the present invention;

[0133] Figure 8 is a schematic structural diagram of a monocular infrared image target detection and depth estimation device disclosed in an embodiment of the present invention;

[0134] Figure 9 is a schematic structural diagram of another monocular infrared image target detection and depth estimation device disclosed in an embodiment of the present invention. Detailed implementation manners

[0135] To enable those skilled in the art to better understand the solutions of the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

[0136] In the description, claims, and above-mentioned drawings of the present invention, terms such as "first" and "second" are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, device, product, or equipment that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes steps or units not listed, or optionally further includes other steps or units inherent to these processes, methods, products, or equipment.

[0137] Reference to "embodiment" herein means that a specific feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the present invention. The phrase appears in various places in the specification and does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.

[0138] The present invention discloses a monocular infrared image target detection and depth estimation method and device, which can obtain a training data sample set and a training data sample label set; perform data preprocessing and data augmentation processing on the training data sample set to obtain a training data set; use a feature extraction network model to perform feature extraction on the training data set to obtain a feature map; perform splicing and fusion processing on the feature map to obtain a fused feature map; preset a target detection and depth estimation network model; use the fused feature map to train the target detection and depth estimation network model to obtain a trained target detection and depth estimation network model; obtain a data sample to be estimated, and use the trained target detection and depth estimation network model to process the data sample to be estimated to obtain a target detection and depth estimation result. The present invention improves the accuracy of infrared image depth estimation and realizes instance segmentation of infrared image targets while estimating the depth. The following will be described in detail respectively.

[0139] Embodiment 1

[0140] Please refer to Figure 1 , Figure 1 which is a schematic flowchart of a monocular infrared image target detection and depth estimation method disclosed in an embodiment of the present invention. Among them, Figure 1 The described monocular infrared image target detection and depth estimation method is applied to fields such as measurement, navigation, and autonomous driving, and is not limited in the embodiments of the present invention.

[0141] As Figure 1 shown, the monocular infrared image target detection and depth estimation method may include the following operations:

[0142] S1. Obtain a training data sample set and a training data sample label set; the training data sample set includes infrared images and depth information corresponding to the pixels of the infrared images;

[0143] The training data sample label set includes the category of the target, the target mask, the target position rectangle, the target depth information, and the image geometric information;

[0144] S2. Perform data preprocessing and data augmentation on the training data sample set to obtain a training data set;

[0145] S3. Use a feature extraction network model to extract features from the training data set to obtain a feature map; the feature map includes a downsampled 1 / 2 - fold feature map, a downsampled 1 / 4 - fold feature map, a downsampled 1 / 8 - fold feature map, a downsampled 1 / 16 - fold feature map, and a downsampled 1 / 32 - fold feature map;

[0146] S4. Use a feature fusion model to process the feature map to obtain a fused feature map;

[0147] The feature fusion model includes a Feature Pyramid Network (FPN) and a Path Aggregation Network (PAN);

[0148] The Feature Pyramid Network is composed of a Spatial Pyramid Pooling module (SPP), an upsampled 1 / 16 - fold module, a first channel concatenation module, a first convolutional layer, an upsampled 1 / 8 - fold module, and a second channel concatenation module connected in series;

[0149] The Path Aggregation Network is composed of a second convolutional layer, a downsampled 1 / 16 - fold module, a third channel concatenation module, a third convolutional layer, a downsampled 1 / 32 - fold module, and a fourth channel concatenation module connected in series;

[0150] S5. Process the fused feature map to obtain a target classification prediction result and a localization box regression prediction result;

[0151] S6. Process the fused feature map to obtain an instance segmentation and depth estimation result of the target;

[0152] S7. Use the target classification prediction result and the localization box regression prediction result, the instance segmentation and depth estimation result of the target, and the training data sample label set to calculate the loss, and use the backpropagation gradient descent method to minimize the loss to obtain a globally optimal target detection model and a globally optimal instance segmentation depth estimation model;

[0153] S8. Obtain the infrared image to be processed, and process the infrared image to be processed by using the globally optimal object detection model and the globally optimal instance segmentation depth estimation model to obtain the object classification prediction result, the localization box regression prediction result, the instance segmentation and depth estimation result of the object. Figure 2 It is a schematic diagram of the object detection and depth estimation network model disclosed in the embodiment of the present invention.

[0154] Optionally, obtain a number of data source samples of the infrared image required and the depth corresponding to the infrared image pixels through an infrared detector, and perform annotation processing on the data. The target position box (x, y, w, h) is the coordinate of the upper left corner of the rectangle in the image. The data contains infrared images with a sufficient number and a sufficient number of types (spring, summer, autumn, winter, sunny, rainy, cloudy, snowy). In this embodiment, there is only one category of the target, that is, to detect the person in the infrared image, so class ∈ {0, 1}, where 0 is the background and 1 is the target person. The target label in this embodiment includes the position and contour standard of the person in the infrared image and the depth value (distance from the camera) relative to the sensor. The target box label in the label data is the position marked manually in the image, and at the same time, the geometric information of the target person in the image is obtained; the depth information is collected by devices such as 3D radar and GPS, and the obtained depth information is stored corresponding to the target box information and geometric information; all the label data can also be the existing general data sets.

[0155] Optionally, for data preprocessing: based on the long side of the original image and keeping the aspect ratio, make up the black edge on the right side or the lower side, and scale it to a resolution of 512 * 512. Example: The image resolution is 480 * 320. Based on the long side of 480 and keeping the aspect ratio of 1:1, make up the black edge to 480 * 480, and then enlarge it to 512 * 512.

[0156] The image scaling ratio λ ∈ (0.75, 1.25).

[0157] Adjust the image to multiply the contrast and brightness of the source image by a random value θ ∈ (0.7, 1.3) to simulate the change of radiation attenuation.

[0158] The image rotation angle is α ∈ (-5°, 5°).

[0159] Optionally, performing data preprocessing and data augmentation processing on the training data sample set to obtain a training data set, including:

[0160] S21. Perform scaling processing and white edge filling processing on the training data sample set to obtain a preprocessed training data sample set;

[0161] S22. Randomly select M preprocessed training data samples from the preprocessed training data sample set, and perform λ-fold scaling on the M preprocessed training data samples and their corresponding target masks and target position bounding boxes to obtain M scaled preprocessed training data samples, where M is an integer greater than or equal to 1;

[0162] S23. Randomly select N preprocessed training data samples from the preprocessed training data sample set, and perform cropping on the N preprocessed training data samples and their corresponding target masks and target position bounding boxes to obtain N cropped preprocessed training data samples, where N is an integer greater than or equal to 1;

[0163] S24. Randomly extract P preprocessed training data samples from the preprocessed training data sample set, and adjust the brightness and contrast of the P preprocessed training data samples to obtain the first preprocessed training data sample set, where P is an integer greater than or equal to 1;

[0164] S25. Randomly extract Q preprocessed training data samples from the preprocessed training data sample set, and perform mirroring on the head images and their corresponding target masks and target position bounding boxes in the Q preprocessed training data samples to obtain a mirrored preprocessed training data sample set, where Q is an integer greater than or equal to 1;

[0165] S26. Randomly extract O preprocessed training data samples from the preprocessed training data sample set, and perform rotation on the head images and their corresponding target masks and target position bounding boxes in the O preprocessed training data samples to obtain a rotated preprocessed training data sample set, where O is an integer greater than or equal to 1;

[0166] S27. Add the M scaled preprocessed training data samples, N cropped preprocessed training data samples, the first preprocessed training data sample set, the mirrored preprocessed training data sample set, and the rotated preprocessed training data sample set to the preprocessed training data sample set to obtain a training data set.

[0167] Optionally, the feature extraction network model is composed of a cascade of a first downsampling module, a second downsampling module, a third downsampling module, a fourth downsampling module, and a fifth downsampling module, and each time the width and height of the image are downsampled by one-half;

[0168] Each downsampling module includes a convolutional downsampling module (Convolution→Batch-Normalization→Leaky-Relu, CBL), a dilated convolutional downsampling module, a residual module, and a channel concatenation module;

[0169] The convolutional downsampling module, the dilated convolutional downsampling module (Dilated Convolution→Batch–Normalization→Leaky-Relu, DCBL), and the residual module are in a parallel relationship;

[0170] The channel splicing module fuses the outputs of the convolutional downsampling module, the dilated convolutional downsampling module, and the residual module to obtain a fused output result;

[0171] The convolutional downsampling module is composed of a convolutional layer (Convolution), batch normalization (Batch-Normalization), and an activation function (Active-Function, Leaky-Relu) connected in series;

[0172] The dilated convolutional downsampling module is composed of a dilated convolutional layer (Dilated Convolution), batch normalization, and an activation function connected in series;

[0173] The residual module is composed of two convolutional modules connected in series, and the intermediate output is short-circuited to the output;

[0174] The dilated convolutional downsampling module is used to increase the receptive field of the convolutional downsampling feature map;

[0175] The residual module is used to prevent the vanishing gradient during the training process.

[0176] Optionally, using the feature fusion model to process the feature map to obtain a fused feature map includes:

[0177] S41, input the downsampled 1 / 32 times feature map P 32 into the spatial pyramid pooling module of the feature pyramid network to obtain a feature fusion map P1;

[0178] S42, upsample the feature fusion map P1 by 1 / 16 times to obtain an upsampled 1 / 16 times feature map;

[0179] S43, perform channel splicing on the upsampled 1 / 16 times feature map and the downsampled 1 / 16 times feature map P 16 and perform convolutional fusion to obtain a first feature fusion map P 16+ ;

[0180] S43, upsample the first feature fusion map P 16+ by 1 / 8 times to obtain an upsampled 1 / 8 times feature map;

[0181] S44, perform channel splicing on the upsampled 1 / 8 times feature map and the downsampled 1 / 8 times feature map P8 and perform convolutional fusion to obtain a second feature fusion map P8+ ;

[0182] S45, splice the second feature fusion map P 8+ with the downsampled 1 / 8 - fold feature map P8 in channels and perform convolutional fusion to obtain the fused downsampled 1 / 8 - fold feature map P 8* ;

[0183] S46, downsample the second feature fusion map P 8+ by 1 / 16 times to obtain the downsampled 1 / 16 - fold feature map;

[0184] S47, splice the downsampled 1 / 16 - fold feature map with the first feature fusion map P 16+ in channels and perform convolutional fusion to obtain the fused downsampled 1 / 16 - fold feature map P 16* ;

[0185] S48, downsample the fused downsampled 1 / 16 - fold feature map P 16* by 1 / 32 times to obtain the downsampled 1 / 32 - fold feature map;

[0186] S49, splice the downsampled 1 / 32 - fold feature map with the feature fusion map P1 in channels and perform convolutional fusion to obtain the fused downsampled 1 / 32 - fold feature map P 32* ;

[0187] The fused downsampled 1 / 8 - fold feature map P 8* , the fused downsampled 1 / 16 - fold feature map P 16* and the fused downsampled 1 / 32 - fold feature map P 32* constitute the fused feature map.

[0188] Optionally, processing the fused feature map to obtain the target classification prediction result and the localization box regression prediction result includes:

[0189] Using the target detection network loss function to process the fused feature map to obtain the target classification prediction result and the localization box regression prediction result;

[0190] The target detection network loss function is:

[0191] Loss = Loss clc + Loss conf + Loss loc

[0192] where Loss clc is the classification loss, using multi - class binary cross - entropy loss, Loss confFor the confidence loss, the CIOU method is adopted. In the CIOU method, IoU(BP, BGT) is used as the confidence, where BP is the predicted target box and BGT is the target box of the true label; Loss loc is the localization regression loss;

[0193] The localization regression loss Loss loc is:

[0194] Loss clc = (x - x*) 2 + (y - y*) 2 + (w - w*) 2 + (h - h*) 2

[0195] where x, y, w, h are the coordinate information of the predicted box, and x*, y*, w*, h* are the coordinate information of the true label box.

[0196] Optionally, the processing of the fused feature map to obtain the instance segmentation and depth estimation results of the target includes:

[0197] S61. According to the target classification prediction result and the localization box regression prediction result, process the fused sampled 1 / 8 - fold feature map P8*, the fused sampled 1 / 16 - fold feature map P16*, and the fused sampled 1 / 32 - fold feature map P32* to obtain the first feature map, the second feature map, and the third feature map with 1 / 8 - fold downsampling;

[0198] S62. Use the instance segmentation and depth estimation loss function to process the first feature map, the second feature map, and the third feature map with 1 / 8 - fold downsampling to obtain the instance segmentation and depth estimation results of the target.

[0199] Optionally, the instance segmentation and depth estimation loss function is:

[0200] Loss1 = Loss mask + Loss death + Loss ge o m

[0201] where Loss mask is the instance segmentation loss, and binary cross - entropy loss is adopted for each pixel. Loss death is the depth estimation loss, and the MSE loss function is adopted for the depth value. Loss geom is the geometric loss, and the Smooth - L1 loss function is adopted.

[0202] Target confidence loss:

[0203]

[0204]

[0205] Among them, o i ∈ {0, 1} indicates whether there is a target in the predicted target box i, represents the sigmoid probability that there is a target in the predicted target box i. c i represents the probability that there is a target in the predicted target box i.

[0206] Object classification loss:

[0207]

[0208]

[0209] Among them, O ij ∈ {0, 1} indicates whether there is a target of the j-th class in the predicted i-th target box, represents the sigmoid probability that there is a target of the j-th class in the predicted i-th target box, where C ij is the probability that there is a target of the j-th class in the i-th target box predicted by the network, which can be obtained through the sigmoid function

[0210] Object localization loss:

[0211]

[0212] σ(x) = Sigmoid(x)

[0213] is the offset of the center coordinates of the target box predicted by the model; is the width and height scaling factor of the target box predicted by the model; is the center point and width and height scaling factor of the true target box in the image, that is, the label.

[0214]

[0215]

[0216]

[0217]

[0218] Among them, grid cell represents the grid area of S×S (7×7, 13×13, 26×26) obtained by equally dividing the original image proportionally. The checkerboard coordinates correspond to the original image, as Figure 3 shown, and each square is called a grid cell. is the upper left corner coordinates of each grid cell, is the width and height of the model prediction box template, and the model will scale based on this during actual prediction.

[0219] Instance segmentation loss function:

[0220] Loss mask (p, q) = -∑p clc log2(q clc )

[0221] p clc is the probability that the predicted target pixel belongs to a certain class, and q clc = 1 - p clc . The instance segmentation loss is the sum of the binary cross-entropy of all pixels within the target box.

[0222] Depth estimation loss function:

[0223] The loss of depth estimation is composed of the depth loss and geometric loss of pixels, and the calculation is as follows:

[0224]

[0225] where n is the number of targets predicted in the current image; is the depth label value; is the geometric label value (which can be aspect ratio, pixel area, etc.). Figure 4 is a schematic diagram of the composition of the object detection and depth estimation network disclosed in the embodiments of the present invention; Figure 5 is a schematic diagram of the training process of the object detection and depth estimation network model disclosed in the embodiments of the present invention; Figure 6 is a schematic diagram of the input and output of the object detection and depth estimation network model disclosed in the embodiments of the present invention; during the training process, the loss calculation is divided into two stages. The first stage is the prediction of object classification and object box localization, and the second stage is the prediction of instance segmentation and depth information. The output of the first stage classification and localization prediction is the class, confidence, center point of the object box, and length and width (x, y, w, h). The loss functions used are the class loss Loss clc , the confidence loss Loss conf , and the localization loss Loss loc . The output of the second stage instance segmentation and depth information prediction is the contour mask (mask) of the predicted target and the depth information of the target. The loss functions used are the instance segmentation loss Loss mask (p, q), the depth loss Loss depth , and the geometric loss Loss geom。In this embodiment, the data passes through a network model and is trained using the stochastic gradient descent method for loss. When the training result reaches the optimal for the current dataset, the training is completed. If used subsequently, the infrared image can be input into the model, and the model will output the position, contour mask, and depth of the target person in the infrared image. Figure 7 is the instance segmentation and target depth estimation result disclosed in the embodiment of the present invention.

[0226] Embodiment 2

[0227] Please refer to Figure 8 , Figure 8 is a schematic structural diagram of a monocular infrared image target detection and depth estimation device disclosed in the embodiment of the present invention. Among them, Figure 8 The described monocular infrared image target detection and depth estimation device is applied to fields such as measurement, navigation, and autonomous driving, and the embodiment of the present invention does not make limitations.

[0228] As Figure 8 shown, the monocular infrared image target detection and depth estimation device may include the following operations:

[0229] S301, a data acquisition module, for acquiring a training data sample set and a training data sample label set; the training data sample set includes infrared images and depth information corresponding to the pixels of the infrared images;

[0230] The training data sample label set includes the category of the target, the target mask, the target position rectangle, the target depth information, and the image geometric information;

[0231] S302, a preprocessing module, for performing data preprocessing and data augmentation processing on the training data sample set to obtain a training data set;

[0232] S303, a feature extraction module, for using a feature extraction network model to perform feature extraction on the training data set to obtain a feature map; the feature map includes a downsampled 1 / 2 - fold feature map, a downsampled 1 / 4 - fold feature map, a downsampled 1 / 8 - fold feature map, a downsampled 1 / 16 - fold feature map, and a downsampled 1 / 32 - fold feature map;

[0233] S304, a feature fusion module, for using a feature fusion model to process the feature map to obtain a fused feature map;

[0234] The feature fusion model includes a feature pyramid network and a path aggregation network;

[0235] The feature pyramid network is composed of a spatial pyramid pooling module, an upsampled 1 / 16 - fold module, a first channel splicing module, a first convolutional layer, an upsampled 1 / 8 - fold module, and a second channel splicing module connected in series;

[0236] The path aggregation network is composed of a second convolutional layer, a downsampling 1 / 16 times module, a third channel splicing module, a third convolutional layer, a downsampling 1 / 32 times module, and a fourth channel splicing module connected in series;

[0237] S305, a prediction module, is used to process the fused feature map to obtain a target classification prediction result and a localization box regression prediction result;

[0238] S306, an estimation module, is used to process the fused feature map to obtain an instance segmentation and depth estimation result of the target;

[0239] S307, a training module, is used to calculate losses by using the target classification prediction result and the localization box regression prediction result, the instance segmentation and depth estimation result of the target, and the training data sample label set, and use the backpropagation gradient descent method to minimize the losses, so as to obtain a globally optimal target detection model and a globally optimal instance segmentation depth estimation model;

[0240] S308, a target detection and depth estimation module, is used to obtain an infrared image to be processed, and process the infrared image to be processed by using the globally optimal target detection model and the globally optimal instance segmentation depth estimation model to obtain a target classification prediction result and a localization box regression prediction result, and an instance segmentation and depth estimation result of the target.

[0241] Embodiment III

[0242] Please refer to Figure 9 , Figure 9 which is a schematic structural diagram of another monocular infrared image target detection and depth estimation device disclosed in the embodiments of the present invention. Among them, Figure 9 the described monocular infrared image target detection and depth estimation device is applied to fields such as measurement, navigation, and autonomous driving, which are not limited in the embodiments of the present invention. As Figure 9 shown, the monocular infrared image target detection and depth estimation device may include the following operations:

[0243] A memory 401 storing executable program code;

[0244] A processor 402 coupled to the memory 401;

[0245] The processor 402 calls the executable program code stored in the memory 401 and is used to execute the steps in the monocular infrared image target detection and depth estimation method described in Embodiment I.

[0246] Embodiment IV

[0247] An embodiment of the present invention discloses a computer-readable storage medium that stores a computer program for electronic data exchange. The computer program causes a computer to execute the steps in the monocular infrared image target detection and depth estimation method described in Embodiment 1.

[0248] The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed to multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without creative effort.

[0249] Through the above specific description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electrically-erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc memories, a magnetic disk memory, a tape memory, or any other computer-readable medium capable of carrying or storing data.

[0250] Finally, it should be noted that: The single - eye infrared image target detection and depth estimation method and device disclosed in the embodiments of the present invention only disclose the preferred embodiments of the present invention. It is only used to illustrate the technical solutions of the present invention, rather than limiting it; Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some of the technical features; And these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A monocular infrared image target detection and depth estimation method, characterized in that The method includes: S1. Obtain a training data sample set and a training data sample label set; the training data sample set includes infrared images and depth information corresponding to the pixels of the infrared images; The training data sample label set includes the category of the target, the target mask, the target position rectangle, the target depth information, and the image geometric information; S2. Perform data preprocessing and data augmentation on the training data sample set to obtain a training data set; S3. Use a feature extraction network model to extract features from the training data set to obtain a feature map; the feature map includes a feature map downsampled by 1 / 2 times, a feature map downsampled by 1 / 4 times, a feature map downsampled by 1 / 8 times, a feature map downsampled by 1 / 16 times, and a feature map downsampled by 1 / 32 times; S4. Use a feature fusion model to process the feature map to obtain a fused feature map; The feature fusion model includes a feature pyramid network and a path aggregation network; The feature pyramid network is composed of a spatial pyramid pooling module, an upsampling by 1 / 16 times module, a first channel splicing module, a first convolutional layer, an upsampling by 1 / 8 times module, and a second channel splicing module connected in series; The path aggregation network is composed of a second convolutional layer, a downsampling by 1 / 16 times module, a third channel splicing module, a third convolutional layer, a downsampling by 1 / 32 times module, and a fourth channel splicing module connected in series; S5. Process the fused feature map to obtain a target classification prediction result and a localization box regression prediction result; S6. Process the fused feature map to obtain an instance segmentation and depth estimation result of the target; S7. Use the target classification prediction result and the localization box regression prediction result, the instance segmentation and depth estimation result of the target, and the training data sample label set to calculate the loss, and use the backpropagation gradient descent method to minimize the loss to obtain a globally optimal target detection model and a globally optimal instance segmentation depth estimation model; S8. Obtain an infrared image to be processed, and use the globally optimal target detection model and the globally optimal instance segmentation depth estimation model to process the infrared image to be processed to obtain a target classification prediction result and a localization box regression prediction result, and an instance segmentation and depth estimation result of the target.

2. The monocular infrared image target detection and depth estimation method according to claim 1, characterized in that, The performing data preprocessing and data augmentation on the training data sample set to obtain a training data set includes: S21. Perform scaling processing and white edge filling processing on the training data sample set to obtain a preprocessed training data sample set; S22. Randomly select M preprocessed training data samples from the preprocessed training data sample set, and perform λ-fold scaling processing on the M preprocessed training data samples and the corresponding target mask and target position rectangle to obtain M scaled preprocessed training data samples, where M is an integer greater than or equal to 1; S23. Randomly select N preprocessed training data samples from the preprocessed training data sample set, and perform cropping processing on the N preprocessed training data samples and the corresponding target mask and target position rectangle to obtain N cropped preprocessed training data samples, where N is an integer greater than or equal to 1; S24. Randomly select P pre - processed training data samples from the pre - processed training data sample set, and perform brightness and contrast adjustment on the P pre - processed training data samples to obtain the first pre - processed training data sample set, where P is an integer greater than or equal to 1; S25. Randomly select Q pre - processed training data samples from the pre - processed training data sample set, and perform mirror processing on the head images, corresponding target masks, and target position rectangular frames in the Q pre - processed training data samples to obtain the mirror - pre - processed training data sample set, where Q is an integer greater than or equal to 1; S26. Randomly select O pre - processed training data samples from the pre - processed training data sample set, and perform rotation processing on the head images, corresponding target masks, and target position rectangular frames in the O pre - processed training data samples to obtain the rotation - pre - processed training data sample set, where O is an integer greater than or equal to 1; S27. Add the M down - sampled pre - processed training data samples, N cropped pre - processed training data samples, the first pre - processed training data sample set, the mirror - pre - processed training data sample set, and the rotation - pre - processed training data sample set to the pre - processed training data sample set to obtain the training data set.

3. The monocular infrared image target detection and depth estimation method according to claim 1, characterized in that The feature extraction network model is composed of a first down - sampling module, a second down - sampling module, a third down - sampling module, a fourth down - sampling module, and a fifth down - sampling module connected in series. Each time, the width and height of the image are down - sampled by one - half; Each down - sampling module includes a convolutional down - sampling module, a dilated convolutional down - sampling module, a residual module, and a channel splicing module; The convolutional down - sampling module, the dilated convolutional down - sampling module, and the residual module are in a parallel relationship; The channel splicing module fuses the outputs of the convolutional down - sampling module, the dilated convolutional down - sampling module, and the residual module to obtain a fused output result; The convolutional down - sampling module is composed of a convolutional layer, batch normalization, and an activation function connected in series; The dilated convolutional down - sampling module is composed of a dilated convolutional layer, batch normalization, and an activation function connected in series; The residual module is composed of two convolutional modules connected in series, and the intermediate output is short - circuited to the output; The dilated convolutional down - sampling module is used to increase the receptive field of the convolutional down - sampling feature map; The residual module is used to prevent gradient disappearance during the training process.

4. The monocular infrared image target detection and depth estimation method according to claim 1, characterized in that Processing the feature map using the feature fusion model to obtain a fused feature map includes: S41, input the downsampled 1 / 32 - fold feature map P 32 into the spatial pyramid pooling module of the feature pyramid network to obtain a feature fusion map P1; S42. Upsample the feature fusion map P1 by 1 / 16 times to obtain the upsampled 1 / 16 - times feature map; S43, concatenate the upsampled 1 / 16 - fold feature map and the downsampled 1 / 16 - fold feature map P 16 in the channel dimension and perform convolutional fusion to obtain the first feature fusion map P 16+ ; S43, upsample the first feature fusion map P 16+ by a factor of 1 / 8 to obtain an upsampled feature map with a factor of 1 / 8; S44, perform channel concatenation on the upsampled 1 / 8 - fold feature map and the downsampled 1 / 8 - fold feature map P8, and perform convolution fusion to obtain a second feature fusion map P 8+ ; S45, concatenate the second feature fusion map P 8+ with the downsampled 1 / 8 - fold feature map P8 in the channel dimension and perform convolutional fusion to obtain the fused downsampled 1 / 8 - fold feature map P 8* ; S46, downsample the second feature fusion graph P 8+ by 1 / 16 times to obtain a downsampled feature map with a 1 / 16 times reduction; S47, combine the downsampled 1 / 16 - fold feature map with the first feature fusion map P 16+ perform channel concatenation and convolution fusion to obtain the fused downsampled 1 / 16 - fold feature map P 16* ; S48, downsample the fused sampling 1 / 16 - fold feature map P 16* by 1 / 32 - fold to obtain a 1 / 32 - fold downsampled feature map; S49, the 1 / 32-downsampled feature map is concatenated with the feature fusion map P1 in the channel dimension and then undergoes convolutional fusion to obtain the fused 1 / 32-downsampled feature map P 32* ; The fused sampled 1 / 8 - fold feature map P 8* , the fused sampled 1 / 16 - fold feature map P 16* and the fused sampled 1 / 32 - fold feature map P 32* constitute the fused feature map.

5. The monocular infrared image target detection and depth estimation method according to claim 1, characterized in that Processing the fused feature map to obtain the target classification prediction result and the localization box regression prediction result includes: Using the target detection network loss function to process the fused feature map to obtain the target classification prediction result and the localization box regression prediction result; The target detection network loss function is: Loss=Loss clc +Loss conf +Loss loc Among them, Loss clc is the classification loss, and the multi-class binary cross-entropy loss is adopted. Loss conf is the confidence loss, and the CIOU method is adopted. In the CIOU method, IoU(BP, BGT) is used as the confidence, where BP is the predicted target box and BGT is the target box of the true label; Loss loc is the localization regression loss; The localization regression loss Loss loc is as follows: Loss clc =(x - x*) 2 +(y - y*) 2 +(w - w*) 2 +(h - h*) 2 Where x, y, w, h are the coordinate information of the predicted box, and x*, y*, w*, h* are the coordinate information of the true label box.

6. The monocular infrared image target detection and depth estimation method according to claim 4, characterized in that Processing the fused feature map to obtain the instance segmentation and depth estimation results of the target includes: S61. According to the target classification prediction result and the localization box regression prediction result, process the fused sampling 1 / 8 - fold feature map P8*, the fused sampling 1 / 16 - fold feature map P16*, and the fused sampling 1 / 32 - fold feature map P32* to obtain the first feature map, the second feature map, and the third feature map that are downsampled by 1 / 8 - fold; S62. Use the instance segmentation and depth estimation loss function to process the first feature map, the second feature map, and the third feature map that are downsampled by 1 / 8 - fold to obtain the instance segmentation and depth estimation results of the target.

7. The monocular infrared image target detection and depth estimation method according to claim 6, characterized in that The instance segmentation and depth estimation loss function is: Loss1=Loss mask +Loss death +Loss geom Among them, Loss mask is the instance segmentation loss, and the binary cross-entropy loss is adopted for each pixel. Loss death is the depth estimation loss, and the MSE loss function is adopted for the depth value. Loss geom is the geometric loss, and the Smooth-L1 loss function is adopted.

8. A monocular infrared image target detection and depth estimation device, characterized in that, The device includes: A data acquisition module, configured to acquire a training data sample set and a training data sample label set; the training data sample set includes infrared images and depth information corresponding to the pixels of the infrared images; The training data sample label set includes the category of the target, the target mask, the target position rectangle box, the target depth information, and the image geometry information; A pre - processing module, configured to perform data pre - processing and data augmentation processing on the training data sample set to obtain a training data set; A feature extraction module, configured to use a feature extraction network model to extract features from the training data set to obtain feature maps; the feature maps include a downsampled 1 / 2 - fold feature map, a downsampled 1 / 4 - fold feature map, a downsampled 1 / 8 - fold feature map, a downsampled 1 / 16 - fold feature map, and a downsampled 1 / 32 - fold feature map; A feature fusion module, configured to use a feature fusion model to process the feature maps to obtain fused feature maps; The feature fusion model includes a feature pyramid network and a path aggregation network; The feature pyramid network is composed of a spatial pyramid pooling module, an upsampling 1 / 16 - fold module, a first channel splicing module, a first convolutional layer, an upsampling 1 / 8 - fold module, and a second channel splicing module connected in series; The path aggregation network is composed of a second convolutional layer, a downsampling 1 / 16 - fold module, a third channel splicing module, a third convolutional layer, a downsampling 1 / 32 - fold module, and a fourth channel splicing module connected in series; A prediction module, configured to process the fused feature maps to obtain a target classification prediction result and a localization box regression prediction result; An estimation module, configured to process the fused feature maps to obtain the instance segmentation and depth estimation results of the target; A training module, configured to use the target classification prediction result and the localization box regression prediction result, the instance segmentation and depth estimation results of the target, and the training data sample label set to calculate the loss, and use the backpropagation gradient descent method to minimize the loss to obtain a globally optimal target detection model and a globally optimal instance segmentation depth estimation model; A target detection and depth estimation module, configured to acquire an infrared image to be processed, and use the globally optimal target detection model and the globally optimal instance segmentation depth estimation model to process the infrared image to be processed to obtain a target classification prediction result, a localization box regression prediction result, and the instance segmentation and depth estimation results of the target.

9. A monocular infrared image target detection and depth estimation device, characterized in that, The device includes: A memory storing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory and executes the monocular infrared image target detection and depth estimation method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer storage medium stores computer instructions which, when called, are used to execute the monocular infrared image target detection and depth estimation method according to any one of claims 1-7.

Images