Target detection method, system and equipment based on thermal imager and deep learning

Abstract

The present invention belongs to the field of target detection technology, and specifically relates to a target detection method, system and equipment based on thermal imagers and deep learning, which aims to solve the problem of low accuracy and precision of target detection due to the special environment of coal mining. The present invention includes: synchronously collecting visible light and infrared light video images of mine working conditions, as well as dust concentration: if the dust concentration of the mine working conditions is lower than the set threshold, extracting the high-level multi-scale feature map group and the low-level multi-scale feature map group of the image collected by the visible light imager, and identifying the target detection result; otherwise, extracting the thermal imaging contour and position, obtaining the alternative target detection result, constructing a multi-scale pyramid pooling residual network, extracting the features of the corresponding visible light image, and screening to obtain the final target detection result. The present invention can adapt to the environment of high dust concentration and dim vision during coal mining production, automatically and accurately perform high-precision target detection, and avoid safety accidents caused by inadequate supervision.

Description

Technical Field

[0001] The present invention belongs to the field of target detection technology, and specifically relates to a target detection method, system and equipment based on thermal imagers and deep learning. Background Art

[0002] With the increasing mechanization of coal mining equipment, the number of machines on tunneling faces has increased. During production operations, the constant back-and-forth movement of equipment poses significant safety risks to personnel working on the tunneling face. This is especially true during production, where high dust concentrations, poor visibility, and loud noise create significant blind spots during equipment movement, making collisions between equipment and personnel highly likely. Furthermore, due to the large number of operators and the lack of self-discipline among some employees, inadequate supervision often leads to safety accidents.

[0003] With the advancement and development of science and technology, the use of modern monitoring equipment to replace traditional manual monitoring has become a feature of the modern coal mining industry. Among them, target detection is an important part of coal mine safety detection. Most existing technologies use the following methods for target detection: first, a background model is obtained using statistical methods, and the background model is updated in real time to adapt to changes in light and the scene itself. Morphological methods and the detection of connected domain areas are used for post-processing to eliminate the influence of noise and background disturbances. Shadows are detected in the HSV color space to obtain accurate moving targets.

[0004] However, during coal mining operations, dust concentrations are high and visibility is dim. Simply capturing images with visible light cameras for target detection can lead to insufficient texture information due to environmental factors, resulting in low target detection sensitivity and the risk of false positives and missed detections. Furthermore, dust obstructs monitoring footage from certain scenes and time periods, preventing accurate real-time recording of the current video footage.

[0005] If only infrared cameras are used for target detection, there is a possibility that some human-like, hot objects may be misidentified as humans due to their imaging characteristics, making it difficult to ensure the safety of personnel and equipment in coal mining operations. Summary of the Invention

[0006] In order to solve at least one of the above-mentioned technical problems existing in the prior art, the present invention provides a target detection method, system and device based on thermal imager and deep learning.

[0007] The present invention is implemented using the following technical solution: a target detection method based on thermal imager and deep learning, comprising the following steps:

[0008] S10: Collecting the dust concentration of the mine working condition using a dust monitor, determining and executing: If the dust concentration of the mine working condition is lower than a set threshold, collecting a video image of the mine working condition using a visible light imager, and skipping to step S20; otherwise, collecting a video image of the mine working condition using a thermal imager and a visible light imager simultaneously, and skipping to step S30;

[0009] S20: Use a deep convolutional neural network to extract high-level and low-level multi-scale feature map groups from images captured by the visible light imager, perform iterative feedback fusion of the feature map groups, and combine the complementary relationships between the features in the fused feature map groups to obtain the position of the target detection bounding box in the image and identify the target detection results.

[0010] S30: extracting the thermal imaging contour of the image captured by the thermal imager and the position of the thermal imaging contour in the image, and obtaining the candidate target detection result;

[0011] S40: Construct a multi-scale pyramid pooling residual network to extract features of the image captured by the visible light imager corresponding to the image captured by the thermal imager, and filter the candidate target detection results through a fully connected network to obtain a final target detection result.

[0012] Preferably, the method for performing iterative feedback fusion of the feature map group in step S20 is:

[0013] Establish Gaussian pyramids and Laplacian pyramids corresponding to each feature map in the high-level multi-scale feature map group;

[0014] The Gaussian pyramid and Laplacian pyramid are fused with the corresponding low-level multi-scale feature maps in the feature channel dimension.

[0015] Preferably, between obtaining the position of the target detection bounding box in the image and identifying the target detection result in step S20, a step of fine-tuning the bounding box is further provided:

[0016] B10: Combined with the gradient function, the position loss of the bounding box is calculated with the goal of minimizing the deviation between the predicted bounding box output by the deep convolutional neural network and the corresponding true bounding box;

[0017] B20: Calculate the position adjustment amount of the bounding box based on the calculated position loss of the bounding box, and adjust the position of the bounding box.

[0018] Preferably, the position loss of the bounding box is calculated as follows:

[0019]

[0020]

[0021] Among them, F (A,B)is the position loss of the bounding box, A is the predicted bounding box output by the deep convolutional neural network, area(A) is the area of ​​bounding box A, B is the corresponding real bounding box, area(B) is the area of ​​bounding box B, ∩ represents the intersection, ∪ represents the union, d0 represents the Euclidean distance between the center points of bounding box A and bounding box B, d c is the distance of the diagonal of the bounding box B, ω * is the penalty factor, F IoU_Loss is the expression of the IoU_Loss function.

[0022] Preferably, the penalty factor ω * The parameter acquisition formula based on measuring the aspect ratio of bounding box A and bounding box B is:

[0023]

[0024]

[0025] Among them, ν is a parameter that measures the aspect ratio of the bounding box A and the bounding box B, w gt and h gt Corresponding to the width and height of the real bounding box B, w p and h p They correspond to the width and height of the predicted bounding box A respectively.

[0026] Preferably, step S30 includes:

[0027] The thermal imaging contour of the image captured by the thermal imager is extracted by an adaptive edge detection method, and the position of the thermal imaging contour in the image is calculated;

[0028] The similarity between the extracted contour and each target image pre-stored in the infrared thermal imaging library is calculated respectively, and the target image category and the corresponding thermal imaging contour position with a similarity greater than the set value are taken as the candidate target detection results.

[0029] Preferably, the method for constructing the multi-scale pyramid pooling residual network in step S40 is:

[0030] The convolutional layers of the deep convolutional neural network are replaced with dilated convolutional layers of a set scale, and the last average pooling layer is replaced with a multi-scale pyramid pooling layer.

[0031] A multi-layer residual block is added between the set layers of the network after updating the convolutional layer and the pooling layer to construct a multi-scale pyramid pooling residual network.

[0032] Preferably, the multi-layer residual block is a two-layer residual structure or a three-layer residual structure;

[0033] The two-layer residual structure consists of a first convolutional layer with a kernel of 3×3 and a second convolutional layer with a kernel of 3×3, connected in sequence. The input of the first convolutional layer and the output of the second convolutional layer are spliced ​​together.

[0034] The three-layer residual structure consists of a third convolutional layer with a kernel of 1×1, a fourth convolutional layer with a kernel of 3×3, and a fifth convolutional layer with a kernel of 1×1, which are connected sequentially. The input of the third convolutional layer and the output of the fifth convolutional layer are spliced ​​together.

[0035] The present invention provides a target detection system based on thermal imagers and deep learning, including a data acquisition module configured to acquire video images of mine working conditions through a visible light imager under different working conditions, or to acquire video images of mine working conditions through a thermal imager and a visible light imager simultaneously;

[0036] The dust concentration detection and jump module collects the dust concentration of the mine working condition through the dust monitor, judges and executes: if the dust concentration of the mine working condition is lower than the set threshold, the data acquisition module collects the video image of the mine working condition through the visible light imager and jumps to the first target detection module; otherwise, the data acquisition module collects the video image of the mine working condition through the thermal imager and the visible light imager synchronously, and jumps to the second target detection module;

[0037] The first target detection module is configured to use a deep convolutional neural network to extract a high-level multi-scale feature map group and a low-level multi-scale feature map group from an image captured by a visible light imager, perform iterative feedback fusion of the feature map groups, and combine the complementary relationship between the features in the fused feature map groups to obtain the position of the target detection bounding box in the image and identify the target detection result;

[0038] The second target detection module is configured to extract the thermal imaging contour and the position of the thermal imaging contour in the image from the image captured by the thermal imager, obtain the candidate target detection results, construct a multi-scale pyramid pooling residual network, extract the features of the image captured by the visible light imager corresponding to the image captured by the thermal imager, and filter the candidate target detection results through the fully connected network to obtain the final target detection result.

[0039] The present invention also provides a storage device storing a plurality of programs, wherein the programs are suitable for being loaded and executed by a processor to implement a target detection method based on a thermal imager and deep learning.

[0040] The present invention also provides a processing device, including a processor suitable for executing various programs; and a storage device suitable for storing multiple programs; the above programs are suitable for being loaded and executed by the processor to implement a target detection method based on a thermal imager and deep learning.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] (1) The target detection method based on thermal imager and deep learning of the present invention improves the target recognition rate of the downhole monitoring system by fusing the image features of visible light camera and thermal imager.

[0043] (2) The target detection method of the present invention is based on thermal imagers and deep learning. By fusing the video information of visible light and thermal imagers, it can realize real-time monitoring of all working conditions of the underground environment without being affected by environmental factors such as light intensity and dust concentration.

[0044] (3) The target detection method based on thermal imagers and deep learning of the present invention comprehensively processes the image features of different image acquisition devices deployed in the same scene, and uses a multi-scale feature map group to train and optimize the parameters of the deep learning model, thereby improving the accuracy of target recognition and reducing the false alarm rate, thereby improving the personnel recognition accuracy of the underground monitoring system. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0046] Figure 1 Schematic diagram of the process of target detection method based on thermal imager and deep learning of the present invention;

[0047] Figure 2 This is a schematic diagram of constructing a Gaussian pyramid according to an embodiment of a target detection method based on a thermal imager and deep learning according to the present invention. DETAILED DESCRIPTION

[0048] The technical solutions in the embodiments of the present invention are clearly and completely described in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other implementations derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0049] It should be noted that the structures, proportions, sizes, etc. illustrated in the drawings of this specification are only used to match the contents disclosed in the specification for people familiar with this technology to understand and read, and are not used to limit the conditions under which the present invention can be implemented. Therefore, they have no substantive technical significance. Any structural modification, change in proportional relationship or adjustment of size should fall within the scope of the technical content disclosed in the present invention without affecting the efficacy and purpose that can be achieved by the present invention. It should be noted that in this specification, relational terms such as first and second are only used to distinguish one entity from several other entities, and do not necessarily require or imply any actual relationship or order between these entities.

[0050] In order to more clearly illustrate the target detection method based on thermal imager and deep learning of the present invention, the following Figure 1 Each step in the embodiment of the present invention is described in detail.

[0051] The target detection method based on a thermal imager and deep learning according to the first embodiment of the present invention includes the following steps:

[0052] S10: Collecting the dust concentration of the mine working condition using a dust monitor, determining and executing: If the dust concentration of the mine working condition is lower than a set threshold, collecting a video image of the mine working condition using a visible light imager, and skipping to step S20; otherwise, collecting a video image of the mine working condition using a thermal imager and a visible light imager simultaneously, and skipping to step S30;

[0053] In step S10, if the dust concentration in the mine working condition is lower than the set threshold, that is, the current working condition is relatively good and the clarity of the visible light image is sufficient to support the accuracy and precision of the target detection result, only the visible light imager is started to collect the video image of the mine working condition.

[0054] S20: Use a deep convolutional neural network to extract high-level and low-level multi-scale feature map groups from images captured by the visible light imager, perform iterative feedback fusion of the feature map groups, and combine the complementary relationships between the features in the fused feature map groups to obtain the position of the target detection bounding box in the image and identify the target detection results.

[0055] The method for iterative feedback fusion of feature map groups in step S20 is:

[0056] Establish Gaussian pyramids and Laplacian pyramids corresponding to each feature map in the high-level multi-scale feature map group;

[0057] An image pyramid is a collection of images arranged in a pyramidal shape, each derived from the same original image, with decreasing resolution. An image pyramid is constructed by sequential downsampling until a predetermined termination condition is reached. The base of the pyramid is a high-resolution representation of the image being processed, while the top is a low-resolution approximation. These layers of images can be likened to a pyramid: the higher the level, the smaller the image and the lower the resolution.

[0058] The Gaussian pyramid is a series of images obtained by downsampling the maximum resolution image at the bottom. The bottom image has the highest resolution, and the resolution of the images gets lower as you go up. Figure 2 FIG. 1 is a schematic diagram showing the construction of a Gaussian pyramid according to an embodiment of a target detection method based on a thermal imager and deep learning according to the present invention. The downsampling process is as follows:

[0059] (1) For a given image, a Gaussian smoothing process is first performed, that is, a Gaussian convolution layer with a set kernel is used to perform a convolution operation on the image;

[0060] (2) Then the image is sampled again, even rows and even columns are removed from the image, and a Gaussian downsampled image is obtained;

[0061] (3) Repeat operations (1) and (2) on this image until the set stopping condition is reached to obtain a Gaussian pyramid.

[0062] For the obtained Gaussian pyramid, if the image is only expanded by 2 times in each direction, the newly added rows and columns are filled with 0, and then the enlarged image is convolved with the same kernel as the Gaussian convolution layer, the approximate value of the "newly added pixels" can be obtained. The obtained image is the upsampled image. This cycle is repeated until an image with the same resolution as the original image is obtained.

[0063] However, during the downsampling and upsampling process, some information is lost, and this information may contain key information for subsequent target detection.

[0064] Therefore, for the top layer of the Gaussian pyramid obtained by downsampling, upsampling is performed through the Laplace transform prediction method, that is, the downsampling residual of the corresponding pyramid layer is predicted by Laplace. The residual of each layer constitutes the Laplace pyramid. The image is restored by combining the Laplace pyramid and upsampling. The final image can be restored to the original given image to the greatest extent.

[0065] The Gaussian pyramid and Laplacian pyramid are fused with the corresponding low-level multi-scale feature maps in the feature channel dimension to obtain a multi-scale fused feature map group.

[0066] Between obtaining the position of the target detection bounding box in the image and identifying the target detection result in step S20, a step of fine-tuning the bounding box is also provided:

[0067] B10: Combined with the gradient function, the position loss of the bounding box is calculated with the goal of minimizing the deviation between the predicted bounding box output by the deep convolutional neural network and the corresponding true bounding box;

[0068] B20: Calculate the position adjustment amount of the bounding box based on the calculated position loss of the bounding box, and adjust the position of the bounding box.

[0069] The position loss of the bounding box is calculated as:

[0070]

[0071]

[0072] Among them, F (A,B) is the position loss of the bounding box, A is the predicted bounding box output by the deep convolutional neural network, area(A) is the area of ​​bounding box A, B is the corresponding real bounding box, area(B) is the area of ​​bounding box B, ∩ represents the intersection, ∪ represents the union, d0 represents the Euclidean distance between the center points of bounding box A and bounding box B, d c is the distance of the diagonal of the bounding box B, ω * is the penalty factor, F IoU_Loss is the expression of the IoU_Loss function.

[0073] Penalty factor ω * The parameter acquisition formula based on measuring the aspect ratio of bounding box A and bounding box B is:

[0074]

[0075]

[0076] Among them, ν is a parameter that measures the aspect ratio of the bounding box A and the bounding box B, w gt and h gt Corresponding to the width and height of the real bounding box B, w p and h p They correspond to the width and height of the predicted bounding box A respectively.

[0077] Step B20: Calculate the position adjustment amount of the bounding box according to the calculated position loss of the bounding box, and adjust the position of the bounding box.

[0078] S30: extract the thermal imaging contour of the image captured by the thermal imager and the position of the thermal imaging contour in the image, and obtain the alternative target detection result; if the dust concentration of the mine working condition is not lower than the set threshold, that is, the current working condition is relatively poor, and the clarity of the visible light image is insufficient to support the accuracy and precision of the target detection result, it is necessary to synchronously start the thermal imager and the visible light imager to collect the video image of the mine working condition.

[0079] Step S30 extracts the thermal imaging profile of the image captured by the thermal imager and the position of the thermal imaging profile in the image, and obtains the candidate target detection result:

[0080] The thermal imaging contour of the image captured by the thermal imager is extracted by an adaptive edge detection method, and the position of the thermal imaging contour in the image is calculated;

[0081] The similarity between the extracted contour and each target image pre-stored in the infrared thermal imaging library is calculated respectively, and the target image category and the corresponding thermal imaging contour position with a similarity greater than the set value are taken as the candidate target detection results.

[0082] S40: Construct a multi-scale pyramid pooling residual network to extract features of the image captured by the visible light imager corresponding to the image captured by the thermal imager, and filter the candidate target detection results through a fully connected network to obtain a final target detection result.

[0083] The method for constructing the multi-scale pyramid pooling residual network in step S40 is:

[0084] The convolutional layers of the deep convolutional neural network are replaced with dilated convolutional layers of a set scale, and the last average pooling layer is replaced with a multi-scale pyramid pooling layer.

[0085] The dilated convolution can be achieved by filling the convolution kernel of the standard convolution with 0, or by sampling the input at equal intervals.

[0086] The dilation ratio of the dilated convolution can be set according to the actual application. In one embodiment of the present invention, the dilation ratio of the dilated convolution is a combination of r=1 / 2 / 3, that is, each dilated convolution performs a convolution operation on the feature with a different dilation ratio (r=1, r=2, r=3), and the results of the operation are fused as the processing result of the dilated convolution layer.

[0087] In one embodiment of the present invention, the multi-scale pyramid pooling layer has three scales, dividing the features transmitted by the previous layer of the network into three scales. The first scale is the feature map with the same original feature, the second scale is to divide the original feature map into 2×2=4 feature maps of the same size, and the third scale is to divide the original feature map into 4×4=16 feature maps of the same size. In this way, the features transmitted by the previous layer of the network become 21 feature maps.

[0088] A multi-layer residual block is added between the set layers of the network after updating the convolutional layer and the pooling layer to construct a multi-scale pyramid pooling residual network.

[0089] The multi-layer residual block is a two-layer residual structure or a three-layer residual structure;

[0090] The two-layer residual structure consists of a first convolutional layer with a kernel of 3×3 and a second convolutional layer with a kernel of 3×3, connected in sequence. The input of the first convolutional layer and the output of the second convolutional layer are spliced ​​together.

[0091] The three-layer residual structure consists of a third convolutional layer with a kernel of 1×1, a fourth convolutional layer with a kernel of 3×3, and a fifth convolutional layer with a kernel of 1×1, which are connected sequentially. The input of the third convolutional layer and the output of the fifth convolutional layer are spliced ​​together.

[0092] The loss function used in the training of the multi-scale pyramid pooling residual network is:

[0093]

[0094] Among them, m is the number of training samples in network training, p is the number of target detection categories of training samples, and x i is the feature corresponding to the i-th training sample, y i is the training sample x i The label, W and b are the weight matrix and bias matrix of the network respectively, T represents the matrix transformation; j is the number of the jth category of the target detection of the training sample; e is the natural logarithm.

[0095] Although the various steps in the above embodiment are described in the above-mentioned order, those skilled in the art will understand that in order to achieve the effect of this embodiment, different steps do not have to be executed in such an order. They can be executed simultaneously (in parallel) or in a reverse order. These simple changes are within the scope of protection of the present invention.

[0096] A second embodiment of the present invention provides a target detection system based on a thermal imager and deep learning, including a data acquisition module configured to acquire video images of mine working conditions using a visible light imager under different working conditions, or to acquire video images of mine working conditions using a thermal imager and a visible light imager simultaneously;

[0097] The dust concentration detection and jump module collects the dust concentration of the mine working condition through the dust monitor, judges and executes: if the dust concentration of the mine working condition is lower than the set threshold, the data acquisition module collects the video image of the mine working condition through the visible light imager and jumps to the first target detection module; otherwise, the data acquisition module collects the video image of the mine working condition through the thermal imager and the visible light imager synchronously, and jumps to the second target detection module;

[0098] The first target detection module is configured to use a deep convolutional neural network to extract a high-level multi-scale feature map group and a low-level multi-scale feature map group from an image captured by a visible light imager, perform iterative feedback fusion of the feature map groups, and combine the complementary relationship between the features in the fused feature map groups to obtain the position of the target detection bounding box in the image and identify the target detection result;

[0099] The second target detection module is configured to extract the thermal imaging contour and the position of the thermal imaging contour in the image from the image captured by the thermal imager, obtain the candidate target detection results, construct a multi-scale pyramid pooling residual network, extract the features of the image captured by the visible light imager corresponding to the image captured by the thermal imager, and filter the candidate target detection results through the fully connected network to obtain the final target detection result.

[0100] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working process and related instructions of the system described above can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0101] It should be noted that the target detection system based on thermal imagers and deep learning provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the modules or steps in the embodiments of the present invention can be decomposed or combined. For example, the modules in the above embodiment can be combined into one module, or further divided into multiple sub-modules to complete all or part of the functions described above. The names of the modules and steps involved in the embodiments of the present invention are only for distinguishing the modules or steps and are not regarded as improper limitations on the present invention.

[0102] An electronic device according to a third embodiment of the present invention includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the processor, and the instructions are used to be executed by the processor to implement the above-mentioned target detection method based on thermal imagers and deep learning.

[0103] A computer-readable storage medium according to a fourth embodiment of the present invention stores computer instructions, which are used to be executed by a computer to implement the above-mentioned target detection method based on thermal imager and deep learning.

[0104] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes and related instructions of the storage device and processing device described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0105] Those skilled in the art should be able to appreciate that, in conjunction with the modules and method steps of each example described in the embodiments disclosed herein, it is possible to implement them with electronic hardware, computer software, or a combination of the two, and the programs corresponding to the software modules and method steps can be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art. In order to clearly illustrate the interchangeability of electronic hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0106] The terms "first", "second", etc. are used to distinguish similar objects, rather than to describe or indicate a particular order or sequence.

[0107] The term "comprise" or any other similar term is intended to cover non-exclusive inclusion such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0108] Thus far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art may make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will fall within the scope of protection of the present invention.

Claims

1. A target detection method based on thermal imager and deep learning, characterized in that: The following steps are involved: S10: Collecting the dust concentration of the mine working condition using a dust monitor, determining and executing: If the dust concentration of the mine working condition is lower than a set threshold, collecting a video image of the mine working condition using a visible light imager, and skipping to step S20; otherwise, collecting a video image of the mine working condition using a thermal imager and a visible light imager simultaneously, and skipping to step S30; S20: Using a deep convolutional neural network to extract a high-level multi-scale feature map group and a low-level multi-scale feature map group of the image captured by the visible light imager, performing iterative feedback fusion of the feature map groups, combining the complementary relationship between the features in the fused feature map groups, obtaining the position of the target detection bounding box in the image, and identifying the target detection result; after this step is completed, steps S30 and S40 are not continued; S30: extracting a thermal imaging contour of the image captured by the thermal imager by using an adaptive edge detection method, and calculating a position of the thermal imaging contour in the image; Calculate the similarity between the extracted contour and each target image pre-stored in the infrared thermal imaging library respectively, and use the target image category and corresponding thermal imaging contour position with a similarity greater than the set value as the candidate target detection result; S40: Construct a multi-scale pyramid pooling residual network to extract features of the image captured by the visible light imager corresponding to the image captured by the thermal imager, and filter the candidate target detection results through a fully connected network to obtain a final target detection result.

2. The target detection method based on thermal imager and deep learning according to claim 1, characterized in that: The method for iterative feedback fusion of feature map groups in step S20 is: Establish Gaussian pyramids and Laplacian pyramids corresponding to each feature map in the high-level multi-scale feature map group; The Gaussian pyramid and Laplacian pyramid are fused with the corresponding low-level multi-scale feature maps in the feature channel dimension.

3. The target detection method based on thermal imager and deep learning according to claim 2, characterized in that: Between obtaining the position of the target detection bounding box in the image and identifying the target detection result in step S20, a step of fine-tuning the bounding box is also provided: B10: Combined with the gradient function, the position loss of the bounding box is calculated with the goal of minimizing the deviation between the predicted bounding box output by the deep convolutional neural network and the corresponding true bounding box; B20: Calculate the position adjustment amount of the bounding box based on the calculated position loss of the bounding box, and adjust the position of the bounding box.

4. The target detection method based on thermal imager and deep learning according to claim 3, characterized in that: The position loss of the bounding box is calculated as: in, is the position loss of the bounding box, is the predicted bounding box output by the deep convolutional neural network, For the bounding box The area, is the corresponding true bounding box, For the bounding box The area, Represents the intersection, represents the union of sets, Represents a bounding box With bounding box The Euclidean distance of the center point, For the bounding box The diagonal distance, is the penalty factor, for The expression of the function.

5. The target detection method based on thermal imager and deep learning according to claim 4, characterized in that: Penalty Factor Based on the measured bounding box and bounding box The parameter formula for obtaining the aspect ratio is: in, Is the measurement bounding box and bounding box The aspect ratio parameter, and Corresponding to the real bounding box The width and height, and Then they correspond to the predicted bounding boxes width and height.

6. The target detection method based on thermal imager and deep learning according to claim 1, characterized in that: The method for constructing the multi-scale pyramid pooling residual network in step S40 is: The convolutional layers of the deep convolutional neural network are replaced with dilated convolutional layers of a set scale, and the last average pooling layer is replaced with a multi-scale pyramid pooling layer. A multi-layer residual block is added between the set layers of the network after updating the convolutional layer and the pooling layer to construct a multi-scale pyramid pooling residual network.

7. The target detection method based on thermal imager and deep learning according to claim 6, characterized in that: The multi-layer residual block is a two-layer residual structure or a three-layer residual structure; The two-layer residual structure consists of a first convolutional layer with a kernel of 3×3 and a second convolutional layer with a kernel of 3×3, connected in sequence. The input of the first convolutional layer and the output of the second convolutional layer are spliced ​​together. The three-layer residual structure consists of a third convolutional layer with a kernel of 1×1, a fourth convolutional layer with a kernel of 3×3, and a fifth convolutional layer with a kernel of 1×1, which are connected sequentially. The input of the third convolutional layer and the output of the fifth convolutional layer are spliced ​​together.

8. A target detection system based on a thermal imager and deep learning, used to implement the target detection method based on a thermal imager and deep learning according to any one of claims 1 to 7, characterized in that: It includes a data acquisition module configured to acquire video images of mine working conditions under different working conditions through a visible light imager, or to acquire video images of mine working conditions simultaneously through a thermal imager and a visible light imager; The dust concentration detection and jump module collects the dust concentration of the mine working condition through the dust monitor, judges and executes: if the dust concentration of the mine working condition is lower than the set threshold, the data acquisition module collects the video image of the mine working condition through the visible light imager and jumps to the first target detection module; otherwise, the data acquisition module collects the video image of the mine working condition through the thermal imager and the visible light imager synchronously, and jumps to the second target detection module; The first target detection module is configured to use a deep convolutional neural network to extract a high-level multi-scale feature map group and a low-level multi-scale feature map group from an image captured by a visible light imager, perform iterative feedback fusion of the feature map groups, and combine the complementary relationship between the features in the fused feature map groups to obtain the position of the target detection bounding box in the image and identify the target detection result; The second target detection module extracts the thermal imaging contour of the image captured by the thermal imager through an adaptive edge detection method and calculates the position of the thermal imaging contour in the image; The similarities between the extracted contours and the target images pre-stored in the infrared thermal imaging library are calculated respectively, and the target image categories and corresponding thermal imaging contour positions with similarities greater than the set value are used as the candidate target detection results. A multi-scale pyramid pooling residual network is constructed to extract the features of the images collected by the visible light imager corresponding to the images collected by the thermal imager. The candidate target detection results are screened through a fully connected network to obtain the final target detection result.

9. A storage device storing a plurality of programs, characterized in that: The above program is suitable for being loaded and executed by a processor to implement the target detection method based on thermal imager and deep learning as described in any one of claims 1 to 7.

10. A processing device comprising a processor adapted to execute various programs; and a storage device adapted to store a plurality of programs; characterized in that: The above program is suitable for being loaded and executed by a processor to implement: the target detection method based on thermal imager and deep learning as described in any one of claims 1-7.

Images