Target object detection method, device and system

Abstract

The invention relates to the technical field of image detection, in particular to a target object detection method, device and system and a computer readable storage medium. According to the method, the deep learning model and the image segmentation algorithm are combined, the detection results obtained through detection in two modes are preliminarily screened, the detection results with high accuracy are subjected to cross-linking processing, the first detection result and the second detection result are fused according to the cross-linking comparison, and the detection result for the target object is generated. Through the mode, the characteristic of high accuracy of the recognition result of the deep learning model and the characteristic of high recall rate of the image segmentation algorithm are combined, targets with similar shapes can be accurately recognized through the deep learning model, and the defect of low precision of the image segmentation algorithm is overcome; the majority of targets in the image can be detected through the image segmentation algorithm, the problem that a deep learning model is inaccurate in small target recognition is solved, and the accuracy of target object detection can be greatly improved through combination of the two.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image detection technology, and specifically to a target object detection method, device, system and computer-readable storage medium. Background Art

[0002] With the development of modern technology, various automated intelligent devices are being used more and more widely, such as robots, drones, etc. They play an important role in all aspects of people's production and life. People can use automated intelligent devices to achieve map surveying, material transportation, aerial photography and other work and activities. However, with the widespread application of intelligent devices, how to effectively monitor and detect intelligent devices has become increasingly important.

[0003] The inventors of the present application discovered during their research that when monitoring smart devices, the identification of smart devices is the key point. For example, when detecting the position of a drone, in order to identify and detect the drone even in weather with low visibility or at night, thermal imaging technology is usually used. For example, drone detection is performed using infrared light. However, existing infrared target detection technologies are often greatly affected by external factors. Due to the influence of heat conduction effects and air scattering, the edges of images detected by infrared light are blurred and noise is easily generated, resulting in relatively low detection accuracy of current infrared target detection technologies. Summary of the invention

[0004] In view of the above problems, the embodiments of the present application provide a target detection method, device, system and computer-readable storage medium to solve the above technical problems existing in the prior art.

[0005] In one aspect of an embodiment of the present application, a method for detecting a target object is provided, the method comprising:

[0006] Acquire an infrared image of the target detection area;

[0007] Detecting the target in the infrared image by using a pre-trained deep learning model to generate a first detection result, where the first detection result includes a first area corresponding to the target and a first confidence level corresponding to the first area;

[0008] The infrared image is processed by a preset image segmentation algorithm to generate an adaptive threshold;

[0009] Binarizing the infrared image according to the adaptive threshold to generate a second detection result, wherein the second detection result includes a second area corresponding to the target object and a second confidence level corresponding to the second area;

[0010] Screening the first region according to the first confidence level using a preset target object recognition model to generate a first candidate region;

[0011] Screening the second detection result according to the second confidence level using a preset target object recognition model to generate a second candidate area;

[0012] Performing cross-linking processing on the first candidate region and the second candidate region to determine an intersection-over-union ratio;

[0013] The first candidate region and the second candidate region are fused according to the intersection-and-union comparison to generate a target region corresponding to the target object.

[0014] In some embodiments, preferably, the deep learning model includes a backbone network, a neck network and a head network, and the neck network includes a small target detection layer;

[0015] The detecting the target object in the infrared image by using a pre-trained deep learning model to generate a first detection result includes:

[0016] The backbone network includes a plurality of convolutional layers, and after the backbone network processes the infrared image through a second convolutional layer, it outputs a first feature map to the small target detection layer;

[0017] The neck network generates a second feature map after three downsamplings, and the first feature map and the second feature map have the same resolution;

[0018] The small target detection layer performs fusion processing on the first feature map and the second feature map, and outputs the fusion process to the head network;

[0019] The head network outputs a first detection result.

[0020] In some embodiments, preferably, the infrared image is processed by a preset image segmentation algorithm to generate an adaptive threshold value, which includes:

[0021] Processing the infrared image to generate a grayscale image corresponding to the infrared image;

[0022] Generate a mean value, a variance value and a brightness range corresponding to each pixel according to the grayscale image;

[0023] An adaptive threshold is determined according to the mean value, the variance value, the brightness range and the preset weight information.

[0024] In some embodiments, preferably, the step of processing the infrared image to generate a grayscale image corresponding to the infrared image includes:

[0025] The infrared image is subjected to image channel separation, and a corresponding grayscale image is generated from the separated single-channel image.

[0026] In some embodiments, preferably, generating an adaptive threshold according to the mean value, variance value, brightness range and preset weight information includes:

[0027] Obtain a first difference value of the mean value corresponding to each pixel of the grayscale image, and assign first preset weight information to the mean value according to the first difference value;

[0028] Obtain a second difference value of the variance corresponding to each pixel of the grayscale image, and assign second preset weight information to the variance according to the second difference value;

[0029] Acquire a maximum value and a minimum value within the brightness range, determine an extreme value difference of brightness according to the maximum value and the minimum value, and assign third preset weight information to the brightness range according to the extreme value difference;

[0030] An adaptive threshold is generated according to the mean, the first preset weight information, the variance, the second preset weight information, the extreme value difference and the third preset weight information.

[0031] In some embodiments, preferably, the step of screening the first region according to the first confidence level using a preset target object recognition model to generate a first candidate region includes:

[0032] Identify the first area using a preset object recognition model to generate category information corresponding to the first area;

[0033] If the category information corresponding to the first region is the same as the category of the target object, and the first confidence level is greater than or equal to a preset first confidence level threshold, taking the first region as a first candidate region;

[0034] The step of screening the second detection result according to the second confidence level using a preset target object recognition model to generate a second candidate area includes:

[0035] Identify the second area using a preset object recognition model to generate category information corresponding to the second area;

[0036] If the category information corresponding to the second area is the same as the category of the target object, and the second confidence level is greater than or equal to a preset second confidence level threshold, the second area is used as a second candidate area.

[0037] In some embodiments, preferably, the step of fusing the first candidate region and the second candidate region according to the intersection-and-union ratio to generate the target region corresponding to the target object includes:

[0038] If the intersection-over-union ratio is greater than or equal to a preset overlap threshold, the first candidate region and the second candidate region are fused to generate a target region corresponding to the target object.

[0039] In some embodiments, preferably, the method further comprises:

[0040] If the intersection-over-union ratio is less than a preset overlap threshold, N first candidate regions and N second candidate regions are obtained in descending order according to the first confidence and the second confidence to perform fusion processing to generate a target region corresponding to the target object, wherein N is a preset target number threshold and N is a positive integer.

[0041] In some embodiments, preferably, the target region includes the area of ​​the target region;

[0042] The method further comprises:

[0043] Acquire distance information of a target object corresponding to the target area;

[0044] Determine an area threshold corresponding to the target object according to the distance information;

[0045] The target area is filtered according to the area of ​​the target area and the area threshold.

[0046] In another aspect of the embodiment of the present application, a target object detection device is further provided, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus;

[0047] The memory is used to store at least a program, and the program enables the processor to perform the operations of the target object detection method proposed in any of the above embodiments.

[0048] In a third aspect of the embodiments of the present application, a target object detection system is further proposed, the system comprising a radar, a camera and the target object detection device proposed in the above embodiments;

[0049] The radar is used to obtain azimuth information of the target object;

[0050] The camera is used to photograph the target object according to the azimuth information to obtain an infrared image;

[0051] The target object detection device is used to detect the target object according to the infrared image.

[0052] In a fourth aspect of an embodiment of the present application, a computer-readable storage medium is further proposed, wherein at least one program is stored in the storage medium. When the program is run on an object detection device, the object detection device enables the object detection device to perform the operations of the object detection method provided in the above embodiment.

[0053] In summary, the target object detection method, system and device proposed in the embodiments of the present application combine the deep learning model and the image segmentation algorithm, and after preliminary screening of the detection results obtained by the two detection methods, the detection results with higher accuracy are cross-linked, and the first detection result and the second detection result are fused according to the cross-linking ratio to generate the detection result for the target object. In this way, the high accuracy of the recognition results of the deep learning model and the high recall rate of the image segmentation algorithm are combined. The deep learning model can accurately identify targets of similar shapes, making up for the lack of accuracy of the image segmentation algorithm; the image segmentation algorithm can detect most targets in the image, making up for the problem of inaccurate recognition of small targets by the deep learning model. The combination of the two can greatly improve the accuracy of target detection.

[0054] The above description is only an overview of the technical solution of the embodiment of the present application. In order to more clearly understand the technical means of the embodiment of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiment of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The accompanying drawings are only used to illustrate the embodiments and are not to be considered as limiting the present application. In addition, the same reference symbols are used to represent the same components throughout the accompanying drawings. In the accompanying drawings:

[0056] Figure 1 A schematic diagram of the structure of a target detection system proposed in an embodiment of the present application;

[0057] Figure 2 A schematic diagram of the structure of a target object detection device proposed in an embodiment of the present application;

[0058] Figure 3 A flow chart of a target object detection method proposed in an embodiment of the present application;

[0059] Figure 4 A schematic diagram of a deep learning model architecture proposed in an embodiment of the present application;

[0060] Figure 5 A schematic diagram of infrared image recognition using a deep learning model proposed in an embodiment of the present application;

[0061] Figure 6A schematic diagram of the image segmentation algorithm flow proposed in the embodiment of the present application;

[0062] Figure 7 A schematic diagram of detecting a target object by using an image segmentation algorithm proposed in an embodiment of the present application;

[0063] Figure 8 This is a schematic diagram of the target object detection results after fusion proposed in the embodiment of the present application. DETAILED DESCRIPTION

[0064] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein.

[0065] In order to effectively monitor and manage smart devices, how to effectively detect the target object is the first problem that needs to be solved. For example, for the management of drones, the range of drone activities is very large. If you want to effectively manage drone equipment, you must first be able to detect the existence of drones.

[0066] In order to effectively detect smart devices, radar and cameras are usually used to detect smart devices. Radar detection technology can accurately and quickly detect the azimuth information of smart devices. Cameras can obtain images of the monitored area by taking pictures under visible conditions, and then analyze the images to detect smart devices. However, in weather with low visibility or at night, it is impossible to capture smart devices through visible light images.

[0067] In order to solve the problem of identifying smart devices under low visibility conditions, smart devices can be detected through thermal imaging technology, such as infrared target detection technology, using an infrared camera to perform thermal imaging of the target area, and then analyzing the infrared image to achieve detection of smart devices. However, the inventors of this application found in their research that compared with visible light images, the detection of smart devices by infrared thermal imaging has the characteristics of poor resolution, low contrast, low signal-to-noise ratio, blurred visual effects, and small amount of information. It is necessary to pre-process the image through image enhancement technology to obtain a comfortable visual imaging effect. At the same time, infrared images will be affected by the outside world and generate noise in each link of imaging. For example, due to the heat conduction effect and air scattering, the image edge is blurred, the contrast is low, and there will be speckle noise caused by uneven temperature distribution. In addition, due to the gain of the infrared camera detector and the influence of the internal electronic circuit, salt and pepper noise and stripe noise will also appear, which seriously affects the recognition effect.

[0068] In order to solve the above-mentioned problems in detecting targets by infrared means, the applicant of the present application has proposed a target detection method, device, system and computer-readable storage medium, which can realize accurate detection of smart devices by infrared thermal imaging technology under conditions of low visibility. In the present application, it is proposed to combine a deep learning model with an image segmentation algorithm, use a deep learning model to detect infrared images to generate a first detection result, process the infrared image by an image segmentation algorithm, generate an adaptive threshold, segment the infrared image based on the adaptive threshold, segment the relatively abrupt targets in the image as much as possible, and generate a second detection result. After obtaining the first detection result and the second detection result, the detection results obtained by the two detection methods are preliminarily screened, and then the screened detection results with higher accuracy are cross-linked, and the first detection result and the second detection result are fused according to the cross-linking ratio to generate a detection result for the target. In this way, the high accuracy of the recognition results of the deep learning model and the high recall rate of the image segmentation algorithm are combined. The deep learning model can accurately identify targets of similar shapes, making up for the problem of insufficient accuracy of the image segmentation algorithm; the image segmentation algorithm can detect most targets in the image, making up for the problem of insufficient recognition of small targets by the deep learning model. The combination of the two can greatly improve the accuracy of target detection. At the same time, when performing image segmentation based on adaptive thresholds, the appropriate segmentation threshold can be automatically selected according to the local or global characteristics of the image to achieve image binarization, which can better cope with the image segmentation needs under different scenes and lighting conditions.

[0069] The target object detection method, device and system proposed in the embodiments of the present application can detect a variety of intelligent devices, including but not limited to drones, robots, etc. In the embodiments of the present application, only drones are used as an example for illustration.

[0070] Figure 1 The structure diagram of an object detection system proposed in the embodiment of the present application is shown, and the object detection system is also an application scenario of the object detection method proposed in the embodiment of the present application. Figure 1 In the embodiment, the target object detection system includes a target object detection device 10, a radar 20 and a camera 30.

[0071] exist Figure 1In the embodiment, the target detection system is used to detect the drone 41 in the target detection area, and the target detection device 10 is connected to the radar 20 and the camera 30 respectively, wherein the radar 20 is used to wirelessly scan the target detection area to obtain the azimuth of the target, and the camera 30 is used to shoot the target detection area to obtain an infrared image. In practical applications, the target detection area can be first scanned by the radar 20, and when the azimuth of the target is obtained, the azimuth is sent to the camera 30 and the target detection device 10. The camera 30 can rotate the shooting angle of the lens according to the azimuth information to ensure that the image of the target can be included in the captured infrared image. The radar 20 can also obtain the distance information of the target, that is, determine the direct connection distance of the target from the target detection system. After receiving the infrared image taken by the camera, the target detection device 10 analyzes the infrared image by executing the target detection method proposed in the embodiment of the present application to obtain the position information of the target. Among them, in the target detection system, the radar 20, camera 30 and target detection device 10 can be integrated into one, or can be separately set into different entities, and each entity is connected by a line, which is not limited in this application.

[0072] Figure 2 FIG. 1 shows a structural diagram of a target object detection device proposed in an embodiment of the present application. Figure 2 As shown, the target detection device 10 may include: a processor 102 , a memory 106 , a communication interface 104 and a communication bus 108 .

[0073] The processor 102, the memory 106 and the communication interface 104 communicate with each other via a communication bus 108. The memory 106 is used to store at least one program 110, and the program 110 enables the processor 102 to execute the target object detection method proposed in the embodiment of the present application.

[0074] The processor 102 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the drone direction determination device may be processors of the same type, such as one or more CPUs; or may be processors of different types, such as one or more CPUs and one or more ASICs.

[0075] The memory 106 is used to store the program 110. The memory 106 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0076] The program 110 may be specifically called by the processor 102 to execute the target object detection method proposed in the embodiment of the present application.

[0077] It should be noted that the target detection device 10 can be a terminal device or a remote server. When the target detection method is run through a remote server, the remote server can be connected to the above-mentioned radar 20 and camera 30 through a terminal device to transmit the data obtained by the radar 20 and the infrared image obtained by the camera 30 to the target detection device 10. At the same time, the target detection device 10 can also display the target detection result to the user through the terminal device. The product form of the target detection device proposed in the embodiment of the present application is not limited in this application.

[0078] When the target object detection system proposed in the embodiment of the present application detects the target object in the target detection area, the target object detection device runs the target object detection method proposed in the embodiment of the present application. Specifically, the flow chart of the target object detection method proposed in the embodiment of the present application is as follows: Figure 3 As shown, the method comprises the following steps:

[0079] Step S110: Acquire an infrared image of the target detection area;

[0080] In the embodiment of the present application, the camera is responsible for photographing the target detection area and sending the photographed infrared image to the target object detection device.

[0081] Step S120: Detecting the target object in the infrared image by using a pre-trained deep learning model to generate a first detection result;

[0082] When the target detection device receives the infrared image acquired by the camera, it first analyzes the infrared image through a preset deep learning model, wherein the deep learning model is pre-trained. Before the target detection device executes the target detection method, the deep learning model needs to be trained first. The training data can be selected from drone devices with different scenes, different image quality, and different shapes for training.

[0083] The deep learning model can select a suitable deep learning model as needed, for example: in the embodiment of the present application, when selecting the deep learning model, the characteristics of the target object are taken into consideration, such as drones. Drones are often relatively small in size, and a deep learning model is required to be able to detect small targets. Therefore, in the embodiment of the present application, preferably, the YOLOv5 model is used as the deep learning model.

[0084] like Figure 4As shown, the structure diagram of the YOLOv5 model is shown. The YOLOv5 model is usually used for target detection tasks. The model is divided into three main parts: Backbone, Neck and Head.

[0085] The Backbone network includes an input layer (input), a SPPF (Spatial Pyramid Pooling-Fast) layer, a convolutional layer (Conv) and a C3 module.

[0086] The input layer is used to receive raw data input, which can be images, text, audio or other types of data. In the embodiment of the present application, the input layer is used to receive infrared images acquired by the camera. At the same time, the input layer also defines the dimensions and format of the data received by the model. The input layer specifies the height, width and number of channels of the input image, such as Figure 4 As shown, the input layer outputs 3 channels, with a height and width of 640, where the 3 channels are 3 channels of RGB images. The SPPF (Spatial Pyramid Pooling-Fast) layer is an improved pooling layer used to extract multi-scale features. The convolution layer (Conv) is used for feature extraction. Figure 4 Multiple convolutional layers are shown in FIG, and each convolutional layer is followed by the size of a feature map. The C3 module is a custom module for further extracting features. Multiple C3 modules are shown in the figure, and each module is followed by the size of a feature map.

[0087] The Neck network includes an upsample layer, a concatenation layer, and a C3 module, wherein the upsample layer is used to upsample the image to increase the resolution of the feature map. The concatenation layer is used to concatenate the feature map to fuse features at different levels. The C3 module is a custom module for further processing features.

[0088] The Head network includes a detection layer (Detect) and a convolution layer (Conv), wherein the detection layer (Detect) is used to output the final detection result, such as a bounding box and a category, etc. The convolution layer (Conv) is used to generate the final detection feature.

[0089] In the embodiment of the present application, in order to improve the recognition of small targets, a small target detection layer is added in the Neck (neck network), such as Figure 4As shown in , the detection capability of small targets is improved by adding a small target detection layer. In the YOLOv5 model, the detection of small targets can be achieved by adding a detection head at a deeper level of the backbone network. Figure 4 As shown in the figure, the P2 layer feature extraction detection head of the backbone network has a resolution of 160x160 pixels, which is equivalent to only performing two downsampling operations in the backbone network, and contains richer underlying feature information of the target. In this way, small targets can be detected quickly and effectively.

[0090] In the YOLOv5 model, after the small target detection layer is added, the backbone network processes the infrared image through the second convolution layer, and outputs the first feature map to the small target detection layer; the neck network generates a second feature map after three downsamplings, and the first feature map and the second feature map have the same resolution; the small target detection layer fuses the first feature map and the second feature map, and outputs them to the head network; the head network outputs a first detection result, and the first detection result includes a first area corresponding to the target object and a first confidence level corresponding to the first area, wherein the first confidence level is used to indicate the accuracy of the prediction result, and a higher confidence level indicates a more accurate prediction result.

[0091] After adding the small target detection layer, the connection layer (Concat) no longer outputs a 256*80*80 feature map to C3. The Neck part in the original network structure does not have a 160x 160 feature map. Through the small target detection layer, the 80x 80 feature map can be upsampled again to obtain a 160x 160 feature map, which can be concat-fused with the 160x160 feature map of the second layer in Backbone to obtain a 160x 160 feature map. The feature map is then passed to the detection layer (Detect) of the Head (head network) through C3.

[0092] Furthermore, in order to improve the recognition effect of the YOLOv5 model on small targets, YOLOv5 also introduces the attention mechanism to further improve the performance of the model. The attention mechanism can help the model pay more attention to the important areas in the image, thereby improving the accuracy of detection. For example, the CBAM (Convolutional Block Attention Module) attention module can be introduced. This is a lightweight attention module that can be trained end-to-end. CBAM will derive the attention map along the two independent dimensions of channel and space in turn, and then multiply the attention map with the output feature map to enhance important features and suppress unimportant features. The SE (Squeeze-and-Excitation) module can also be introduced. The SE module improves the representation ability of the model by adaptively recalibrating channel features. The main idea is to enhance useful features and suppress irrelevant features by learning the relationship between channels. The ECA (Efficient Channel Attention) module can also be introduced. ECA is an improved version of the SE module. It uses 1D convolution to more efficiently interact between channels, effectively reducing the number of parameters and calculations.

[0093] In addition, YOLOv5 can be improved by adding custom attention modules. For example, you can add a SplitAttention module, which generates attention weights through an MLP (multi-layer perceptron) layer, then calculates the attention distribution through a spatial displacement operation and a Softmax layer, and finally applies the attention weights and sums them to generate the output. This approach can enhance the model's ability to capture key features.

[0094] The above attention mechanism can be integrated into specific locations in the YOLOv5 network, such as after certain convolutional layers in the feature extraction stage, or inserted into the Neck (neck network). Usually inserted into the CSP block or part of the PAN (Path Aggregation Network), it can significantly improve the model performance.

[0095] The embodiment of the present application effectively improves the detection performance of small targets by adding a small target detection layer and introducing an attention mechanism in YOLOv5, and enhances the model's attention to important features, so as to better identify targets in infrared images.

[0096] Figure 5 The first detection result generated after analyzing and predicting the infrared image through the deep learning model is shown. Figure 5 It can be seen that the deep learning model can more accurately identify the shape contour of the target object, wherein the contour area corresponds to the first area corresponding to the identified target object.

[0097] Step S130: Processing the infrared image by a preset image segmentation algorithm to generate an adaptive threshold;

[0098] In order to improve the accuracy of detecting the target by infrared detection technology, the embodiment of the present application further processes the infrared image by an image segmentation algorithm. Image segmentation algorithms, based on threshold segmentation, edge detection, region growing, etc., are usually used to extract regions or objects of interest from images. The advantage of image segmentation algorithms is that they can provide a higher recall rate, which refers to the ability of the algorithm to identify all or most of the real targets in the image, and is an important performance indicator for evaluating image segmentation algorithms. That is, the ability to detect most targets in the image through the image segmentation algorithm can complement the above-mentioned target detection through the deep learning model.

[0099] At the same time, in actual applications, when detecting targets through infrared detection technology, especially when detecting through infrared technology in outdoor environments, the background of the infrared image is relatively complex and the lighting conditions are in an uneven state. In order to more accurately segment the infrared image, an image segmentation algorithm based on an adaptive threshold is proposed in an embodiment of the present application to segment the infrared image. The image segmentation algorithm based on an adaptive threshold can adjust the segmentation threshold according to the local characteristics of the image, and is suitable for infrared images under different lighting and background conditions.

[0100] Specifically, the steps include:

[0101] Step S131: Processing the infrared image to generate a grayscale image corresponding to the infrared image;

[0102] When processing infrared images, it is first necessary to separate the image channels of the infrared images, and generate the corresponding grayscale images from the separated single-channel images. By converting the input color image into a grayscale image, the subsequent calculations can be simplified because the grayscale image has only one channel. The channel separation can separate multiple color channels of the image, such as the red, green, and blue channels in the RGB color space.

[0103] Step S132: generating a mean value, a variance value and a brightness range corresponding to each pixel according to the grayscale image;

[0104] After the grayscale image corresponding to the infrared image is generated, the corresponding statistical features are calculated, including the mean, variance and brightness range, wherein the brightness range includes the maximum brightness and the minimum brightness.

[0105] The mean value is mainly used to reflect the overall brightness level of the infrared image, and can be obtained by calculating the global or local mean value of the image.

[0106] The variance is used to reflect the distribution of pixel values, that is, the contrast of the image, and can be obtained by calculating the global or local variance of the image.

[0107] The brightness range is used to reflect the brightness characteristics of the image, which is very important for segmentation in the highlight area, and includes the maximum brightness value and the minimum brightness value.

[0108] Step S133: determining an adaptive threshold according to the mean value, variance value, brightness range and preset weight information.

[0109] In order to flexibly adapt to different characteristics and conditions of infrared images, balance the influence of different features, adapt to different image characteristics, improve the accuracy and robustness of image segmentation, and optimize the segmentation effect, different weight information is set for different statistical features in an embodiment of the present application, wherein the weight information is allocated with reference to the difference between different statistical features, including: after calculating the first difference value of the mean corresponding to each pixel of the grayscale image information, first preset weight information is allocated to the mean according to the first difference value; after calculating the second difference value of the variance corresponding to each pixel of the grayscale image information, second preset weight information is allocated to the variance according to the second difference value; after obtaining the maximum and minimum values ​​within the brightness range, the extreme value difference of brightness is determined according to the maximum and minimum values, and third preset weight information is allocated to the brightness range according to the extreme value difference.

[0110] After obtaining the above information, the adaptive threshold is calculated using the following weighted formula:

[0111] Adaptive threshold=first preset weight information*mean+second preset weight information*variance+third preset weight information*(maximum value-minimum value) / 2.

[0112] Step S140: binarizing the infrared image according to the adaptive threshold to generate a second detection result;

[0113] According to the calculated adaptive threshold, the image is converted into a binary image, wherein pixels above the adaptive threshold are set to one color, such as white, and pixels below the adaptive threshold are set to another color, such as black.

[0114] The second detection result includes a second area corresponding to the target object and a second confidence level corresponding to the second area.

[0115] After completing the binarization process, the binarized image can also be post-processed, such as denoising, filling small holes, etc., to improve the quality of image segmentation.

[0116] In the embodiment of the present application, the infrared image is segmented by an image segmentation algorithm based on an adaptive threshold, which improves the recall rate of target recognition and complements the target recognition method based on a deep learning model.

[0117] Step S150: Screening the first region according to the first confidence level using a preset target object recognition model to generate a first candidate region;

[0118] After completing the detection of the infrared image through the above-mentioned deep learning model and image segmentation algorithm respectively, the detection result is sent to the preset target object recognition model for filtering, wherein the target object recognition model can be a classifier, such as: a drone classifier, which is mainly used to classify and filter the detection results. The target object recognition model identifies the first area and generates category information corresponding to the first area. For example: if the first area is identified as a drone, it is determined whether the actual category of the target object is consistent with the category identified by the first area. If they are consistent, it is further determined whether the corresponding confidence level is greater than the preset first confidence level threshold. If it is greater than or equal to, it means that the accuracy of the first area is high, and it is screened out as candidate area information. Figure 7 As shown in the figure, the effect diagram after the infrared image is processed by the image segmentation algorithm based on adaptive threshold, the figure shows multiple highlighted small targets that can be identified, avoiding the omission of target objects when identifying them.

[0119] Step S160: Screening the second detection result according to the second confidence level using a preset target object recognition model to generate a second candidate area;

[0120] Similarly, the detection result of the image segmentation algorithm is sent to the target object recognition model for filtering. If the category of the target object is the same as the category information corresponding to the second region, and the second confidence is greater than or equal to the preset second confidence threshold, the second region is selected as the second candidate region.

[0121] Step S170: performing cross-linking processing on the first candidate region and the second candidate region to determine an intersection-over-union ratio;

[0122] After the detection results of the deep learning model and the image segmentation algorithm are screened by the target recognition model, the regions with higher confidence are screened out as candidate regions. In order to further improve the accuracy of target recognition, the embodiment of the present application further cross-links the candidate regions determined by the above two methods to merge the two. For each candidate region, their intersection over union (IoU) is calculated, where IoU is an indicator used to evaluate the degree of overlap of different candidate regions, and the calculation formula is:

[0123] IoU = overlapping area / total area;

[0124] The overlapping area is the intersection area of ​​the bounding boxes of the two candidate regions, and the total area is the union area of ​​the two bounding boxes of the two candidate regions.

[0125] Step S180: performing a fusion process on the first candidate region and the second candidate region according to the intersection-and-union comparison to generate a target region corresponding to the target object.

[0126] After determining the IoU between different candidate regions, if the IoU between different candidate regions is greater than or equal to the preset overlap threshold, it means that the two candidate regions correspond to the same target object, and the first candidate region and the second candidate region are fused to generate the target region corresponding to the target object, which is used as the region where the final target object is located. The preset overlap threshold is used to determine whether the two candidate regions belong to the same target, for example: 0.5, if the IoU is greater than 0.5, it means that the two candidate regions correspond to the same target object, and if it is less than 0.5, it means that the two candidate regions are different targets.

[0127] For candidate regions whose IoU is greater than a preset overlap threshold, their bounding boxes are merged. The merging method can be to calculate the minimum enclosing rectangle of the overlapping area, or to use a weighted average method to determine the final bounding box position.

[0128] like Figure 8 As shown in the figure, the recognition results of the determined target objects are shown after merging the candidate areas with higher cross-linking ratio. Compared with the deep learning model that can only recognize larger targets and the image segmentation algorithm that can recognize multiple targets, the target area after the results of the two methods are fused takes into account both the accuracy and comprehensiveness of the recognition. Figure 8 The target objects are accurately identified, and each target object has relatively accurate contour information. Therefore, the target object detection method proposed in the embodiment of the present application takes into account the respective advantages of the deep learning model and the image segmentation algorithm, greatly improving the accuracy and comprehensiveness of target object detection.

[0129] Furthermore, due to the heat conduction effect and air scattering in outdoor environments, there is often speckle noise caused by uneven temperature distribution. In order to further improve the accuracy of target detection, inaccurate detection targets are filtered out to the greatest extent. At the same time, in order to avoid incorrectly filtering out some important targets, in the embodiment of the present application, for the case where the intersection-over-union ratio is less than the preset overlap threshold, N first candidate areas and second candidate areas are obtained in descending order according to the first confidence level and the second confidence level for fusion processing to generate the target area corresponding to the target, where N is the preset target number threshold and N is a positive integer.

[0130] When the intersection over union (IoU) is less than the preset overlap threshold, it means that only one method has detected the target object. In this case, in order to avoid missing it, further judgment is needed. However, considering that in actual target detection, the number of targets in infrared images is generally not too many, for example, when detecting drones, there are not too many drones in the same picture. By using this feature, the number of targets can be set, and the identified candidate areas can be sorted according to the confidence level. According to the set number of targets, a certain number of candidate areas with higher confidence levels can be selected to perform target area fusion processing. This can filter out redundant targets while taking into account the accuracy and comprehensiveness of target detection.

[0131] At the same time, after the target area where the target object is located is determined through the above-mentioned embodiment, the target object can be further tracked. During the tracking process, due to the real-time detection characteristics of the embodiment of the present application, the recognition results of the front and back frames of the infrared image can be selected, and the image in the tracking process can be corrected in real time to improve the accuracy of target management.

[0132] In order to further improve the accuracy of target detection, information can be obtained by radar detection to further screen the detected targets. It can be seen from the above embodiments that the target detection system usually also includes a radar, through which the azimuth of the target can be obtained, and the distance information of the target can also be obtained. In practice, taking drones as an example, the types of drones are limited, and the sizes of drones are relatively limited. In the target detection system, the type and size of the drone can be set in advance to filter the detected targets. In actual applications, the shooting distance of the camera from the target will cause the imaging size and area of ​​the target in the infrared image to be different. Utilizing this feature, the embodiment of the present application pre-sets the mapping relationship between the shooting distance and the imaging area of ​​the target in the target detection device, and then judges the area of ​​the detected target through the mapping relationship, thereby filtering out the misidentified targets.

[0133] Specifically, during detection, the target detection device obtains the distance information of the target corresponding to the target area; then, based on the distance information and the mapping relationship between the target imaging area and the shooting distance pre-set in the target detection device, the area threshold corresponding to the target under the distance information is determined; then, the target area is filtered based on the area of ​​the target area and the area threshold, wherein the area of ​​the target area corresponding to the target can be obtained by the above-mentioned image segmentation algorithm. In this way, the detected targets can be further filtered, thereby improving the accuracy of detection.

[0134] In summary, it can be seen from the above embodiments that the target detection method, system and device proposed in this application combine the deep learning model and the image segmentation algorithm. After the detection results obtained by the two detection methods are preliminarily screened, the detection results with higher accuracy are cross-linked, and the first detection result and the second detection result are fused according to the cross-linking ratio to generate the detection result for the target. In this way, the high accuracy of the recognition results of the deep learning model and the high recall rate of the image segmentation algorithm are combined. The deep learning model can accurately identify targets of similar shapes, which makes up for the lack of accuracy of the image segmentation algorithm; the image segmentation algorithm can detect most targets in the image, which makes up for the problem of inaccurate recognition of small targets by the deep learning model. The combination of the two can greatly improve the accuracy of target detection. At the same time, the infrared image is processed by an image segmentation algorithm based on an adaptive threshold, and a suitable threshold is automatically selected according to the local or global characteristics of the image to realize the binarization of the image, and relatively abrupt targets in the image are segmented as much as possible, which can better cope with the image segmentation requirements under different scenes and lighting conditions, and greatly improve the accuracy of detection under different detection conditions.

[0135] An embodiment of the present application further provides a computer-readable storage medium, in which executable instructions are stored. When the executable instructions are executed on a target detection device, the target detection device executes the target detection method provided in any of the above embodiments.

[0136] An embodiment of the present application also provides a target object detection program, which is used to execute the target object detection method provided in the above embodiment.

[0137] The algorithm or display provided here are not inherently related to any specific computer, virtual system or other equipment. Various general systems can also be used together with the teaching based on this. According to the above description, it is obvious to construct the structure required for this type of system. In addition, the present application embodiment is not directed to any specific programming language yet. It should be understood that various programming languages ​​can be utilized to realize the content of the present application described here, and the above description of specific languages ​​is to disclose the best mode of implementation of the present application.

[0138] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0139] Similarly, it should be understood that in order to streamline the present application and help understand one or more of the various aspects of the invention, in the above description of exemplary embodiments of the present application, the various features of the embodiments of the present application are sometimes grouped together into a single embodiment, figure, or description thereof.

[0140] Those skilled in the art will appreciate that the modules in the devices in the embodiments can be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments can be combined into one module or unit or component, and they can be divided into multiple submodules or subunits or subassemblies. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying abstract and drawings) and all processes or units of any method or device disclosed in this manner can be combined in any combination. Unless otherwise explicitly stated, each feature disclosed in this specification (including the accompanying abstract and drawings) can be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0141] It should be noted that the above embodiments illustrate the present application rather than limit the present application, and those skilled in the art may design alternative embodiments without departing from the scope. The steps in the above embodiments, unless otherwise specified, should not be understood as limiting the execution order.

Claims

1. A method for detecting a target object, characterized in that: include: Acquire an infrared image of the target detection area; Detecting the target in the infrared image by using a pre-trained deep learning model to generate a first detection result, where the first detection result includes a first area corresponding to the target and a first confidence level corresponding to the first area; The infrared image is processed by a preset image segmentation algorithm to generate an adaptive threshold; Binarizing the infrared image according to the adaptive threshold to generate a second detection result, wherein the second detection result includes a second area corresponding to the target object and a second confidence level corresponding to the second area; Screening the first region according to the first confidence level using a preset target object recognition model to generate a first candidate region; Screening the second detection result according to the second confidence level using a preset target object recognition model to generate a second candidate area; Performing cross-linking processing on the first candidate region and the second candidate region to determine an intersection-over-union ratio; The first candidate region and the second candidate region are fused according to the intersection-and-union comparison to generate a target region corresponding to the target object.

2. The target object detection method according to claim 1, characterized in that: The deep learning model includes a backbone network, a neck network and a head network, and the neck network includes a small target detection layer; The detecting the target object in the infrared image by using a pre-trained deep learning model to generate a first detection result includes: The backbone network includes a plurality of convolutional layers, and after the backbone network processes the infrared image through a second convolutional layer, it outputs a first feature map to the small target detection layer; The neck network generates a second feature map after three downsamplings, and the first feature map and the second feature map have the same resolution; The small target detection layer performs fusion processing on the first feature map and the second feature map, and outputs the fusion process to the head network; The head network outputs a first detection result.

3. The target object detection method according to claim 1, characterized in that: The step of processing the infrared image by a preset image segmentation algorithm to generate an adaptive threshold value includes: Processing the infrared image to generate a grayscale image corresponding to the infrared image; Generate a mean value, a variance value and a brightness range corresponding to each pixel according to the grayscale image; An adaptive threshold is determined according to the mean value, the variance value, the brightness range and the preset weight information.

4. The target object detection method according to claim 3, characterized in that: The step of processing the infrared image to generate a grayscale image corresponding to the infrared image includes: The infrared image is subjected to image channel separation, and a corresponding grayscale image is generated from the separated single-channel image.

5. The target object detection method according to claim 3 or 4, characterized in that: Generating an adaptive threshold according to the mean value, the variance value, the brightness range and the preset weight information includes: Obtain a first difference value of the mean value corresponding to each pixel of the grayscale image, and assign first preset weight information to the mean value according to the first difference value; Obtain a second difference value of the variance corresponding to each pixel of the grayscale image, and assign second preset weight information to the variance according to the second difference value; Acquire a maximum value and a minimum value within the brightness range, determine an extreme value difference of brightness according to the maximum value and the minimum value, and assign third preset weight information to the brightness range according to the extreme value difference; An adaptive threshold is generated according to the mean, the first preset weight information, the variance, the second preset weight information, the extreme value difference and the third preset weight information.

6. The target object detection method according to claim 1, characterized in that: The step of screening the first region according to the first confidence level using a preset target object recognition model to generate a first candidate region includes: Identify the first area using a preset object recognition model to generate category information corresponding to the first area; If the category information corresponding to the first region is the same as the category of the target object, and the first confidence level is greater than or equal to a preset first confidence level threshold, taking the first region as a first candidate region; The step of screening the second detection result according to the second confidence level using a preset target object recognition model to generate a second candidate area includes: Identify the second area using a preset object recognition model to generate category information corresponding to the second area; If the category information corresponding to the second area is the same as the category of the target object, and the second confidence level is greater than or equal to a preset second confidence level threshold, the second area is used as a second candidate area.

7. The target object detection method according to claim 6, characterized in that: The step of fusing the first candidate region and the second candidate region according to the intersection-and-union ratio to generate a target region corresponding to the target object includes: If the intersection-over-union ratio is greater than or equal to a preset overlap threshold, the first candidate region and the second candidate region are fused to generate a target region corresponding to the target object.

8. The target object detection method according to claim 7, characterized in that: The method further comprises: If the intersection-over-union ratio is less than a preset overlap threshold, N first candidate regions and N second candidate regions are obtained in descending order according to the first confidence and the second confidence to perform fusion processing to generate a target region corresponding to the target object, wherein N is a preset target number threshold and N is a positive integer.

9. The target object detection method according to claim 7 or 8, characterized in that: The target region includes the area of ​​the target region; The method further comprises: Acquire distance information of a target object corresponding to the target area; Determine an area threshold corresponding to the target object according to the distance information; The target area is filtered according to the area of ​​the target area and the area threshold.

10. A target object detection device, characterized in that: include: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least a program, and the program enables the processor to perform the operation of the target detection method as described in any one of claims 1-9.

11. A target detection system, characterized in that: comprising a radar, a camera and a target object detection device as claimed in claim 10; The radar is used to obtain azimuth information of the target object; The camera is used to photograph the target object according to the azimuth information to obtain an infrared image; The target object detection device is used to detect the target object according to the infrared image.

12. A computer-readable storage medium, characterized in that: The storage medium stores at least one program, and when the program is executed on the target object detection device, the target object detection device performs the operation of the target object detection method according to any one of claims 1 to 9.

Images