Target detection method

Abstract

The embodiment of the invention provides a target detection method, and relates to the technical field of image processing, and the target detection method comprises the steps: obtaining an infrared image and a visible light image of a first region; respectively detecting objects in the detection area in the infrared image and the detection area in the visible light image to obtain a first detection result and a second detection result; associating the same object in the first detection result and the second detection result to obtain an association result; wherein the association result at least indicates associated objects and / or unassociated objects in the first detection result and the second detection result; and based on the plurality of local features of each object in the association result, performing fusion verification on each object in the association result, and outputting a detection result according to a fusion verification result. According to the target detection method provided by the embodiment of the invention, the detection accuracy can be improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of image processing, in particular, to a target detection method. BACKGROUND

[0002] Target detection is one of the core tasks in the field of computer vision, aiming to detect target objects of interest in images or videos and determine their positions, and is widely used in the fields of security, transportation, medical treatment, etc. With the continuous development of science and technology, machine learning and deep learning technologies are widely used in the field of target detection, aiming to improve the accuracy of target detection.

[0003] Most of the current target detection methods are mainly developed based on visible light images as data basis. However, in adverse environments such as rainy days, smog, insufficient light or complex background, the quality of the obtained visible light images will be greatly reduced, resulting in large detection errors when the target detection method detects such visible light images. Therefore, it is urgent to provide a target detection method that can improve the detection accuracy. SUMMARY

[0004] Embodiments of the present application provide a target detection method that can improve the accuracy of target detection.

[0005] In a first aspect, embodiments of the present application provide a target detection method, comprising: obtaining an infrared image and a visible light image of a first region; detecting objects in a detection region in the infrared image and a detection region in the visible light image, respectively, to obtain a first detection result and a second detection result; wherein the first detection result indicates objects detected from the infrared image, and the second detection result indicates objects detected from the visible light image; associating the same objects in the first detection result and the second detection result to obtain an association result; wherein the association result indicates at least associated objects and / or unassociated objects in the first detection result and the second detection result; based on a plurality of local features of each object in the association result, performing fusion verification on each object in the association result, and outputting a detection result according to the fusion verification result.

[0006] Optionally, the association result includes at least one of a first object, a second object and a third object, the first object is an unassociated object in the first detection result, the second object is an unassociated object in the second detection result, and the third object is an associated object in the first detection result and the second detection result; The fusion verification of each object in the association result based on the plurality of local features of each object in the association result, and outputting a detection result according to the fusion verification result comprises: extracting a plurality of local features of each object in the first detection result from the infrared image; extracting a plurality of local features of each object in the second detection result from the visible light image; determining, for the first object, probabilities that the first object is an object, a background and an uncertain region respectively according to the plurality of local features of the first object based on the infrared image; determining, for the second object, probabilities that the second object is an object, a background and an uncertain region respectively according to the plurality of local features of the second object based on the visible light image; determining, for the third object, probabilities that the third object is an object, a background and an uncertain region respectively according to the plurality of local features of the third object based on the infrared image and the visible light image respectively; and outputting a detection result according to the probabilities that the first object, the second object and the third object are an object, a background and an uncertain region respectively; wherein the background is a region outside the detection region in the infrared image and the visible light image.

[0007] Optionally, the plurality of local features comprises four local features, which are local entropy, local contrast, local gradient and local texture respectively. The determining of the probabilities that the first object is an object, a background and an uncertain region respectively according to the plurality of local features of the first object based on the infrared image comprises: determining four groups of classification probabilities of the first object respectively according to the local entropy, the local contrast, the local gradient and the local texture of the first object based on the infrared image; wherein the four groups of classification probabilities of the first object correspond to the four local features one by one; each group of classification probabilities in the four groups of classification probabilities of the first object comprises probabilities that the first object is an object, a background and an uncertain region respectively under a corresponding local feature; and the determining of the probabilities that the first object is an object, a background and an uncertain region respectively according to the four groups of classification probabilities of the first object.

[0008] Optionally, the plurality of local features comprises four local features, which are local entropy, local contrast, local gradient and local texture respectively. The determining the probabilities of the second object being the object, the background and the uncertain region respectively according to the multiple local features of the second object based on the visible light image comprises: determining four groups of classification probabilities of the second object based on the local entropy, the local contrast, the local gradient and the local texture of the second object based on the visible light image respectively; wherein the four groups of classification probabilities of the second object correspond to the four local features one by one; each group of classification probabilities of the four groups of classification probabilities of the second object comprises the probabilities of the second object being the object, the background and the uncertain region respectively under the corresponding local feature; and determining the probabilities of the second object being the object, the background and the uncertain region respectively according to the four groups of classification probabilities of the second object.

[0009] Optionally, the multiple local features comprise four local features, which are the local entropy, the local contrast, the local gradient and the local texture respectively. The determining the probabilities of the third object being the object, the background and the uncertain region respectively according to the multiple local features of the third object based on the infrared image and the visible light image respectively comprises: determining four groups of classification probabilities of the third object based on the local entropy, the local contrast, the local gradient and the local texture of the third object based on the infrared image respectively; wherein the four groups of classification probabilities of the third object based on the infrared image correspond to the four local features one by one; each group of classification probabilities of the four groups of classification probabilities of the third object based on the infrared image comprises the probabilities of the third object being the object, the background and the uncertain region respectively under the corresponding local feature based on the infrared image; determining the first probabilities of the third object being the object, the background and the uncertain region respectively according to the four groups of classification probabilities of the third object based on the infrared image; determining four groups of classification probabilities of the third object based on the local entropy, the local contrast, the local gradient and the local texture of the third object based on the visible light image respectively; wherein the four groups of classification probabilities of the third object based on the visible light image correspond to the four local features one by one; each group of classification probabilities of the four groups of classification probabilities of the third object based on the visible light image comprises the probabilities of the third object being the object, the background and the uncertain region respectively under the corresponding local feature based on the visible light image; determining the second probabilities of the third object being the object, the background and the uncertain region respectively according to the four groups of classification probabilities of the third object based on the visible light image; and determining the probabilities of the third object being the object, the background and the uncertain region respectively according to the first probabilities and the second probabilities of the third object being the object, the background and the uncertain region respectively.

[0010] Optionally, the outputting the detection result according to the probabilities that the first object, the second object and the third object are the object, the background and the uncertain area respectively comprises: screening an object with a probability higher than the background and the uncertain area from the first object, the second object and the third object as a fourth object; in a case that the fourth object is the associated object, outputting the position and the category of the fourth object according to the position and the category of the fourth object in the infrared image and the visible light image; in a case that the fourth object is the unassociated object, outputting the position and the category of the fourth object according to the position and the category of the fourth object in the infrared image or the visible light image.

[0011] Optionally, the associating the same objects in the first detection result and the second detection result to obtain an association result comprises: constructing a first matrix with I rows and J columns according to the category and the position of each object in the first detection result and the category and the position of each object in the second detection result; wherein I indicates the number of objects in the first detection result, J indicates the number of objects in the second detection result, and an element at the intersection of the ith row and the jth column indicates the distance between the object corresponding to the ith row and the object corresponding to the jth column; on the basis of the first matrix, subtracting the minimum element in the corresponding row from each element in the row and then subtracting the minimum element in the corresponding column from each element in the column to obtain a second matrix; performing a covering step of covering the zero elements in the second matrix with the least number of straight lines; in a case that the number of straight lines is equal to the number of rows of the second matrix, determining the second matrix as a third matrix; in a case that the number of straight lines is less than the number of rows of the second matrix, subtracting the minimum non-zero element of the second matrix from each non-zero element in the second matrix and adding the minimum non-zero element to the zero elements in the second matrix which are covered by the straight lines to obtain a fourth matrix; taking the fourth matrix as the second matrix and starting the covering step until the number of straight lines is equal to the number of rows of the second matrix; determining the row object and the column object corresponding to the zero element in the row with only one zero element in the third matrix as the associated object and determining the objects other than the associated object in the first detection result and the second detection result as the unassociated object.

[0012] Optionally, the detecting objects in the detection region in the infrared image and the detection region in the visible light image respectively to obtain the first detection result and the second detection result comprises: identifying the detection region from the infrared image and the visible light image respectively; performing background processing on regions outside the detection region in the infrared image and the visible light image respectively; and detecting objects in the infrared image and the visible light image after the background processing to obtain the first detection result and the second detection result.

[0013] Optionally, the detecting objects in the detection region in the infrared image and the detection region in the visible light image respectively comprises: detecting objects in the detection region in the infrared image and the detection region in the visible light image respectively by a detection model.

[0014] Optionally, the detection model is trained, and the training process comprises: obtaining a plurality of visible light training samples and a plurality of infrared training samples; performing background processing on regions outside the detection region in each of the visible light training samples and the infrared training samples respectively; labeling positions and categories of objects in each of the visible light training samples and the infrared training samples after the background processing; augmenting the visible light training samples and the infrared training samples to which the categories of objects with small proportions belong; and training a neural network by the visible light training samples and the infrared training samples after the labeling and the augmenting to obtain the detection model.

[0015] In a second aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor; the memory is configured to store a computer program; and the processor is configured to implement the target detection method in any of the first aspect when executing the computer program.

[0016] The target detection method in the embodiment of the present application has the following beneficial effects: Since the infrared image itself has the penetration advantage in complex environments such as low light and smoke shielding, and the visible light image itself has the advantages in details, resolution and color information, etc., compared with the way of target detection only through the visible light image, the embodiment of the present application detects objects in the detection region in the infrared image of the first region and the detection region in the visible light image, which can improve the accuracy of target detection and reduce the influence of complex environments on the detection result.

[0017] Moreover, since the objects in the detection region are detected, the user can set the region to be detected according to the demand, and only the detection region is detected, which improves the detection efficiency and reduces the detection energy consumption.

[0018] And, by associating the same object in the first detection result and the second detection result, and based on the multiple local features of each object in the association result, each object in the association result is fused and verified, so as to output the detection result according to the fusion verification result. That is, by associating the detection results of the infrared image and the visible light image, the information complementarity of different image modalities is fully utilized, and the object can be detected more comprehensively and accurately. And, based on the fusion verification of multiple local features of each object in the association result, the false judgment caused by a single feature is avoided, the accuracy of verification is greatly improved, and the output detection result is more reliable and accurate. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 A flowchart of a target detection method provided by an embodiment of the present application is shown in the figure; Figure 2 A flowchart of detecting an object in a detection area provided by an embodiment of the present application is shown in the figure; Figure 3 A flowchart of determining an association result provided by an embodiment of the present application is shown in the figure; Figure 4 A flowchart of fusion verification provided by an embodiment of the present application is shown in the figure; Figure 5 A flowchart of determining the probabilities of the third object being the object, the background and the uncertain area provided by an embodiment of the present application is shown in the figure; Figure 6 A flowchart of training a detection model provided by an embodiment of the present application is shown in the figure; Figure 7 A structural diagram of an electronic device provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0020] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the drawings. Although some embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms, and should not be interpreted as being limited to the embodiments described herein, on the contrary, these embodiments are provided to make the present application more thorough and complete. It should be understood that the drawings and embodiments of the present application are only for illustrative purposes, and are not intended to limit the scope of protection of the present application.

[0021] It should be understood that each step described in the method embodiments of the present application can be executed in different order and / or in parallel. In addition, the method embodiments can include additional steps and / or omit the execution of the steps shown. The scope of the present application is not limited in this respect.

[0022] The term "include" and variations thereof, as used herein, means "to include, without limitation"; the term "based on" means "based at least in part on"; the term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments"; the term "optional" means "optional in at least one embodiment". Related terms such as "in one "embodiment", "in another embodiment", "in some embodiments" or "in some embodiments" will be understood to mean "in at least one embodiment" or "in at least one additional embodiment". It is noted that the use of "first", "second", or "third" to describe various embodiments is merely illustrative and is not intended to limit the scope of the application. Those skilled in the art will understand that the functions of the various devices, modules, or units described in the present application can be performed by one device, module, or unit, or by a plurality of devices, modules, or units.

[0023] It should be noted that the modification of "one" or "multiple" mentioned in the present application is illustrative and not restrictive, and those skilled in the art should understand that unless otherwise explicitly indicated in the context, it should be understood as "one or more".

[0024] The names of the messages or information exchanged between the devices in the embodiments of the present application are only for illustrative purposes, and are not intended to limit the scope of the messages or information.

[0025] Visible light images can provide rich color and texture information, consistent with the visual habits of the human eye, but in complex environments such as low light or occlusion, their performance may be greatly compromised. Therefore, the target detection method relying solely on visible light images has the problem of large detection error.

[0026] Infrared images have shown significant advantages in low light or no light environments due to their ability to capture thermal radiation, especially in night vision monitoring and target detection fields, and have irreplaceable role. Therefore, in order to improve the accuracy of target detection, the present application utilizes the information complementarity of infrared images and visible light images to perform target detection, thereby improving the accuracy and comprehensiveness of target detection.

[0027] Figure 1 The flowchart of the target detection method provided by the embodiments of the present application is shown. The target detection method can be executed by an electronic device, which includes but is not limited to a desktop computer, a notebook computer, a mobile phone, and other electronic devices with data processing capabilities.

[0028] The target detection method can be applied to the detection of intruding foreign objects in multiple time periods or all day long in railway tracks, traffic bridges, tunnels, open tracks, roads, and green belts.

[0029] For example, in the case of a railway track, the intruding foreign objects include but are not limited to stones, water bottles, animals, branches, workers, and other objects that can cause train operation safety incidents.

[0030] As shown in Figure 1 The target detection method can include the following steps: 101. obtain an infrared image and a visible light image of the first region.

[0031] Exemplarily, in a railway track scene, the first region can be a region where the railway track is located. In a road scene, the first region can be a region where the road is located. In a green belt scene, the first region can be a region where the green belt is located.

[0032] The infrared image of the first region can be obtained by an infrared camera, and the visible light image of the first region can be obtained by a visible light camera. It should be noted that the infrared image and the visible light image of the first region are collected at the same time.

[0033] 102. detect objects in the detection region in the infrared image and the detection region in the visible light image respectively to obtain a first detection result and a second detection result.

[0034] The detection region is a region of interest in the infrared image and the visible light image.

[0035] Exemplarily, in a railway track scene, the detection region can be a region defined by the railway track. In a road scene, the detection region can be a region defined by the road. In a green belt scene, the detection region can be a region defined by the green belt.

[0036] The object can refer to a foreign object that intrudes into the detection region. For example, in a railway track scene, the object is an object that intrudes into the track and affects the safe operation of the train, such as a rock, a person, an animal, etc.

[0037] The first detection result indicates the object detected from the infrared image. Exemplarily, the first detection result can indicate the position and category of the object detected from the infrared image.

[0038] The second detection result indicates the object detected from the visible light image. Exemplarily, the second detection result can indicate the position and category of the object detected from the visible light image.

[0039] The category of the object includes but is not limited to a rock, a person, an animal, etc. The position of the object can be represented by the position of a detection box that frames the object, and the detection box can be a square, a circle, etc.

[0040] The specific implementation process of 102 can be as follows: The detection model is used to detect the objects in the detection region in the infrared image and the detection region in the visible light image respectively.

[0041] The specific implementation of the detection object includes but is not limited to the following two kinds: The first kind, the infrared image is input into the detection model, the detection model can detect the object in the infrared image, and output the position and category of the object detected from the infrared image. Then, according to the position of the detection area and the position of the object detected from the infrared image, the object located in the detection area is screened out, and the position and category of the object located in the detection area are collected as the first detection result.

[0042] The principle of detecting the object in the detection area in the visible light image by the detection model is the same as above, which will not be repeated here.

[0043] The second kind, as shown in Figure 2 includes the following steps: 201. Identify the detection area from the infrared image and the visible light image, respectively.

[0044] 202. Perform background processing on the area outside the detection area in the infrared image and the visible light image, respectively.

[0045] Exemplarily, the background processing can be: setting the area outside the detection area in the infrared image and the visible light image as white, black or mosaic color, etc.

[0046] 203. Detect the object in the infrared image and the visible light image after background processing, respectively, to obtain the first detection result and the second detection result.

[0047] Exemplarily, the infrared image and the visible light image after background processing can be input into the detection model respectively, so that the detection model outputs the first detection result and the second detection result.

[0048] Obviously, there is no object outside the detection area in the infrared image and the visible light image after background processing, so here the object in the infrared image and the visible light image after background processing is directly detected, and the obtained object is the object detected from the detection area of the infrared image and the visible light image.

[0049] It should be noted that the specific structure and training process of the detection model will be described below, which will not be repeated here.

[0050] 103. Associate the same object in the first detection result and the second detection result to obtain an association result; Wherein, the association result at least indicates the associated object and / or the unassociated object in the first detection result and the second detection result.

[0051] The associated object can indicate the common object detected from the infrared image and the visible light image. The unassociated object can indicate the object detected only from the infrared image or the visible light image.

[0052] The specific implementation of 103 includes but is not limited to the following two: Firstly, the distance between two objects of the same category in the first detection result and the second detection result is calculated. The distance can be the distance between the center positions of the detection boxes of the two objects, and can also be the GIoU distance of the detection boxes of the two objects.

[0053] When the distance between the two objects of the same category is less than a preset distance, the two objects of the same category are determined to be the same object (i.e., associated objects). The preset distance can be set according to experimental requirements.

[0054] The objects in the first detection result and the second detection result except the associated objects are determined to be unassociated objects. In this way, the set of associated objects and unassociated objects is the association result.

[0055] Secondly, as shown in Figure 3 , the method can include the following steps: 301. According to the category and position of each object in the first detection result and the category and position of each object in the second detection result, a first matrix of I rows and J columns is constructed.

[0056] Wherein, I indicates the number of objects in the first detection result (the number of objects detected in the detection area of the infrared image), J indicates the number of objects in the second detection result (the number of objects detected in the detection area of the visible light image), and the element located at the intersection of the ith row and the jth column indicates the distance between the object corresponding to the ith row and the object corresponding to the jth column. Each row corresponds to an object detected from the detection area of the infrared image, and each column corresponds to an object detected from the detection area of the visible light image. i is an integer greater than zero and less than or equal to I, and j is an integer greater than zero and less than or equal to J.

[0057] For example, the first matrix may be as follows: ; The distance between the object corresponding to the ith row and the object corresponding to the jth column is calculated as follows: When the categories of the object corresponding to the ith row and the object corresponding to the jth column are different, the distance is set to infinity. When the categories of the object corresponding to the ith row and the object corresponding to the jth column are the same, the calculation formula of the distance may be as follows:

[0058] In the formula, an area of a minimum bounding rectangle of a detection frame of the object corresponding to the i-th row and a detection frame of the object corresponding to the j-th column, an intersection over union, an area of an intersection part of a detection frame of the object corresponding to the i-th row and a detection frame of the object corresponding to the j-th column, an area of a union part of a detection frame of the object corresponding to the i-th row and a detection frame of the object corresponding to the j-th column, i.e., U.

[0059] 302. Based on the first matrix, each element in a row is subtracted by a minimum element in the corresponding row, and each element in a column is subtracted by a minimum element in the corresponding column, to obtain a second matrix.

[0060] Specifically, in the first matrix, each element in a row is subtracted by a minimum element in the corresponding row, and each element in a column is subtracted by a minimum element in the corresponding column in the calculated matrix, to obtain a second matrix.

[0061] 303. Perform a covering step: cover the zero elements in the second matrix with the minimum number of straight lines. Wherein, the straight lines include horizontal straight lines and vertical straight lines.

[0062] 304. In the case where the number of straight lines is equal to the number of rows of the second matrix, determine the second matrix as a third matrix.

[0063] 305. In the case where the number of straight lines is less than the number of rows of the second matrix, subtract each non-zero element in the second matrix by a minimum non-zero element of the second matrix, and add the minimum non-zero element to the zero elements in the second matrix covered by the straight lines (i.e., covered by the horizontal straight lines and the vertical straight lines), to obtain a fourth matrix.

[0064] 306. Take the fourth matrix as the second matrix, and start from the covering step (303) to perform until the number of straight lines is equal to the number of rows of the second matrix.

[0065] 307. Determine the row object (the object detected in the detection area of the infrared image) and the column object (the object detected in the detection area of the visible light image) corresponding to the zero element in the row with only one zero element in the third matrix as the associated object, and determine the objects in the first detection result and the second detection result except the associated object as the unassociated objects.

[0066] 104. Based on the multiple local features of each object in the association result, perform fusion verification on each object in the association result, and output a detection result according to the fusion verification result.

[0067] The plurality of local features include, but are not limited to, at least two of local entropy, local contrast, local gradient, and local texture. It should be noted that in the following, the plurality of local features are exemplarily described by taking the four local features as an example.

[0068] Exemplarily, the association result can include at least one of a first object, a second object, and a third object, the first object being an unassociated object in the first detection result, the second object being an unassociated object in the second detection result, and the third object being an associated object in the first detection result and the second detection result.

[0069] It can be understood that the first object and the second object are unassociated objects in the association result, and the third object is an associated object in the association result.

[0070] Based on this, as shown in Figure 4 , the implementation manner of the fusion verification can be as follows: 401. Extracting, from the infrared image, a plurality of local features of each object in the first detection result.

[0071] The local entropy reflects the gray level dispersion degree of the image region, and the local entropy of an object can be determined according to the height and width of the region occupied by the object in the infrared image and the gray level value of each position in the region occupied by the object in the infrared image.

[0072] Exemplarily, the local entropy of an object can be obtained by the following formula:

[0073] wherein M and N are the width and height of the region occupied by the object in the infrared image, respectively, and G is the gray level value of the position in the region occupied by the object in the infrared image.

[0074] The local contrast reflects the detail information of the image region, and the local contrast of an object can be determined according to the inner window and the outer window of the object in the infrared image, the pixel number and the pixel distribution of the inner window, the pixel number and the pixel distribution of the outer window, and the pixel value of the corresponding position.

[0075] Exemplarily, the local contrast of an object can be obtained by the following formula:

[0076] wherein W and H are the inner window and the outer window of the object in the infrared image, respectively, W is the pixel number of the inner window of the object in the infrared image, H is the pixel number of the outer window of the object in the infrared image, and N is the pixel number of the inner window of the object in the infrared image.​​​ the number of pixels of the outer window of the object in the infrared image, the mean of the pixel distribution of the inner window of the object in the infrared image, the mean of the pixel distribution of the outer window of the object in the infrared image, the pixel value of the corresponding position.

[0077] The local gradient reflects the edge intensity of the image region, and the local gradient of an object can be determined according to the gradients of the region occupied by the object in the infrared image in horizontal and vertical directions and the region occupied by the object in the infrared image.

[0078] Exemplarily, the local gradient of an object can be obtained by the following formula:

[0079] wherein, the gradient of the region occupied by the object in the infrared image in horizontal direction, the gradient of the region occupied by the object in the infrared image in vertical direction, I is the region occupied by the object in the infrared image, and * represents convolution calculation.

[0080] The local texture reflects the texture difference between the object and the background, and the local texture of an object can be determined according to the inner window of the object in the infrared image, the pixel distribution of the inner window and the pixel value of the corresponding position.

[0081] Exemplarily, the local texture of an object can be obtained by the following formula:

[0082] wherein, the inner window of the object in the infrared image, the mean of the pixel distribution of the inner window of the object in the infrared image, the pixel value of the corresponding position.

[0083] Obviously, by the above formula, the local entropy, the local contrast, the local gradient and the local texture of each object in the first detection result can be extracted from the infrared image.

[0084] 402. Extracting a plurality of local features of each object in the second detection result from the visible light image.

[0085] The specific implementation process of 402 can refer to 401, and the difference lies in that the related parameters of the infrared image in the formula are replaced by the related parameters in the visible light image.

[0086] ​​403. For the first object, probabilities of the first object being the object, the background and the uncertain region are determined according to the local features of the first object based on the infrared image.

[0087] For example, the implementation process can be as follows: First, four groups of classification probabilities of the first object are determined according to the local entropy, the local contrast, the local gradient and the local texture of the first object based on the infrared image.

[0088] The four groups of classification probabilities of the first object correspond to the four local features one by one. Each group of classification probabilities of the first object includes probabilities of the first object being the object, the background and the uncertain region under the corresponding local feature.

[0089] For example, the probabilities of the first object being the object (A), the background (B) and the uncertain region (U) under any one of the local features of the first object based on the infrared image can be calculated according to the local feature and in combination with the following formula.

[0090]

[0091] is the probability of the first object being the object under any one of the local features, is the probability of the first object being the background under any one of the local features, is the probability of the first object being the uncertain region under any one of the local features, is any one of the local features of the first object based on the infrared image.

[0092] The probabilities of the first object being the object, the background and the uncertain region under the local entropy are determined as a group of classification probabilities of the first object by the above formula.

[0093] Similarly, the probabilities of the first object being the object, the background and the uncertain region under the local contrast are determined as another group of classification probabilities of the first object by the above formula.

[0094] The probabilities of the first object being the object, the background and the uncertain region under the local gradient are determined as another group of classification probabilities of the first object by the above formula.

[0095] The probabilities of the first object being the object, the background and the uncertain region under the local texture are determined as another group of classification probabilities of the first object by the above formula.

[0096] In this way, the four groups of classification probabilities of the first object are finally obtained.

[0097] ​​​​Then, according to the four sets of classification probabilities of the first object, probabilities of the first object being the object, the background and the uncertain region are determined respectively.

[0098] For example, firstly, the local entropy of the first object is taken as , the local contrast of the first object is taken as , and the conflict factor K is calculated. K can be calculated according to the probabilities of the first object being the object and the background under the local entropy and the local contrast respectively.

[0099] For example, the calculation formula of K is as follows:

[0100] wherein, is the probability of the first object being the object under the local entropy, is the probability of the first object being the background under the local entropy, is the probability of the first object being the object under the local contrast, is the probability of the first object being the background under the local contrast.

[0101] Then, based on K, the probabilities of the first object being the object, the background and the uncertain region under the local entropy respectively, and the probabilities of the first object being the object, the background and the uncertain region under the local contrast respectively, probability fusion is performed to obtain the fused probabilities. For example, the fusion is performed by the following formula:

[0102] wherein, , and represent the probabilities of the first object being the object, the background and the uncertain region after fusion respectively, is the probability of the first object being the uncertain region under the local entropy, is the probability of the first object being the uncertain region under the local contrast.

[0103] Next, , and are reset to , the local gradient of the first object is taken as , the above process is repeated to obtain the fused result.

[0104] Finally, the above result after the second fusion is reset to , the local texture of the first object is taken as , the above process is repeated, and the obtained fused result is determined as the final result, i.e. the probabilities of the first object being the object, the background and the uncertain region are obtained.

[0105] 404. For the second object, determine probabilities of the second object being the object, the background and the uncertain region respectively according to the plurality of local features of the second object based on the visible light image.

[0106] For example, the implementation process can be as follows: First, determine four groups of classification probabilities of the second object respectively according to local entropy, local contrast, local gradient and local texture of the second object based on the visible light image.

[0107] The four groups of classification probabilities of the second object correspond to the four local features respectively. Each group of classification probabilities of the four groups of classification probabilities of the second object includes probabilities of the second object being the object, the background and the uncertain region respectively under the corresponding local feature.

[0108] It should be noted that the implementation process of this step can refer to the related content in the 403 step, and the difference is that the corresponding parameters of the infrared image in the formula are replaced by the corresponding parameters of the visible light image.

[0109] Then, determine probabilities of the second object being the object, the background and the uncertain region respectively according to the four groups of classification probabilities of the second object.

[0110] It should be noted that the implementation process of this step can refer to the related content in the 403 step.

[0111] 405. For the third object, determine probabilities of the third object being the object, the background and the uncertain region respectively according to a plurality of local features of the third object based on the infrared image and the visible light image respectively.

[0112] For example, as shown in Figure 5 the implementation process of this step can be as follows: 501. Determine four groups of classification probabilities of the third object based on the infrared image respectively according to local entropy, local contrast, local gradient and local texture of the third object based on the infrared image.

[0113] The four groups of classification probabilities of the third object based on the infrared image correspond to the four local features respectively.

[0114] Each group of classification probabilities of the four groups of classification probabilities of the third object based on the infrared image includes probabilities of the third object being the object, the background and the uncertain region respectively under the corresponding local feature based on the infrared image.

[0115] 502. Determine first probabilities of the third object being the object, the background and the uncertain region respectively according to the four groups of classification probabilities of the third object based on the infrared image.

[0116] It should be noted that the specific implementation process of 501 and 502 can refer to the related content in 403, which will not be repeated here.

[0117] 503. Determine four groups of classification probabilities of the third object based on the visible light image according to the local entropy, the local contrast, the local gradient and the local texture of the third object based on the visible light image.

[0118] The four groups of classification probabilities of the third object based on the visible light image correspond to the four local features one by one; each group of classification probabilities of the four groups of classification probabilities of the third object based on the visible light image includes probabilities of the third object based on the visible light image and being the object, the background and the uncertain area under the corresponding local feature.

[0119] 504. Determine the second probabilities of the third object being the object, the background and the uncertain area according to the four groups of classification probabilities of the third object based on the visible light image.

[0120] It should be noted that the specific implementation process of 503 and 504 can refer to the related content in 404, which will not be repeated here.

[0121] 505. Determine the probabilities of the third object being the object, the background and the uncertain area according to the first probabilities and the second probabilities of the third object being the object, the background and the uncertain area.

[0122] For example, the first probabilities of the third object being the object, the background and the uncertain area are taken as the related parameters of formula (13)~formula (16), the second probabilities of the third object being the object, the background and the uncertain area are taken as the related parameters of formula (13)~formula (16), and the final fusion results are determined as the probabilities of the third object being the object, the background and the uncertain area by executing formula (13)~formula (16).

[0123] 406. Output the detection result according to the probabilities of the first object, the second object and the third object being the object, the background and the uncertain area.

[0124] The background is an area outside the detection area in the infrared image and the visible light image.

[0125] The specific implementation process can be as follows: Filter out the object whose probability of being the object is higher than that of being the background and the uncertain area from the first object, the second object and the third object, and take the filtered object as the fourth object.

[0126] In the case that the fourth object is the associated object, output the position and the category of the fourth object according to the position and the category of the fourth object in the infrared image and the visible light image.

[0127] ​​For example, the position of the detection box of the fourth object in the infrared image and the position of the detection box of the fourth object in the visible light image are subjected to average weighting processing to obtain the position of the fourth object.

[0128] In the case where the fourth object is an unassociated object, the position and category of the fourth object are output according to the position and category of the fourth object in the infrared image or the visible light image.

[0129] It should be noted that after the fourth object is determined, the position and category of the fourth object are output, and the detection boxes and categories of all objects other than the fourth object among the first object, the second object and the third object are deleted. Obviously, the fourth object is further screened according to the probability, which further ensures the accuracy and reliability of the detection.

[0130] In some possible embodiments, after the position and category of the fourth object are output, the intruder corresponding to the fourth object can also be found according to the position and category of the fourth object. In addition, it is judged whether the intruder will cause a safety event, for example, in the field of rail transit, it is judged whether the intruder will affect the safe operation of the train. If a safety event is caused, information is sent to the dispatch center and the maintenance center for safety investigation and maintenance. If a safety event is not caused, the maintenance center can be fed back so as to perform safety investigation and maintenance after the emergency application processing is completed.

[0131] As can be seen from the above, since the infrared image itself has the penetration advantage in complex environments such as low light and smoke shielding, and the visible light image itself has the advantages in details, resolution and color information, etc., compared with the way of target detection only through the visible light image, the embodiments of the present application can improve the accuracy of target detection by detecting the objects in the detection region in the infrared image of the first region and the detection region in the visible light image, and reduce the influence of complex environments on the detection result.

[0132] Moreover, since the objects in the detection region are detected, the user can set the region to be detected according to the demand, and detection is only performed in the detection region, so that the detection efficiency is improved and the detection energy consumption is reduced.

[0133] Moreover, the same objects in the first detection result and the second detection result are associated, and each object in the association result is subjected to fusion verification based on multiple local features of each object in the association result, so as to output the detection result according to the fusion verification result. That is, by associating the detection results of the infrared image and the visible light image, the information complementarity of different image modalities is fully utilized, so that the objects can be more comprehensively and accurately detected. Moreover, each object in the association result is subjected to fusion verification based on multiple local features, which avoids the possible misjudgment due to single feature judgment, greatly improves the accuracy of verification, and makes the output detection result more reliable and accurate.

[0134] As shown in Figure 6 The training process of the detection model can include the following steps: 601. Obtain a plurality of visible light training samples and a plurality of infrared training samples.

[0135] The plurality of infrared training samples can be obtained by an infrared camera, and the plurality of visible light training samples can be obtained by a visible light camera.

[0136] It should be noted that the visible light camera and the infrared camera are installed in the scene where the target detection method is applied to ensure that the obtained visible light training samples and infrared training samples fit the application scene of the target detection method.

[0137] For example, in a rail transit scene, the visible light camera and the infrared camera can be installed above the area where the rail is located. The infrared training samples and visible light training samples including the detected objects collected at different times (day, night, etc.) and in different weather conditions (rain, snow, and sunny weather, etc.) are obtained by the visible light camera and the infrared camera to improve the diversity of the samples.

[0138] 602. Perform background processing on the area outside the detection area in each visible light training sample and each infrared training sample, respectively.

[0139] The background processing can be completed by humans or by devices. The device-completed mode can be as follows: Identify the detection area in each visible light training sample and each infrared training sample by a network model. Fill the area outside the detection area in each visible light training sample and each infrared training sample with the background, such as changing to white, black, or performing blur processing, etc.

[0140] The network model can use edge recognition of the detection area based on Unet semantic segmentation. The basic idea is to classify each pixel point in the image. The network structure has an encoder and a reverse decoder. The encoder is responsible for feature extraction and down-sampling of the input picture. In this stage, the input image size is reduced by 16 times through 4 maximum value pooling. When decoding, the feature map size is restored to the original size through 4 times 2 times bilinear up-sampling. The same size feature map is connected across layers, and finally the classification result of each pixel point is output. The scene outside the edge of the detection area is filled as the background.

[0141] 603. Label the position and category of the object in each visible light training sample and each infrared training sample after background processing.

[0142] Since the background processing is performed, the labeling amount of the position and category of the object is reduced, and the labeling efficiency is improved.

[0143] 604. Augment the visible light training samples and the infrared training samples to which the objects belonging to the category with small proportion belong.

[0144] For the visible light training samples, the proportion of the objects of each category is calculated. When the proportion is small, the visible light training samples to which the object belongs are augmented.

[0145] Specifically, the visible light training samples can be augmented by image coordinate transformation, which is realized by the following formula:

[0146] wherein, is the original pixel coordinate, is the transformed coordinate. By changing the form of T, the rotation, translation, scaling and other operations of the image are realized, and finally the purpose of expanding the data set is realized.

[0147] For the infrared training samples, the augmentation method is the same as above, which will not be repeated here.

[0148] By augmenting the samples, the diversity of the samples can be further improved, thereby improving the accuracy of the detection model training.

[0149] It should be noted that the augmented training samples here are the background processed training samples.

[0150] 605. Train the neural network by using the visible light training samples and the infrared training samples after labeling and augmentation to obtain a detection model.

[0151] The neural network may, for example, adopt a target detection based on YOLOv10n. The YOLOv10n algorithm can be composed of Input, Backbone, Neck and Head, a total of 4 parts. The Input is the input end, which is mainly responsible for processing various operations of the input image to ensure that the image meets the input requirements of the model. The Backbone part undertakes the core task of feature extraction in YOLOv10n. It extracts rich feature information from the input image through a series of complex convolutional layers and pooling layers, so that the model can better capture the key information in the image. The main responsibility of the Neck end is to effectively integrate the feature information of different levels. While retaining the classic structure of FPN and PANet, the Neck end achieves effective fusion of shallow features and deep features through ingenious design. The Head end adopts an innovative decoupled head structure, which separates the regression branch and the prediction branch, making the model more efficient and accurate in handling target detection tasks.

[0152] The visible light training sample and the infrared training sample after labeling and expansion are input into the neural network, and the neural network is trained by adjusting the weight of the parameters in the neural network, so as to determine the trained neural network as the detection model.

[0153] As Figure 7 The electronic device 700 provided by the embodiment of the present application can include a processor 710 and a memory 720; the memory 720 is configured to store a computer program; and the processor 710 is configured to implement the target detection method as described above when executing the computer program.

[0154] The computer readable storage medium provided by the embodiment of the present application stores a computer program, and when the computer program is executed by a processor, the target detection method as described above is implemented.

[0155] An electronic device 700 that can be a server or a client of the present application will now be described, which is an example of a hardware device that can be applied to various aspects of the present application. The electronic device 700 is intended to represent various forms of digital electronic computing devices, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computing devices. The electronic device 700 can also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown here, their connections, and their functions, as described herein, are meant to be examples only, and are not intended to limit the implementations of the present application described and / or claimed herein.

[0156] The electronic device 700 includes a computing unit that can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) or a computer program loaded from a storage unit into a random access memory (RAM). In the RAM, various programs and data required for device operation can also be stored. The computing unit, the ROM, and the RAM are connected to each other through a bus. An input / output (I / O) interface is also connected to the bus.

[0157] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by computer program instructions related hardware, and the program can be stored in a computer readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiment methods. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0158] The above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application. Within the scope of the technical essence of the present application, those skilled in the art can modify or replace the technical features of the technical solutions described in the above embodiments, and these modifications or replacements still belong to the protection scope of the present application.

Claims

1. A target detection method characterized by, The method comprises: obtaining an infrared image and a visible light image of a first region; detecting objects in a detection region in the infrared image and a detection region in the visible light image respectively to obtain a first detection result and a second detection result; wherein the first detection result indicates objects detected from the infrared image, and the second detection result indicates objects detected from the visible light image; associating the same objects in the first detection result and the second detection result to obtain an association result; wherein the association result indicates at least associated objects and / or unassociated objects in the first detection result and the second detection result; based on a plurality of local features of each object in the association result, performing fusion verification on each object in the association result, and outputting a detection result according to a fusion verification result.

2. The method of claim 1, wherein, The association result comprises at least one of a first object, a second object and a third object, the first object is an unassociated object in the first detection result, the second object is an unassociated object in the second detection result, and the third object is an associated object in the first detection result and the second detection result; The method based on a plurality of local features of each object in the association result, performing fusion verification on each object in the association result, and outputting a detection result according to a fusion verification result comprises: extracting a plurality of local features of each object in the first detection result from the infrared image; extracting a plurality of local features of each object in the second detection result from the visible light image; for the first object, determining probabilities that the first object is an object, a background and an uncertain region respectively according to a plurality of local features of the first object based on the infrared image; for the second object, determining probabilities that the second object is an object, a background and an uncertain region respectively according to a plurality of local features of the second object based on the visible light image; for the third object, determining probabilities that the third object is an object, a background and an uncertain region respectively according to a plurality of local features of the third object based on the infrared image and the visible light image respectively; outputting a detection result according to the probabilities that the first object, the second object and the third object are an object, a background and an uncertain region respectively; wherein the background is a region outside the detection region in the infrared image and the visible light image.

3. The method of claim 2, wherein, The plurality of local features comprises four local features, namely local entropy, local contrast, local gradient and local texture; The method of determining the probabilities that the first object is an object, a background and an uncertain region respectively according to a plurality of local features of the first object based on the infrared image comprises: determining four sets of classification probabilities of the first object respectively according to local entropy, local contrast, local gradient and local texture of the first object based on the infrared image; wherein the four sets of classification probabilities of the first object correspond to the four local features one by one. Each of the four sets of classification probabilities of the first object comprises probabilities of the first object being an object, a background, and an uncertain region under a corresponding local feature, respectively; The probabilities of the first object being an object, a background, and an uncertain region are determined according to the four sets of classification probabilities of the first object.

4. The method of claim 2, wherein, The plurality of local features comprises four local features, which are local entropy, local contrast, local gradient, and local texture, respectively; The determining of the probabilities of the second object being an object, a background, and an uncertain region based on the plurality of local features of the second object based on the visible light image comprises: The four sets of classification probabilities of the second object are determined based on the local entropy, the local contrast, the local gradient, and the local texture of the second object based on the visible light image, respectively; The four sets of classification probabilities of the second object correspond to the four local features one by one; Each of the four sets of classification probabilities of the second object comprises probabilities of the second object being an object, a background, and an uncertain region under a corresponding local feature, respectively; The probabilities of the second object being an object, a background, and an uncertain region are determined according to the four sets of classification probabilities of the second object.

5. The method of claim 2, wherein, The plurality of local features comprises four local features, which are local entropy, local contrast, local gradient, and local texture, respectively; The determining of the probabilities of the third object being an object, a background, and an uncertain region based on the plurality of local features of the third object based on the infrared image and the visible light image comprises: The four sets of classification probabilities of the third object based on the infrared image are determined based on the local entropy, the local contrast, the local gradient, and the local texture of the third object based on the infrared image, respectively; The four sets of classification probabilities of the third object based on the infrared image correspond to the four local features one by one; Each of the four sets of classification probabilities of the third object based on the infrared image comprises probabilities of the third object being an object, a background, and an uncertain region under a corresponding local feature based on the infrared image, respectively; The first probabilities of the third object being an object, a background, and an uncertain region are determined according to the four sets of classification probabilities of the third object based on the infrared image; The four sets of classification probabilities of the third object based on the visible light image are determined based on the local entropy, the local contrast, the local gradient, and the local texture of the third object based on the visible light image, respectively; The four sets of classification probabilities of the third object based on the visible light image correspond to the four local features one by one; Each of the four sets of classification probabilities of the third object based on the visible light image comprises probabilities of the third object being an object, a background, and an uncertain region under a corresponding local feature based on the visible light image, respectively; The second probabilities of the third object being an object, a background, and an uncertain region are determined according to the four sets of classification probabilities of the third object based on the visible light image. According to the first probability and the second probability that the third object is an object, a background and an uncertain area respectively, a probability that the third object is an object, a background and an uncertain area respectively is determined.

6. The method of claim 2, wherein, The outputting of the detection result according to the probability that the first object, the second object and the third object are an object, a background and an uncertain area respectively comprises: An object with a probability higher than the background and the uncertain area is selected from the first object, the second object and the third object as a fourth object; In a case where the fourth object is the associated object, a position and a category of the fourth object are output according to the position and the category of the fourth object in the infrared image and the visible light image; In a case where the fourth object is the unassociated object, a position and a category of the fourth object are output according to the position and the category of the fourth object in the infrared image or the visible light image.

7. The method of claim 1, wherein, The associating of the same object in the first detection result and the second detection result to obtain an associated result comprises: A first matrix of I rows and J columns is constructed according to the category and the position of each object in the first detection result and the category and the position of each object in the second detection result; Wherein, I indicates the number of objects in the first detection result, J indicates the number of objects in the second detection result, and an element located at the intersection of the ith row and the jth column indicates the distance between the object corresponding to the ith row and the object corresponding to the jth column; On the basis of the first matrix, each row element is subtracted by the minimum element in the corresponding row, and each column element is subtracted by the minimum element in the corresponding column to obtain a second matrix; A covering step is performed to cover the zero elements in the second matrix with the least number of straight lines; In a case where the number of straight lines is equal to the number of rows of the second matrix, the second matrix is determined as a third matrix; In a case where the number of straight lines is less than the number of rows of the second matrix, each non-zero element in the second matrix is subtracted by the minimum non-zero element of the second matrix, and the zero element covered by the straight line in the second matrix is added by the minimum non-zero element to obtain a fourth matrix; After the fourth matrix is taken as the second matrix, the covering step is performed until the number of straight lines is equal to the number of rows of the second matrix; The row object and the column object corresponding to the zero element in the row with only one zero element in the third matrix are determined as the associated object, and the objects in the first detection result and the second detection result except the associated object are determined as the unassociated object.

8. The method of claim 1, wherein, The detecting of the object in the detection area in the infrared image and the detection area in the visible light image respectively to obtain the first detection result and the second detection result comprises: The detection area is identified from the infrared image and the visible light image respectively; The area outside the detection area in the infrared image and the visible light image is backgrounded respectively. respectively detecting objects in the infrared image and the visible light image after background processing to obtain the first detection result and the second detection result.

9. The method of claim 1, wherein, The detecting objects in the detection region in the infrared image and the detection region in the visible light image respectively comprises: detecting objects in the detection region in the infrared image and the detection region in the visible light image respectively by a detection model.

10. The method of claim 9, wherein, The detection model is trained, and the training process comprises: obtaining a plurality of visible light training samples and a plurality of infrared training samples; respectively performing background processing on regions outside the detection region in each visible light training sample and each infrared training sample; annotating the positions and categories of objects in each visible light training sample and each infrared training sample after background processing; augmenting the visible light training samples and the infrared training samples to which the objects with small category proportions belong; training a neural network by the visible light training samples and the infrared training samples after annotation and augmentation to obtain the detection model.

Images