Object Detection Method, Device, Storage Medium, and Electronic Device

Abstract

An embodiment of the present application provides an object detection method, apparatus, storage medium, and electronic device. The object detection method includes: performing object detection on a target image, where the target image is a picture obtained by a visible light imaging device photographing an airport runway; when a target object is detected, determining a second region and a third region corresponding to a first region in a first infrared image and a second infrared image respectively, and determining a first heat difference based on the first infrared image and the second region, and determining a second heat difference based on the second infrared image and the third region. The first heat difference is the difference between the heat corresponding to the second region in the first infrared image and the heat corresponding to the pixel points adjacent to the second region, and the second heat difference is the difference between the heat corresponding to the second region and the heat corresponding to the third region; determining whether the target object is an abnormal object on the airport runway according to the first heat difference and the second heat difference.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computers, and in particular, to an object detection method, an apparatus, a storage medium, and an electronic device. Background Art

[0002] The presence of foreign objects on the runway poses a huge safety hazard to aircraft takeoff. Currently, for the detection of foreign objects on the runway, common detection methods include millimeter-wave radar detection and image recognition of vehicle-mounted videos. However, these two types of FOD (Foreign Object Debris) detection systems cannot identify the interference items inherent in the runway itself, such as runway surface texture, material, runway centerline lights, and marking lines, resulting in a high false alarm rate.

[0003] It can be seen that the object detection methods in the related art have the problem of low detection accuracy for foreign objects on the runway. Summary of the Invention

[0004] The embodiments of the present application provide an object detection method, an apparatus, a storage medium, and an electronic device to at least solve the problem that the object detection methods in the related art have low detection accuracy for foreign objects on the runway.

[0005] According to an embodiment of the present application, an object detection method is provided, including: performing object detection on a target picture to obtain a first detection result, where the target picture is a picture obtained by a visible light imaging device photographing an airport runway; in the case where the first detection result indicates that a target object is detected from the target picture, determining regions corresponding to a first region in a first infrared picture and a second infrared picture respectively to obtain a second region and a third region, and determining a first heat difference according to the first infrared picture and the second region, and determining a second heat difference according to the second infrared picture and the third region, where the first region is the region where the target object is located in the target picture, the second region and the third region represent the same region as the first region, the first heat difference is the difference between the heat corresponding to the second region in the first infrared picture and the heat corresponding to the pixel points adjacent to the second region, the first infrared picture is an infrared picture taken at the same time as the target picture, the second heat difference is the difference between the heat corresponding to the second region in the first infrared picture and the heat corresponding to the third region in the second infrared picture, and the second infrared picture is an infrared picture obtained by an infrared camera photographing the airport runway before the shooting time of the first infrared picture; determining a second detection result of the target object according to the first heat difference and the second heat difference, where the second detection result is used to indicate whether the target object is an abnormal object appearing on the airport runway.

[0006] According to another embodiment of the present application, an object detection device is provided, including: a detection unit configured to perform object detection on a target picture to obtain a first detection result, where the target picture is a picture obtained by a visible light imaging device photographing an airport runway; a first determination unit configured to, when the first detection result indicates that a target object is detected in the target picture, respectively determine areas corresponding to a first area in a first infrared picture and a second infrared picture to obtain a second area and a third area, and determine a first heat difference according to the first infrared picture and the second area, and determine a second heat difference according to the second infrared picture and the third area, where the first area is the area where the target object is located in the target picture, the second area and the third area represent the same area as the first area, the first heat difference is the difference between the heat corresponding to the second area in the first infrared picture and the heat corresponding to the pixel points adjacent to the second area, the first infrared picture is an infrared picture taken at the same time as the target picture, the second heat difference is the difference between the heat corresponding to the second area in the first infrared picture and the heat corresponding to the third area in the second infrared picture, and the second infrared picture is an infrared picture obtained by an infrared camera photographing the airport runway before the shooting time of the first infrared picture; a second determination unit configured to determine a second detection result of the target object according to the first heat difference and the second heat difference, where the second detection result is used to indicate whether the target object is an abnormal object appearing on the airport runway.

[0007] According to another embodiment of the present application, a computer-readable storage medium is further provided, in which a computer program is stored, and the computer program is configured to execute the steps in any one of the above method embodiments when running.

[0008] According to another embodiment of the present application, an electronic device is further provided, including a memory and a processor, where a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0009] Through the embodiments of the present application, by using the method of performing object recognition based on the pictures captured by a visible light imaging device and verifying the recognition result with the corresponding infrared pictures, after detecting a target object suspected of being a foreign object and the first area of the target object in the target picture captured by the visible light imaging device, by combining the difference between the area corresponding to the target object and the adjacent area in the infrared picture at the same moment, and the difference between the areas corresponding to the target object in the infrared pictures at different moments, to verify the detected target object, it is possible to avoid the target object recognized being an interference item inherent in the airport runway itself, achieving the technical effect of improving the detection accuracy of foreign objects on the airport runway, and further solving the problem that the object detection method in the related art has a low detection accuracy for foreign objects on the airport runway. Description of the Drawings

[0010] Figure 1 is a schematic diagram of the hardware environment of an object detection method according to an embodiment of the present application;

[0011] Figure 2 is a schematic flowchart of an object detection method according to an embodiment of the present application;

[0012] Figure 3 is a schematic flowchart of another object detection method according to an embodiment of the present application;

[0013] Figure 4 is a schematic diagram of an object detection method according to an embodiment of the present application;

[0014] Figure 5 is a schematic diagram of another object detection method according to an embodiment of the present application;

[0015] Figure 6 is a schematic flowchart of yet another object detection method according to an embodiment of the present application;

[0016] Figure 7 is a schematic flowchart of yet another object detection method according to an embodiment of the present application;

[0017] Figure 8 is a structural block diagram of an object detection device according to an embodiment of the present application. Detailed Embodiments

[0018] In the following, the embodiments of the present application will be described in detail with reference to the drawings and in combination with the embodiments.

[0019] It should be noted that the terms "first", "second", etc. in the description and claims of the embodiments of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence.

[0020] The method embodiments provided in the embodiments of the present application can be executed on a mobile terminal, a computer terminal, or a similar computing device. Taking running on a computer terminal as an example, Figure 1 is a schematic diagram of the hardware environment of an object detection method according to an embodiment of the present application. As Figure 1 shown, the computer terminal may include one or more ( Figure 1 only one is shown in ) a processor 102 (the processor 102 may include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. Among them, the above computer terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those of ordinary skill in the art can understand that Figure 1 the structure shown is only schematic and does not limit the structure of the above computer terminal. For example, the computer terminal may further include more or fewer components than Figure 1 shown, or have a different configuration from Figure 1 shown.

[0021] The memory 104 can be used to store computer programs. For example, software programs and modules of application software, such as the computer program corresponding to the object detection method in the embodiments of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implements the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some instances, the memory 104 may further include a memory remotely disposed relative to the processor 102, and these remote memories may be connected to the mobile terminal through a network. Examples of the above network include, but are not limited to, the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0022] The transmission device 106 is used to receive or send data via a network. Specific examples of the above network may include a wireless network provided by a communication provider of the computer terminal. In one instance, the transmission device 106 includes a network adapter (abbreviated as NIC), which can be connected to other network devices through a base station and thus can communicate with the Internet. In one instance, the transmission device 106 may be a radio frequency (abbreviated as RF) module, which is used to communicate with the Internet wirelessly.

[0023] According to one aspect of the embodiments of the present application, an object detection method is provided. Taking the object detection method in this embodiment to be executed by a computer terminal as an example, Figure 2It is a flowchart of an object detection method according to an embodiment of the present application. As Figure 2 shown, the process includes the following steps:

[0024] Step S202: Perform object detection on the target picture to obtain a first detection result, where the target picture is a picture obtained by a visible light imaging device photographing an airport runway.

[0025] The object detection method in this embodiment can be applied to the scenario of identifying foreign objects on an airport runway. Foreign objects on an airport runway generally refer to some foreign substances, debris or objects that may damage an aircraft on the airport runway. Such as metal parts, waterproof plastic sheets, gravel blocks, newspapers, bottles, luggage tags, etc., which are collectively referred to as foreign objects. Typical ones include: metal devices (nuts, screws, washers, nails, fuses, etc.), mechanical tools, flying objects (personal items, pens, pencils, buttons, etc.), rubber fragments, plastic products, concrete and asphalt fragments (stones, sand, ice slag, etc.), paper products, animals and plants, etc. If these items are sucked into the engine of an aircraft during takeoff, it will cause the engine to fail or even explode, seriously endangering the flight safety of the aircraft and passengers.

[0026] In order to avoid huge safety hazards caused by foreign objects on the airport runway to aircraft takeoff, it is necessary to perform timely foreign object detection before each aircraft takes off. Currently, common FOD detection systems include millimeter-wave radar detection and image recognition of vehicle-mounted videos. However, these two types of FOD detection systems cannot identify interference items inherent in the airport runway itself, such as runway surface texture, material, runway centerline lights and marking lines, resulting in a relatively high false alarm rate. In addition, the cost of millimeter-wave radar is relatively high. The vehicle-mounted video method is easily affected by aircraft takeoff and requires manual inspection at regular intervals.

[0027] In order to solve at least some of the above problems, multiple visible light imaging devices and infrared imaging devices can be installed on both sides of the airport runway. Perform object recognition on the pictures taken by the visible light imaging devices, and then use the infrared pictures taken by the infrared imaging devices for assistance to verify the identified objects, which can avoid the identified target objects being interference items inherent in the airport runway itself, thereby improving the detection accuracy of foreign objects on the airport runway. In addition, the visible light imaging devices and infrared imaging devices fixed on both sides of the airport runway can effectively reduce the detection cost compared with millimeter-wave radar. At the same time, since the positions of the visible light imaging devices and infrared imaging devices are fixed, they are less affected by aircraft takeoff.

[0028] In this embodiment, object detection can be performed on the target picture obtained by the visible light imaging device photographing the airport runway to obtain a first detection result. Here, the first detection result may be that no foreign object is detected, or a target object suspected of being a foreign object is detected.

[0029] Optionally, when the first detection result is that the target object is detected, the first detection result may further include information such as the region of the target object in the target image, the actual position of the target object on the airport runway, and the type of the target object.

[0030] Optionally, before detecting the target object, the relevant detection model can be trained first using the image data in the training pool. The training process includes preprocessing the images and data augmentation, and then performing target detection and target segmentation operations. The preprocessing here can be noise extraction and size scaling of the target images. The data augmentation can be to indicate feature enhancement of the image data.

[0031] The above training operations on the relevant detection model may include the following steps:

[0032] Step 1, set the number of training samples and extract corresponding images.

[0033] Step 2, perform image preprocessing, remove the noise in the images, and scale the image size to a specific size. The specific size can be set to 640*640. During the process of scaling the image size, the aspect ratio of the image can be kept unchanged, the maximum side is scaled to 640, and the smaller side is filled with grayscale.

[0034] Step 3, perform feature enhancement of the training data.

[0035] The feature enhancement uses Mosaic (a data augmentation method) and MixUp (a data augmentation method) to perform data augmentation on the images.

[0036] Step 4, the processed images enter ResNet50 (Residual Neural Network 50, a 50-layer residual network).

[0037] Step 5, the images enter FPN (feature pyramid networks) to extract feature maps.

[0038] Step 6, perform RPN (Region Proposal Network) on the extracted feature maps to obtain the target recommendation regions of the images.

[0039] Step 7, input the obtained feature maps and target recommendation regions into the ROI Align (Region of Interesting Align) network simultaneously to obtain the feature maps of the required size.

[0040] In the ROI Align network, asFigure 4 As shown, the bbox (bounding box, i.e., a rectangular box) region is equally divided according to the required output size. It is very likely that the vertices after equal division do not fall on the actual pixel points. Then, fix 4 points (i.e., Figure 4 the small dots on the right) in each block. For each small dot, weight the values of the 4 actual pixel points closest to it (bilinear interpolation) to obtain the value of the small dot. Four new values will be calculated within one block, and the maximum value among these new values is taken as the output value of this block. Finally, a 2x2 output can be obtained.

[0041] Step 8, the feature map of the required size obtained is passed through the head (i.e., the detection head) layer to obtain the target detection region and the target segmentation region.

[0042] In the head layer, a double-branch will be performed, where one branch is used for the operation of target segmentation, and the other branch is used for the operation of target detection.

[0043] Step 9, the target segmentation region passes through two fully connected layers to obtain the segmentation result.

[0044] The segmentation result is multiple pixel points corresponding to the target object in the target image.

[0045] Step 10, the target detection region passes through the fully connected layer to obtain the target box, classification, and score respectively.

[0046] Optionally, in this embodiment, the object detection performed on the target image can be similar to the operation process of the above training model, except that the object detection performed on the target image does not involve the processes of the above steps 1 and 3.

[0047] Step S204, when the first detection result indicates that a target object is detected from the target image, determine the regions corresponding to the first region in the first infrared image and the second infrared image respectively to obtain the second region and the third region, and determine the first heat difference according to the first infrared image and the second region, and determine the second heat difference according to the second infrared image and the third region, where the first region is the region where the target object is located in the target image, the second region and the third region represent the same region as the first region, the first heat difference is the difference between the heat corresponding to the second region in the first infrared image and the heat corresponding to the pixel points adjacent to the second region, the first infrared image is the infrared image taken at the same time as the target image, the second heat difference is the difference between the heat corresponding to the second region in the first infrared image and the heat corresponding to the third region in the second infrared image, and the second infrared image is the infrared image obtained by the infrared camera taking pictures of the airport runway before the shooting time of the first infrared image.

[0048] When it is determined that the first detection result indicates that a target object is detected from the target picture, the area of the target object (i.e., the first area) can be extracted from the target picture, and the target area can be mapped to multiple infrared pictures to calculate the corresponding infrared heat value according to the infrared pictures. The target object here can be the object with the highest score detected from the target picture and higher than a certain threshold.

[0049] The above mapping of the target area to multiple infrared pictures can refer to determining the areas corresponding to the first area in the first infrared picture and the second infrared picture respectively according to the extracted target area, to obtain the second area and the third area. Here, the first infrared picture is the infrared picture taken at the same time as the target picture. The second infrared picture is the infrared picture obtained by the infrared camera when shooting the airport runway before the shooting time of the first infrared picture.

[0050] After determining the second area and the third area, the first heat difference can be determined according to the first infrared picture and the second area. The first area is the area where the target object is located in the target picture. The second area and the third area represent the same area as the first area. The first heat difference is the difference between the heat corresponding to the second area in the first infrared picture and the heat corresponding to the pixel points adjacent to the second area.

[0051] In addition, the second heat difference can also be determined according to the second infrared picture and the third area. The second heat difference is the difference between the heat corresponding to the second area in the first infrared picture and the heat corresponding to the third area in the second infrared picture.

[0052] Optionally, the above second infrared picture can be the infrared picture taken at the same moment as the shooting moment of the target picture in the previous day. For example, if the target picture is taken at 19:00 today, the N - 1 infrared pictures can be taken at 19:00 yesterday. The second infrared picture can also correspond to the second aircraft before the first aircraft corresponding to the shooting moment of the target picture on the same day. That is, the time difference between the shooting time of the first infrared picture and the time when the first aircraft passes by the infrared camera, and the time difference between the shooting time of this second infrared picture and the time when the second aircraft passes by the infrared camera are the same.

[0053] It should be noted that the above first heat difference and second heat difference can be calculated according to the heat values of each pixel point recorded in the pixel infrared heat value library. The pixel infrared heat value library can save the heat value map coordinates in a time - series manner, and obtain pixel points through the infrared camera to construct the pixel infrared heat value library corresponding to the airport runway.

[0054] Step S206: Determine a second detection result of the target object according to the first temperature difference and the second temperature difference, where the second detection result is used to indicate whether the target object is an abnormal object that appears on the airport runway.

[0055] In this embodiment, according to the determined first temperature difference and second temperature difference, the second detection result of the target object can be determined. The second detection result here can refer to the result of re-verifying the information related to the target object in the first detection result, that is, the second detection result is used to indicate whether the target object is an abnormal object that appears on the airport runway.

[0056] Optionally, when the first temperature difference and the second temperature difference are large, or the sum of the first temperature difference and the second temperature difference is large, it can be determined that the target object detected in the first detection result is indeed a foreign object that suddenly appears on the airport runway, or at least a foreign object that appears compared to the previous infrared image capture.

[0057] Through the above steps, object detection is performed on the target image to obtain a first detection result, where the target image is a picture obtained by a visible light imaging device shooting the airport runway; in the case where the first detection result indicates that a target object is detected in the target image, regions corresponding to the first region are respectively determined in the first infrared image and the second infrared image to obtain a second region and a third region, and a first temperature difference is determined according to the first infrared image and the second region, and a second temperature difference is determined according to the second infrared image and the third region, where the first region is the region where the target object is located in the target image, the second region and the third region represent the same region as the first region, the first temperature difference is the difference between the temperature corresponding to the second region in the first infrared image and the temperature corresponding to the pixel points adjacent to the second region, the first infrared image is an infrared image captured at the same time as the target image, the second temperature difference is the difference between the temperature corresponding to the second region in the first infrared image and the temperature corresponding to the third region in the second infrared image, and the second infrared image is an infrared image obtained by an infrared camera shooting the airport runway before the shooting time of the first infrared image; determine a second detection result of the target object according to the first temperature difference and the second temperature difference, where the second detection result is used to indicate whether the target object is an abnormal object that appears on the airport runway, which can solve the problem that the object detection method in the related technology has low detection accuracy for foreign objects on the airport runway, and achieve the technical effect of improving the detection accuracy of foreign objects on the airport runway.

[0058] In an exemplary embodiment, determining the first temperature difference according to the first infrared image and the second region includes:

[0059] S11. Determine the average heat of each pixel in the second area of the first infrared image to obtain the first average heat;

[0060] S12. In the first infrared image, determine the pixels adjacent to each pixel in the second area to obtain M pixels, and determine the average heat of the M pixels in the first infrared image to obtain the second average heat, where M is a positive integer greater than or equal to 2;

[0061] S13. Determine the first heat difference to be equal to the difference between the first average heat and the second average heat.

[0062] When calculating the first heat difference, the heat differences between the average heat of the second area and multiple adjacent pixels in the second area can be calculated first, and then the average value of the obtained multiple heat differences can be used as the first heat difference.

[0063] In this embodiment, the average heat of each pixel in the second area of the first infrared image can be determined through the aforementioned pixel infrared heat value library to obtain the first average heat. Then, the pixels adjacent to each pixel in the second area are determined in the first infrared image to obtain M pixels. M is a positive integer greater than or equal to 2.

[0064] Optionally, the pixels adjacent to each pixel in the second area may be the pixels adjacent to each pixel located at the edge of the second area in the second area.

[0065] Determine the average heat of the M pixels in the first infrared image through the aforementioned pixel infrared heat value library to obtain the second average heat, and determine the first heat difference according to the difference between the first average heat and the second average heat. Here, the difference between the first average heat and the second average heat may be the first average heat minus the second average heat.

[0066] Through this embodiment, by calculating the heat differences between the average heat of the second area and each adjacent pixel, the calculation efficiency of the heat difference between the target area and adjacent positions can be improved.

[0067] In an exemplary embodiment, determining the second heat difference according to the second infrared image and the third area includes:

[0068] S21. Determine the average heat of each pixel in the second area of the first infrared image to obtain the first average heat;

[0069] S22. Determine the average heat of each pixel in the third area of the second infrared image to obtain the third average heat;

[0070] S23. Determine the second heat difference to be equal to the difference between the first average heat and the third average heat.

[0071] When calculating the second heat difference, it is also possible to first calculate the average heat of the area, calculate the difference between the average heats of the second area and the third area, and use it as the first heat difference in the early stage.

[0072] In this embodiment, after determining the average heat of each pixel point in the second area of the first infrared image to obtain the first average heat, the average heat of each pixel point in the third area of the second infrared image can be determined from the aforementioned pixel infrared heat value library to obtain the third average heat.

[0073] Determine the second heat difference according to the difference between the first average heat and the third average heat. Here, the difference between the first average heat and the third average heat can also be the first average heat minus the third average heat.

[0074] Through this embodiment, by calculating the difference between the average heat of the second area and the average heat of the second area to determine the heat difference between the extracted target area and the heat of the same area at other times, the calculation efficiency can be improved, and thus the detection efficiency of the target object can be improved.

[0075] In an exemplary embodiment, determining the second detection result of the target object according to the first heat difference and the second heat difference includes:

[0076] S31. When the first area includes P pixel points, determine the current position coordinates represented by each pixel point in the P pixel points in the first coordinate system according to the target shooting parameters to obtain P current position coordinates, where P is a positive integer greater than or equal to 2, the target image is a picture obtained by a visible light imaging device shooting an airport runway using the target shooting parameters, and the first coordinate system is a coordinate system established with the position of the visible light imaging device as the reference point;

[0077] S32. Determine the reference position coordinates corresponding to the P pixel points in the preset reference position coordinate database to obtain P reference position coordinates, where the reference position coordinate database includes the position coordinates represented by each pixel point in the pixel point set determined according to the target shooting parameters in the first coordinate system, the pixel point set includes each pixel point in the reference image, and the reference image is a picture obtained by a visible light imaging device shooting an airport runway using the target shooting parameters before the target image;

[0078] S33. Determine the difference between the P current position coordinates and the P reference position coordinates to obtain P first differences;

[0079] S34. When all of the P first differences are less than the first preset threshold, multiply the first heat difference by a preset first proportionality coefficient to obtain a first product value, and multiply the second heat difference by a preset second proportionality coefficient to obtain a second product value;

[0080] S35. When the sum of the first product value and the second product value is greater than or equal to the second preset threshold, determine the second detection result as indicating that the target object is an abnormal object that appears on the airport runway.

[0081] After determining that the first detection result indicates that a target object is detected from the target picture, the difference between the actual position coordinates of each pixel point in the first region corresponding to the target object in the first coordinate system and the reference position coordinates corresponding to each pixel point in the preset reference position coordinate database can be calculated first. When the difference is less than the first preset threshold, then the second detection result of the target object is determined according to the first heat difference and the second heat difference. Here, the first coordinate system can be a coordinate system established with the position of the visible light imaging device as a reference point, and it can be a world coordinate system.

[0082] As Figure 5 shown, the camera coordinate system is also a three-dimensional rectangular coordinate system (Xc, Yc, Zc). The origin of the camera coordinate system is the optical center of the lens. The x and y axes are respectively parallel to the two sides of the image plane, and the z axis is the optical axis of the lens, perpendicular to the image plane. The world coordinate system is a three-dimensional rectangular coordinate system (Xw, Yw, Zw). In the world coordinate system, the spatial positions of the camera and the object to be measured can be described. The transformation from the world coordinate system to the camera coordinate system is a rigid body transformation, that is, only the spatial position (translation) and orientation (rotation) of the object are changed, and the shape of the object is not changed.

[0083] The actual position coordinates of the above pixel points can be determined according to the target shooting parameters. The target shooting parameters can be the parameters used by the visible light imaging device when shooting the target picture, that is, the target picture is a picture obtained by the visible light imaging device shooting the airport runway using the target shooting parameters. Correspondingly, the visible light imaging device can be a binocular imaging device.

[0084] In this embodiment, when the first region includes P pixel points, according to the target shooting parameters, the current position coordinates represented by each pixel point among the P pixel points in the first coordinate system can be determined. The determination method can be to determine the imaging points of the same position on the two cameras and the corresponding positions in the two camera coordinate systems through the camera coordinate systems respectively corresponding to the two cameras in the visible light imaging device and the overlapping region between the two cameras, and then combine the known shooting parameters to determine the depth information of each pixel point, thereby realizing the conversion between the camera coordinate system and the world coordinate system, and obtaining the coordinates of the P pixel points in the second region corresponding to the target object in the world coordinate system.

[0085] The above-mentioned preset reference position coordinate database can be obtained by previously shooting the airport runway using the target shooting parameters by the visible light imaging device before the target picture, and determining the position coordinates represented by each pixel point in the first coordinate system in the set of pixel points determined in a manner similar to the above-mentioned determination of the current position coordinates. The set of pixel points includes each pixel point in the reference picture.

[0086] In this embodiment, when the first region includes P pixel points, the reference position coordinates corresponding to the P pixel points can be determined in the preset reference position coordinate database to obtain P reference position coordinates. And calculate the differences between the P current position coordinates and the P reference position coordinates to obtain P first differences.

[0087] When all of the P first differences are less than the first preset threshold, then according to the first heat difference and the second heat difference, the second detection result of the target object is determined.

[0088] In addition, determining the second detection result of the target object according to the first heat difference and the second heat difference can be to multiply the first heat difference by a preset first proportionality coefficient to obtain a first product value, and multiply the second heat difference by a preset second proportionality coefficient to obtain a second product value. Determine the second detection result according to the sum of the first product value and the second product value.

[0089] When the sum of the first product value and the second product value is greater than or equal to the second preset threshold, the second detection result can be determined to indicate that the target object is an abnormal object appearing on the airport runway. Correspondingly, when the sum of the first product value and the second product value is less than the second preset threshold, the second detection result can be determined to indicate that the target object is a false detection.

[0090] For example, taking the second heat difference as the difference value A of the heat values at other times and the first heat difference as the difference value B of the heat values at adjacent positions as an example, as shown in formula (1), calculate whether the sum of the products of the difference value A of the heat values at other times and the difference value B of the heat values at adjacent positions is less than the set threshold.

[0091] αA + βB < threshold_hot (1)

[0092] Wherein, α is the proportionality coefficient of the difference value A of the calorific value at other times, β is the proportionality coefficient of the difference value B of the heat value at adjacent positions, and threshold_hot is the set threshold value.

[0093] Optionally, when it is determined that the target object is a false detection, the detection result of the target image can be cleared, and the target image can be put into the training pool to retrain the model for object detection described above using the target image.

[0094] Optionally, after determining that the second detection result is used to represent that the target object is an abnormal object appearing on the airport runway, the first detection result of the target object can be displayed on the large screen of the airport central control system, and the detection result can be further verified according to the instructions of the maintenance personnel.

[0095] In an exemplary embodiment, after determining that the second detection result is used to represent that the target object is an abnormal object appearing on the airport runway, the above method further includes:

[0096] S41, when the first instruction received indicates that the second detection result is incorrect, adjusting the first detection parameter, where the first detection parameter includes at least one of the following: the first proportionality coefficient, the second proportionality coefficient, the reference position coordinates corresponding to at least some pixel points in the reference position coordinate database, and the type of the target object.

[0097] After displaying the detection result of the detected target object, it can be determined whether the detection result is incorrect according to the first instruction received. When the first instruction received indicates that the second detection result is incorrect, it can be determined that there is an error in the detection of the target object.

[0098] In this embodiment, according to the indication of the first instruction, the first detection parameter can be adjusted correspondingly. Here, the first detection parameter can include at least one of the following: the first proportionality coefficient, the second proportionality coefficient, the reference position coordinates corresponding to at least some pixel points in the reference position coordinate database, and the type of the target object.

[0099] Optionally, when the first instruction received indicates that the second detection result is incorrect, adjusting the first detection parameter includes:

[0100] When the first instruction indicates that the target object does not exist, reducing the first proportionality coefficient and the second proportionality coefficient according to a first preset ratio;

[0101] When the first instruction indicates the existence of the target object but the position of the target object is misidentified, the candidate position coordinates of P pixel points in the first coordinate system are determined by the positioning device, the P reference position coordinates in the reference position coordinate database are updated to the P candidate position coordinates, and the position of the target object in the first detection result is modified according to the P candidate position coordinates. The positioning device is set on the airport runway or at a position adjacent to the airport runway, and the position coordinates of the positioning device are known position coordinates. The positioning device is used to locate the points on the detected target object;

[0102] When the first instruction indicates the existence of the target object, the position of the target object is correctly identified, but the type of the target object is misidentified, the type of the target object is updated to the detection object type corresponding to the first instruction, and the first proportional coefficient and the second proportional coefficient are increased according to the second preset ratio.

[0103] It should be noted that when the first instruction indicates the non-existence of the target object, the first proportional coefficient and the second proportional coefficient can be first decreased according to the first preset ratio, and then, according to the influence of the adjusted values, different reduction ratios and final values are further determined. The above first preset ratio and second preset ratio can be different ratios or the same ratio, and this embodiment does not limit this.

[0104] The misidentification of the position of the above target object can be determined according to the pixel points of the recognized target object that do not affect the original position. Here, the original position can refer to the position of the pixel points corresponding to the target object in the pre-set reference position coordinate database. Since a foreign object is an object that suddenly appears on the airport runway, if the target object is a foreign object, when constructing the reference position coordinate database, the pixel points at the position where the target object is located are the pixel points captured without the foreign object. When the target object appears at this position, it will inevitably affect the pixel points at the same position in the captured image. When the position of the recognized target object affects the pixel points at this position, it can be determined that the position of the recognized target object is correct; otherwise, it is misidentified.

[0105] It should be noted that the above positioning device can be a light-emitting diode and a photosensitive triode. The light-emitting diode emits light, the light contacts the object and is reflected to the photosensitive triode. According to the reception time of the photosensitive triode, the relative position between the object and the light-emitting diode and the photosensitive triode can be determined. Since the positions of the light-emitting diode and the photosensitive triode are known, the position of the object in the first coordinate system can be determined according to the determined relative position.

[0106] Optionally, when constructing the reference position coordinate database, it can also be jointly created based on the visible light imaging devices, light-emitting diodes, and photosensitive triodes located on both sides of the airport runway. That is, first establish an initial reference position coordinate database through the visible light imaging device, then randomly select multiple pixel points, and update the initial reference position coordinate database based on the positions of these multiple pixel points measured by the light-emitting diodes and photosensitive triodes to obtain the preset reference position coordinate database.

[0107] In the case of incorrect position recognition of the target object, the candidate position coordinates of P pixel points of the target object in the first coordinate system can be determined through the light-emitting diodes and photosensitive triodes, and the relevant parameters in the reference position coordinate database and the position of the target object in the first detection result can be updated based on the determined candidate position coordinates.

[0108] Optionally, updating the relevant parameters in the reference position coordinate database based on the determined candidate position coordinates may refer to updating the P reference position coordinates in the reference position coordinate database, or updating the coordinates of all pixel points in the reference position coordinate database according to the relationship between the P reference position coordinates and the P candidate reference position coordinates.

[0109] Optionally, after completing the above modifications according to the first instruction, the relevant information of the target picture and the modified target picture can be put into the training pool to retrain the model for object detection described above. And when the airport maintenance personnel determine that the target object exists and the position and type are correctly recognized, the maintenance personnel can directly perform anomaly handling.

[0110] Through this embodiment, adjusting the relevant detection parameters according to the indication information can improve the accuracy of subsequent object detection. At the same time, when the first instruction indicates that the target object does not exist, automatically adjusting the proportional coefficient can avoid errors caused by using the same value for different categories.

[0111] In an exemplary embodiment, after performing object detection on the target picture to obtain the first detection result, the above method further includes:

[0112] S51, when the first detection result indicates that a target object is detected from the target picture and there are P pixel points in the first area, determine the current position coordinates represented by each of the P pixel points in the first coordinate system according to the target shooting parameters to obtain P current position coordinates, where P is a positive integer greater than or equal to 2, the target picture is a picture obtained by the visible light imaging device shooting the airport runway using the target shooting parameters, and the first coordinate system is a coordinate system established with the position of the visible light imaging device as the reference point;

[0113] S52. Determine the reference position coordinates corresponding to the P pixel points in the preset reference position coordinate database to obtain P reference position coordinates. Among them, the reference position coordinate database includes the position coordinates represented by each pixel point in the pixel point set determined according to the target shooting parameters in the first coordinate system. The pixel point set includes each pixel point in the reference picture, and the reference picture is a picture obtained by the visible light imaging device using the target shooting parameters to shoot the airport runway before the target picture;

[0114] S53. Determine the differences between the P current position coordinates and the P reference position coordinates to obtain P first differences;

[0115] S54. When not all of the P first differences are less than the first preset threshold, determine the candidate position coordinates of the P pixel points in the first coordinate system through the positioning device to obtain P candidate position coordinates. Among them, the positioning device is set on the airport runway or at a position adjacent to the airport runway, and the position coordinates of the positioning device are known position coordinates. The positioning device is used to locate the points on the detected target object;

[0116] S55. Determine the differences between the P candidate position coordinates and the P reference position coordinates to obtain P second differences;

[0117] S56. Adjust the position of the target object in the first detection result according to the P second differences to obtain the first adjusted detection result.

[0118] In this embodiment, when not all of the P first differences are less than the first preset threshold, the calculation of the first heat difference and the second heat difference may not be performed. Directly determine the candidate position coordinates of the P pixel points in the first coordinate system through the positioning device to obtain P candidate position coordinates.

[0119] By calculating the differences between the P candidate position coordinates determined by the positioning device and the P reference position coordinates determined by the reference position coordinate data, P second differences are obtained. Then, according to the P second differences, the relevant parameters participating in determining the position of the target object are adjusted to facilitate improving the accuracy of subsequent object detection. At the same time, the position of the target object in the first detection result determined by this object detection can be adjusted to obtain the first adjusted detection result.

[0120] Through this embodiment, based on the differences between the P candidate position coordinates determined by the positioning device and the P reference position coordinates determined by the reference position coordinate data, the position of the currently detected object is adjusted, which can improve the accuracy of object detection.

[0121] In an exemplary embodiment, adjusting the position of the target object in the first detection result according to the P second differences to obtain the first adjusted detection result includes:

[0122] S61. When all of the P second differences are less than a third preset threshold, adjust the first coordinate system to a second coordinate system such that all of the P third differences are less than a first preset threshold, and modify the position of the target object in the first detection result according to the second coordinate system, where the P third differences are the differences between the P updated position coordinates and the P reference position coordinates, and the P updated position coordinates are the candidate position coordinates of the P pixel points determined by the positioning device in the second coordinate system.

[0123] When it is determined that all of the P second differences are less than the third preset threshold, the foregoing first coordinate system can be recalibrated. The calibration method can be to adjust the first coordinate system to a second coordinate system so that all of the P third differences are less than the first preset threshold, that is, to make the position coordinate information of the same target determined according to the target shooting parameters of the visible light imaging device match the position coordinate information of the pixel point corresponding to the target in the reference pixel coordinate system.

[0124] In addition, after obtaining the second coordinate system, the position of the target object in the first detection result can also be modified according to the second coordinate system.

[0125] Optionally, adjusting the position of the target object in the first detection result according to the P second differences to obtain a first adjusted detection result further includes:

[0126] When all of the P second differences are not less than the third preset threshold, determine the differences between the P candidate position coordinates and the P current position coordinates to obtain P fourth differences;

[0127] When all of the P fourth differences are not less than a fourth preset threshold, update the P reference position coordinates in the reference position coordinate database to the P candidate position coordinates, and modify the position of the target object in the first detection result according to the P candidate position coordinates.

[0128] The calculation processes of the foregoing P candidate position coordinates and the P current position coordinates can be the same as the descriptions in the foregoing embodiments, and are not elaborated herein. When all of the P second differences are not less than the third preset threshold, the differences between the P candidate position coordinates and the P current position coordinates can be calculated to obtain P fourth differences.

[0129] Optionally, when all of the P fourth differences are less than the fourth preset threshold, the detection result of the currently determined target object can be directly displayed on the large screen of the airport central control system for maintenance personnel to judge the detection result.

[0130] When the P fourth differences are not all less than the fourth preset threshold, the P candidate position coordinates can be used to update the reference position coordinate database. The update here can refer to updating the P reference position coordinates in the reference position coordinate database according to the determined candidate position coordinates, or can refer to updating the coordinates of all pixel points in the reference position coordinate database according to the relationship between the P reference position coordinates and the P candidate reference position coordinates.

[0131] Optionally, after adjusting the position of the target object in the above first detection result, the adjusted position and other detection results of the target object can be displayed on the large screen of the airport central control system for maintenance personnel to judge the detection results.

[0132] Through this embodiment, according to the pixel coordinate position library and the real-time distance detected by the light-emitting diode and the photosensitive triode, the pixel coordinate position library can be automatically corrected, which can avoid the error caused by the long-term inability to correct the binocular camera ranging, thereby improving the detection accuracy of object detection.

[0133] In an exemplary embodiment, after adjusting the position of the target object in the first detection result according to the P second differences to obtain the first adjusted detection result, the method further includes:

[0134] S71, when the received second instruction indicates that the target object does not exist, modifying the first adjusted detection result to indicate that there is no target object;

[0135] S72, when the second instruction indicates that the target object exists but the object type of the target object is incorrect, updating the object type of the target object in the first adjusted detection result to the detection object type corresponding to the second instruction.

[0136] After completing the adjustment of the position of the target object in the first detection result according to the P second differences, the first adjusted detection result can be verified according to the received second instruction. The second instruction here can be an instruction similar to the first instruction. It can be an instruction generated according to the feedback of the airport maintenance personnel on the detection results displayed on the large screen of the airport central control system.

[0137] In this embodiment, when the received second instruction indicates that the target object does not exist, the first adjusted detection result can be modified to indicate that there is no target object. The detection result corresponding to the target picture can also be deleted, and the target picture can be added to the training pool.

[0138] When the second instruction indicates the existence of the target object but the object type of the target object is incorrect, the object type of the target object in the first adjustment detection result can be updated to the detection object type corresponding to the second instruction. At the same time, the corrected detection result corresponding to the target picture and the target image can be put into the training pool to retrain the model for object detection mentioned above.

[0139] Through this embodiment, by adjusting relevant detection parameters according to the indication information, the accuracy of subsequent object detection can be improved. At the same time, by automatically adding pictures and labels to the training pool, automatic training of the model and parameter update can be carried out, which can improve the accuracy of the model for object detection.

[0140] The object detection method in the embodiment of the present application will be explained below with reference to optional examples.

[0141] In this optional example, an object detection method is provided. By adopting optimizations such as data augmentation and step-by-step training during network training, and adding means such as region extraction, pixel value corresponding coordinate comparison, infrared camera imaging analysis, and physical ranging during network inference, the detection accuracy of the model can be improved.

[0142] The object detection method in this optional example can be as Figure 6 and Figure 7 shown, and may include the following steps:

[0143] Step 1, access the video stream and extract pictures from the video streams of the infrared camera and the visible light camera.

[0144] Step 2, obtain the current time of the system.

[0145] Step 3, the RGB enters the target detection and target segmentation process of the picture.

[0146] Step 4, determine whether a new target enters the field of view. If so, execute the next step; if not, return to execute Step 1.

[0147] Step 5, judge whether the score value is greater than the set threshold. If so, execute the next step; if not, return to execute Step 1.

[0148] Step 6, extract the target area, and at the same time map the target area to the infrared image picture.

[0149] Calculate the difference value A between the calorific value of this area and the calorific values at other times according to the pixel infrared calorific value library. Other times refer to the difference in time from the time when the plane passed this time and the same difference time from the time when the plane passed last time or the same time yesterday. Calculate the difference B between the calorific value of this area and the heat value of the adjacent position according to the pixel infrared calorific value library.

[0150] Step 7: Calculate the pixel coordinates of each position in the target area according to the pixel coordinate position library.

[0151] Step 8: Calculate whether the difference between the actual coordinates of the pixel positions obtained from the two visible lights and the coordinates of the positions in the coordinate library is less than the set threshold. If it is less than the set threshold, proceed to the next step; otherwise, execute Step 21.

[0152] Step 9: Determine the product of the difference value A and the proportionality coefficient α, and the product of the difference value B and the proportionality coefficient β. Judge whether the sum of the two products is less than the set threshold. If so, proceed to the next step; otherwise, execute Step 11.

[0153] Step 10: Determine it as a false detection, clear the picture label, and put the data into the training pool.

[0154] Step 11: Display the result on the large screen of the airport central control system.

[0155] Step 12: The maintenance personnel determine whether a target appears. If not, proceed to the next step; if so, execute Step 14.

[0156] Step 13: Adjust the values of α and β.

[0157] Overall, simultaneously reduce the values of α and β. According to the influence of adjusting the values of α and β, further determine the different reduction ratios and final values of α and β.

[0158] Step 14: The maintenance personnel further determine whether the target affects the pixels at the original position. If not, proceed to the next step; if so, execute Step 16.

[0159] Step 15: Update the pixel coordinate position library according to the pixel coordinate positions detected by the light-emitting diode and the photosensitive triode.

[0160] Step 16: The airport maintenance personnel determine whether the target type is correct. If correct, proceed to the next step; if not, execute Step 18.

[0161] Step 17: The airport maintenance personnel perform exception handling. The picture is detected correctly and does not need to be put into the training pool again.

[0162] Step 18: Correct the type label of the detected target.

[0163] Step 19: Adjust the values of α and β for this category. Simultaneously increase the values of α and β according to a certain ratio. According to the influence of adjusting the values of α and β, further determine the different amplification ratios and final values of α and β.

[0164] Step 20: Correct the label and put the picture and label into the training pool.

[0165] Step 21: Determine whether the difference between the position coordinates and the pixel coordinate position library is less than the set threshold through the light-emitting diode and the photosensitive triode. If so, proceed to the next step; otherwise, execute Step 30.

[0166] Step 22: Re-calibrate the coordinate systems of the two cameras so that the position coordinate information and the pixel coordinate position library information detected by the two cameras match.

[0167] Step 23: Display the result on the large screen of the airport central control system.

[0168] Step 24: Determine whether a target appears by the maintenance personnel. If not, proceed to the next step; if so, execute Step 26.

[0169] Step 25: Delete the picture label and add the picture to the training pool.

[0170] Step 26: Determine whether the category is correct by the maintenance personnel. If so, proceed to the next step; if not, execute Step 28.

[0171] Step 27: The airport maintenance personnel perform exception handling. The picture is detected correctly and does not need to be put into the training pool again.

[0172] Step 28: Correct the category label of the detected target.

[0173] Step 29: Correct the label and put the picture and the label into the training pool.

[0174] Step 30: The detection distance by the light-emitting diode and the photosensitive triode is less than the set threshold compared with the detection distance by the camera. If so, return to Step 23; if not, proceed to the next step.

[0175] Step 31: Update the pixel coordinate position library according to the detection distance by the light-emitting diode and the photosensitive triode, and at the same time return to execute Step 23.

[0176] Step 32: When the newly added data in the training pool meets the set threshold, re-train the model.

[0177] Step 33: Update the original model parameters with the trained model parameters.

[0178] Step 34: The program ends.

[0179] Through this optional example, data augmentation is performed during the training process, which can increase the generalization ability of the model. At the same time, using light-emitting diodes and photosensitive triodes as the reference for calibrating the actual coordinates of pixel points can improve the measurement offset caused by the vibration of the camera at the airport. In addition, by combining the ranging of light-emitting diodes and photosensitive triodes with binocular ranging and using an infrared camera as an auxiliary, automatic correction of the detection results and detection parameters can be achieved, thereby improving the accuracy of object detection.

[0180] It should be noted that for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that this application is not limited by the described action sequence, because according to this application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0181] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the embodiments of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disc), and includes several instructions for causing a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0182] According to another aspect of the embodiments of this application, an object detection device is further provided. This device is used to implement the object detection method provided in the above embodiments, and those that have been described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that can implement a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware is also possible and contemplated.

[0183] Figure 8 is a structural block diagram of an object detection device according to an embodiment of this application, as Figure 8 shown, this device includes:

[0184] A detection unit 802, configured to perform object detection on a target picture to obtain a first detection result, where the target picture is a picture obtained by a visible light imaging device shooting an airport runway;

[0185] The first determination unit 804, connected to the detection unit 802, is configured to, when the first detection result indicates that a target object is detected from the target picture, respectively determine the regions corresponding to the first region in the first infrared picture and the second infrared picture to obtain a second region and a third region, and determine a first heat difference according to the first infrared picture and the second region, and determine a second heat difference according to the second infrared picture and the third region, where the first region is the region where the target object is located in the target picture, the second region and the third region represent the same region as the first region, the first heat difference is the difference between the heat corresponding to the second region in the first infrared picture and the heat corresponding to the pixel points adjacent to the second region, the first infrared picture is an infrared picture taken at the same time as the target picture, the second heat difference is the difference between the heat corresponding to the second region in the first infrared picture and the heat corresponding to the third region in the second infrared picture, and the second infrared picture is an infrared picture obtained by the infrared camera taking pictures of the airport runway before the shooting time of the first infrared picture;

[0186] The second determination unit 806, connected to the first determination unit 804, is configured to determine a second detection result of the target object according to the first heat difference and the second heat difference, where the second detection result is used to indicate whether the target object is an abnormal object appearing on the airport runway.

[0187] Through the embodiments of the present application, object detection is performed on a target picture to obtain a first detection result, where the target picture is a picture obtained by a visible light imaging device taking pictures of the airport runway; when the first detection result indicates that a target object is detected from the target picture, the regions corresponding to the first region are respectively determined in the first infrared picture and the second infrared picture to obtain a second region and a third region, and a first heat difference is determined according to the first infrared picture and the second region, and a second heat difference is determined according to the second infrared picture and the third region, where the first region is the region where the target object is located in the target picture, the second region and the third region represent the same region as the first region, the first heat difference is the difference between the heat corresponding to the second region in the first infrared picture and the heat corresponding to the pixel points adjacent to the second region, the first infrared picture is an infrared picture taken at the same time as the target picture, the second heat difference is the difference between the heat corresponding to the second region in the first infrared picture and the heat corresponding to the third region in the second infrared picture, and the second infrared picture is an infrared picture obtained by the infrared camera taking pictures of the airport runway before the shooting time of the first infrared picture; a second detection result of the target object is determined according to the first heat difference and the second heat difference, where the second detection result is used to indicate whether the target object is an abnormal object appearing on the airport runway, which can solve the problem that the object detection method in the related art has low detection accuracy for foreign objects on the airport runway, and achieve the technical effect of improving the detection accuracy of foreign objects on the airport runway.

[0188] Optionally, the first determination unit includes:

[0189] A first determination module, configured to determine an average heat of each pixel point in a second region in a first infrared image, so as to obtain a first average heat;

[0190] A second determination module, configured to determine pixel points adjacent to each pixel point in the second region in the first infrared image, so as to obtain M pixel points, and determine an average heat of the M pixel points in the first infrared image, so as to obtain a second average heat, where M is a positive integer greater than or equal to 2;

[0191] A third determination module, configured to determine the first heat difference to be equal to a difference between the first average heat and the second average heat.

[0192] Optionally, the first determination unit includes:

[0193] A fourth determination module, configured to determine an average heat of each pixel point in a second region in the first infrared image, so as to obtain a first average heat;

[0194] A fifth determination module, configured to determine an average heat of each pixel point in a third region in a second infrared image, so as to obtain a third average heat;

[0195] A sixth determination module, configured to determine the second heat difference to be equal to a difference between the first average heat and the third average heat.

[0196] Optionally, the second determination unit includes:

[0197] A seventh determination module, configured to, when the first region includes P pixel points, determine current position coordinates represented by each of the P pixel points in a first coordinate system according to target shooting parameters, so as to obtain P current position coordinates, where P is a positive integer greater than or equal to 2, the target image is an image obtained by a visible light imaging device shooting an airport runway using the target shooting parameters, and the first coordinate system is a coordinate system established with the position of the visible light imaging device as a reference point;

[0198] An eighth determination module, configured to determine reference position coordinates corresponding to the P pixel points in a preset reference position coordinate database, so as to obtain P reference position coordinates, where the reference position coordinate database includes position coordinates represented by each pixel point in a pixel point set determined according to the target shooting parameters in the first coordinate system, the pixel point set includes each pixel point in a reference image, and the reference image is an image obtained by the visible light imaging device shooting the airport runway using the target shooting parameters before the target image;

[0199] A ninth determination module, configured to determine the differences between P current position coordinates and P reference position coordinates to obtain P first differences;

[0200] A multiplication module, configured to multiply the first heat difference by a preset first proportionality coefficient to obtain a first product value, and multiply the second heat difference by a preset second proportionality coefficient to obtain a second product value when all of the P first differences are less than a first preset threshold;

[0201] A tenth determination module, configured to determine the second detection result as representing that the target object is an abnormal object appearing on the airport runway when the sum of the first product value and the second product value is greater than or equal to a second preset threshold.

[0202] Optionally, the above device further includes:

[0203] A first adjustment unit, configured to adjust first detection parameters when, after determining the second detection result as representing that the target object is an abnormal object appearing on the airport runway, a first instruction received indicates that the second detection result is incorrect, where the first detection parameters include at least one of the following: the first proportionality coefficient, the second proportionality coefficient, the reference position coordinates corresponding to at least some pixel points in the reference position coordinate database, and the type of the target object.

[0204] Optionally, the first adjustment unit includes:

[0205] A first execution module, configured to reduce the first proportionality coefficient and the second proportionality coefficient by a first preset ratio when the first instruction indicates that the target object does not exist;

[0206] A second execution module, configured to determine candidate position coordinates of P pixel points in a first coordinate system through a positioning device when the first instruction indicates that the target object exists but the position of the target object is incorrectly recognized, update the P reference position coordinates in the reference position coordinate database to the P candidate position coordinates, and modify the position of the target object in the first detection result according to the P candidate position coordinates, where the positioning device is disposed on the airport runway or at a position adjacent to the airport runway, the position coordinates of the positioning device are known position coordinates, and the positioning device is used to position points on the detected target object;

[0207] An update module, configured to update the type of the target object to the detection object type corresponding to the first instruction and increase the first proportionality coefficient and the second proportionality coefficient by a second preset ratio when the first instruction indicates that the target object exists, the position of the target object is correctly recognized, but the type of the target object is incorrectly recognized.

[0208] Optionally, the above device further includes:

[0209] A third determination unit, configured to, after performing object detection on a target picture to obtain a first detection result, when the first detection result indicates that a target object is detected from the target picture and a first area includes P pixel points, determine, according to target shooting parameters, the current position coordinates represented by each of the P pixel points in a first coordinate system, so as to obtain P current position coordinates, where P is a positive integer greater than or equal to 2, the target picture is a picture obtained by a visible light imaging device shooting an airport runway using the target shooting parameters, and the first coordinate system is a coordinate system established with the position of the visible light imaging device as a reference point;

[0210] A fourth determination unit, configured to determine, in a preset reference position coordinate database, the reference position coordinates corresponding to the P pixel points, so as to obtain P reference position coordinates, where the reference position coordinate database includes the position coordinates represented by each pixel point in a pixel point set determined according to the target shooting parameters in the first coordinate system, the pixel point set includes each pixel point in a reference picture, and the reference picture is a picture obtained by the visible light imaging device shooting the airport runway using the target shooting parameters before the target picture;

[0211] A fifth determination unit, configured to determine the differences between the P current position coordinates and the P reference position coordinates, so as to obtain P first differences;

[0212] A sixth determination unit, configured to, when not all of the P first differences are less than a first preset threshold, determine, by using a positioning device, the candidate position coordinates of the P pixel points in the first coordinate system, so as to obtain P candidate position coordinates, where the positioning device is arranged on the airport runway or at a position adjacent to the airport runway, the position coordinates of the positioning device are known position coordinates, and the positioning device is used to position points on the detected target object;

[0213] A seventh determination unit, configured to determine the differences between the P candidate position coordinates and the P reference position coordinates, so as to obtain P second differences;

[0214] A second adjustment unit, configured to adjust the position of the target object in the first detection result according to the P second differences, so as to obtain a first adjusted detection result.

[0215] Optionally, the second adjustment unit includes:

[0216] A third execution module, configured to adjust the first coordinate system to a second coordinate system when all of the P second differences are less than a third preset threshold, so that all of the P third differences are less than a first preset threshold, and modify the position of the target object in the first detection result according to the second coordinate system, where the P third differences are the differences between the P updated position coordinates and the P reference position coordinates, and the P updated position coordinates are the candidate position coordinates of the P pixel points determined by the positioning device in the second coordinate system.

[0217] Optionally, the second adjustment unit includes:

[0218] An eleventh determination module, configured to determine the differences between the P candidate position coordinates and the P current position coordinates to obtain P fourth differences when not all of the P second differences are less than the third preset threshold;

[0219] A fourth execution module, configured to update the P reference position coordinates in the reference position coordinate database to the P candidate position coordinates and modify the position of the target object in the first detection result according to the P candidate position coordinates when not all of the P fourth differences are less than a fourth preset threshold.

[0220] Optionally, the above device further includes:

[0221] A modification unit, configured to modify the first adjusted detection result to that there is no target object when, after adjusting the position of the target object in the first detection result according to the P second differences to obtain a first adjusted detection result, the received second instruction indicates that the target object does not exist;

[0222] An update unit, configured to update the object type of the target object in the first adjusted detection result to the detection object type corresponding to the second instruction when the second instruction indicates that the target object exists but the object type of the target object is incorrect.

[0223] It should be noted that the above-mentioned modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: all of the above modules are located in the same processor; or, the above-mentioned modules are respectively located in different processors in any combination form.

[0224] According to another aspect of the embodiments of the present application, there is also provided a computer-readable storage medium, in which a computer program is stored, where the computer program is configured to execute the steps in any one of the above method embodiments when running.

[0225] In an exemplary embodiment, the above computer-readable storage medium may include, but is not limited to: various media such as USB flash drives, read-only memory (ROM for short), random access memory (RAM for short), mobile hard disks, magnetic disks, or optical discs that can store computer programs.

[0226] According to another aspect of the embodiments of the present application, there is also provided an electronic device, including a memory and a processor. A computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0227] In an exemplary embodiment, the above electronic device may further include a transmission device and input / output devices. Among them, the transmission device is connected to the above processor, and the input / output devices are connected to the above processor.

[0228] Specific examples in this embodiment may refer to the examples described in the above embodiments and exemplary embodiments, and will not be elaborated herein.

[0229] Obviously, those skilled in the art should understand that the above modules or steps of the embodiments of the present application can be implemented by a general computing device. They can be concentrated on a single computing device or distributed on a network composed of multiple computing devices. They can be implemented by program codes executable by the computing device, so that they can be stored in a storage device and executed by the computing device. And in some cases, the steps shown or described can be executed in a different order from here, or they can be separately made into individual integrated circuit modules, or multiple modules or steps among them can be made into a single integrated circuit module to implement. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0230] The above are only the preferred embodiments of the present application and are not used to limit the embodiments of the present application. For those skilled in the art, the embodiments of the present application can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the principle of the embodiments of the present application shall be included in the protection scope of the embodiments of the present application.

Claims

1. An object detection method, characterized in that, it includes: Performing object detection on a target picture to obtain a first detection result, where the target picture is a picture obtained by a visible light imaging device photographing an airport runway; When the first detection result indicates that a target object is detected from the target picture, determining regions corresponding to a first region in a first infrared picture and a second infrared picture respectively to obtain a second region and a third region, and determining a first heat difference according to the first infrared picture and the second region, and determining a second heat difference according to the second infrared picture and the third region, where the first region is the region where the target object is located in the target picture, the second region and the third region represent the same region as the first region, the first heat difference is the difference between the heat corresponding to the second region in the first infrared picture and the heat corresponding to the pixel points adjacent to the second region, the first infrared picture is an infrared picture taken at the same time as the target picture, the second heat difference is the difference between the heat corresponding to the second region in the first infrared picture and the heat corresponding to the third region in the second infrared picture, the second infrared picture is an infrared picture obtained by an infrared camera photographing the airport runway before the shooting time of the first infrared picture, the first heat difference and the second heat difference are calculated according to the heat values of each pixel point recorded in the pixel infrared heat value library, and the pixel infrared heat value library stores the coordinates of the heat value map in a time series manner, and obtains pixel points through the infrared camera to construct an infrared heat value library corresponding to the airport runway; Determining a second detection result of the target object according to the first heat difference and the second heat difference, where the second detection result is used to indicate whether the target object is an abnormal object appearing on the airport runway.

2. The method according to claim 1, characterized in that, the determining the first heat difference according to the first infrared picture and the second region includes: Determining the average value of the heat of each pixel point in the second region in the first infrared picture to obtain a first average heat; Determining the pixel points adjacent to each pixel point in the second region in the first infrared picture to obtain M pixel points, and determining the average value of the heat of the M pixel points in the first infrared picture to obtain a second average heat, where M is a positive integer greater than or equal to 2; Determining the first heat difference to be equal to the difference between the first average heat and the second average heat.

3. The method according to claim 1, characterized in that, the determining the second heat difference according to the second infrared picture and the third region includes: Determining the average value of the heat of each pixel point in the second region in the first infrared picture to obtain a first average heat; Determining the average value of the heat of each pixel point in the third region in the second infrared picture to obtain a third average heat; Determine the second heat difference to be equal to the difference between the first average heat and the third average heat.

4. The method according to claim 1, wherein, determining the second detection result of the target object according to the first heat difference and the second heat difference includes: when the first region includes P pixel points, determining the current position coordinates represented by each of the P pixel points in the first coordinate system according to the target shooting parameters, obtaining P current position coordinates, where P is a positive integer greater than or equal to 2, the target picture is a picture obtained by the visible light imaging device shooting the airport runway using the target shooting parameters, and the first coordinate system is a coordinate system established with the position of the visible light imaging device as the reference point; determining the reference position coordinates corresponding to the P pixel points in the preset reference position coordinate database, obtaining P reference position coordinates, where the reference position coordinate database includes the position coordinates represented by each pixel point in the pixel point set determined according to the target shooting parameters in the first coordinate system, the pixel point set includes each pixel point in the reference picture, and the reference picture is a picture obtained by the visible light imaging device shooting the airport runway using the target shooting parameters before the target picture; determining the difference between the P current position coordinates and the P reference position coordinates, obtaining P first differences; when all of the P first differences are less than the first preset threshold, multiplying the first heat difference by the preset first proportionality coefficient to obtain a first product value, and multiplying the second heat difference by the preset second proportionality coefficient to obtain a second product value; when the sum of the first product value and the second product value is greater than or equal to the second preset threshold, determining the second detection result as representing that the target object is an abnormal object appearing on the airport runway.

5. The method according to claim 4, wherein, after determining the second detection result as representing that the target object is an abnormal object appearing on the airport runway, the method further includes: when the received first instruction indicates that the second detection result is incorrect, adjusting the first detection parameters, where the first detection parameters include at least one of the following: the first proportionality coefficient, the second proportionality coefficient, the reference position coordinates corresponding to at least some pixel points in the reference position coordinate database, the type of the target object.

6. The method according to claim 5, wherein, adjusting the first detection parameters when the received first instruction indicates that the second detection result is incorrect includes: when the first instruction indicates that the target object does not exist, reducing the first proportionality coefficient and the second proportionality coefficient according to a first preset ratio; When the first instruction indicates the existence of the target object but the position of the target object is incorrectly recognized, the candidate position coordinates of the P pixel points in the first coordinate system are determined by a positioning device, the P reference position coordinates in the reference position coordinate database are updated to the P candidate position coordinates, and the position of the target object in the first detection result is modified according to the P candidate position coordinates, where the positioning device is arranged on the airport runway or at a position adjacent to the airport runway, the position coordinates of the positioning device are known position coordinates, and the positioning device is used to locate the points on the detected target object; When the first instruction indicates the existence of the target object, the position of the target object is correctly recognized, but the type of the target object is incorrectly recognized, the type of the target object is updated to the detection object type corresponding to the first instruction, and the first proportionality coefficient and the second proportionality coefficient are increased according to a second preset ratio.

7. The method according to claim 1, wherein, after performing object detection on a target picture to obtain a first detection result, the method further includes: when the first detection result indicates that a target object is detected from the target picture and there are P pixel points in the first region, the current position coordinates represented by each of the P pixel points in the first coordinate system are determined according to target shooting parameters, obtaining P current position coordinates, where P is a positive integer greater than or equal to 2, the target picture is a picture obtained by the visible light imaging device shooting the airport runway using the target shooting parameters, and the first coordinate system is a coordinate system established with the position of the visible light imaging device as a reference point; the reference position coordinates corresponding to the P pixel points are determined in a preset reference position coordinate database, obtaining P reference position coordinates, where the reference position coordinate database includes the position coordinates represented by each pixel point in a pixel point set determined according to the target shooting parameters, the pixel point set includes each pixel point in a reference picture, and the reference picture is a picture obtained by the visible light imaging device shooting the airport runway using the target shooting parameters before the target picture; the differences between the P current position coordinates and the P reference position coordinates are determined, obtaining P first differences; when the P first differences are not all less than a first preset threshold, the candidate position coordinates of the P pixel points in the first coordinate system are determined by a positioning device, obtaining P candidate position coordinates, where the positioning device is arranged on the airport runway or at a position adjacent to the airport runway, the position coordinates of the positioning device are known position coordinates, and the positioning device is used to locate the points on the detected target object; the differences between the P candidate position coordinates and the P reference position coordinates are determined, obtaining P second differences; Adjust the position of the target object in the first detection result according to the P second differences to obtain a first adjusted detection result.

8. The method according to claim 7, wherein, the adjusting the position of the target object in the first detection result according to the P second differences to obtain a first adjusted detection result includes: When all of the P second differences are less than a third preset threshold, adjust the first coordinate system to a second coordinate system such that all of the P third differences are less than the first preset threshold, and modify the position of the target object in the first detection result according to the second coordinate system, where the P third differences are the differences between the P updated position coordinates and the P reference position coordinates, and the P updated position coordinates are the candidate position coordinates of the P pixel points determined by the positioning device in the second coordinate system.

9. The method according to claim 7, wherein, the adjusting the position of the target object in the first detection result according to the P second differences to obtain a first adjusted detection result includes: When the P second differences are not all less than the third preset threshold, determine the differences between the P candidate position coordinates and the P current position coordinates to obtain P fourth differences; When the P fourth differences are not all less than a fourth preset threshold, update the P reference position coordinates in the reference position coordinate database to the P candidate position coordinates, and modify the position of the target object in the first detection result according to the P candidate position coordinates.

10. The method according to claim 8, wherein, after the adjusting the position of the target object in the first detection result according to the P second differences to obtain a first adjusted detection result, the method further includes: When the received second instruction indicates that the target object does not exist, modify the first adjusted detection result to indicate that the target object does not exist; When the second instruction indicates that the target object exists but the object type of the target object is incorrect, update the object type of the target object in the first adjusted detection result to the detection object type corresponding to the second instruction.

11. An object detection device, wherein, comprises: a detection unit configured to perform object detection on a target picture to obtain a first detection result, where the target picture is a picture obtained by a visible light imaging device photographing an airport runway; The first determination unit is configured to, when the first detection result indicates that a target object is detected from the target picture, respectively determine the regions corresponding to the first region in the first infrared picture and the second infrared picture to obtain a second region and a third region, and determine a first heat difference according to the first infrared picture and the second region, and determine a second heat difference according to the second infrared picture and the third region, where the first region is the region where the target object is located in the target picture, the second region and the third region represent the same region as the first region, the first heat difference is the difference between the heat corresponding to the second region in the first infrared picture and the heat corresponding to the pixel points adjacent to the second region, the first infrared picture is an infrared picture taken at the same time as the target picture, the second heat difference is the difference between the heat corresponding to the second region in the first infrared picture and the heat corresponding to the third region in the second infrared picture, the second infrared picture is an infrared picture obtained by the infrared camera taking pictures of the airport runway before the shooting time of the first infrared picture, the first heat difference and the second heat difference are calculated according to the heat values of each pixel point recorded in the pixel infrared heat value library, the pixel infrared heat value library stores the coordinates of the heat value map in a time-series manner, and obtains pixel points through the infrared camera to construct an infrared heat value library corresponding to the airport runway; The second determination unit is configured to determine a second detection result of the target object according to the first heat difference and the second heat difference, where the second detection result is used to indicate whether the target object is an abnormal object appearing on the airport runway.

12. A computer-readable storage medium, characterized in that, the computer-readable storage medium stores a computer program, where when the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 10 are implemented.

13. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, when the processor executes the computer program, the steps of the method described in any one of claims 1 to 10 are implemented.

Images