Detection method, device and unmanned aerial vehicle

Abstract

The embodiment of the application provides a detection method, device and unmanned aerial vehicle, the method can be applied to the unmanned aerial vehicle field, the method comprises the following steps: obtaining a first image and a second image of a target scene; performing matching cost detection on the first image and the second image to obtain a matching cost detection result; inputting the matching cost detection result into a first neural network to obtain a disparity prediction result; inputting the matching cost detection result into a second neural network to obtain an uncertainty prediction result; inputting the disparity prediction result and the uncertainty prediction result into a third neural network to obtain a detection result, wherein the detection result is used to indicate whether the disparity prediction result of the first image and / or the second image is invalid. Through the above method, whether the disparity prediction result of the whole image is invalid can be detected, so that the three-dimensional reconstruction in the unmanned aerial vehicle scene is better completed, and the visual obstacle avoidance in the environment perception is better performed.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of unmanned aerial vehicles, and more particularly, to a detection method, device and unmanned aerial vehicle. BACKGROUND

[0002] Three-dimensional reconstruction in the unmanned aerial vehicle scenario refers to a process of using sensors and computing devices carried by an unmanned aerial vehicle, collecting a large amount of images or laser scanning data, and using computer vision and computer graphics technology to convert the data into a high-precision three-dimensional model or map.

[0003] When performing three-dimensional reconstruction, a stereo matching algorithm needs to be used, which can infer the three-dimensional position of an object in a scene based on the disparity information between multiple images. Although the stereo matching algorithm has achieved excellent performance in obtaining disparity information, considering the migration problem between the real scene and the training scene of the algorithm, and the fact that the quality of the image may be affected by the outside world, the stereo matching algorithm still has the phenomenon of prediction error. Because it is impossible to determine whether the stereo matching algorithm has made a prediction error, which may lead to disastrous consequences, how to determine whether the disparity prediction result is invalid is a problem that needs to be solved. SUMMARY

[0004] Embodiments of the present application provide a detection method, device and unmanned aerial vehicle, which can detect whether the disparity prediction result of the entire image is invalid, thereby better completing three-dimensional reconstruction in the unmanned aerial vehicle scenario and better performing visual obstacle avoidance in environmental perception.

[0005] In a first aspect, a detection method is provided, which includes: obtaining a first image and a second image of a target scene; performing matching cost detection on the first image and the second image to obtain a matching cost detection result, the matching cost detection result being used to indicate the similarity of a pixel pair in the first image and the second image; inputting the matching cost detection result into a first neural network to obtain a disparity prediction result; inputting the matching cost detection result into a second neural network to obtain an uncertainty prediction result, the uncertainty prediction result being used to indicate the degree of credibility of disparity prediction according to the pixel pair; and inputting the disparity prediction result and the uncertainty prediction result into a third neural network to obtain a detection result, the detection result being used to indicate whether the disparity prediction result of the first image and / or the second image is invalid.

[0006] Optionally, the first image and the second image can be obtained by a camera deployed on the unmanned aerial vehicle, or can be obtained by multiple cameras deployed on the unmanned aerial vehicle.

[0007] Optionally, when the first image and the second image are acquired by a binocular camera, the first image can be an image acquired by a left camera, and the second image can be an image acquired by a right camera, or the second image can be an image acquired by the left camera, and the first image can be an image acquired by the right camera.

[0008] Optionally, the input in the first neural network can be a cost volume in the matching cost detection result, which is used to represent the similarity between the feature maps at the pixel level to find the correspondence.

[0009] Optionally, the third neural network can be a fusion network.

[0010] Optionally, the detection result is used to indicate whether the disparity prediction result of the first image and / or the second image is invalid, which can be understood as the detection result being used to determine whether the disparity prediction result of the first image and / or the second image is invalid due to environmental factors.

[0011] In the embodiments of the present application, by inputting the matching cost detection result into the first neural network and the second neural network, the disparity prediction result and the uncertainty prediction result are obtained, and then the disparity prediction result and the uncertainty prediction result are input into the third neural network to obtain the detection result used to indicate whether the disparity prediction result of the entire image (the first image and / or the second image) is invalid. In this way, it can be detected whether the disparity prediction result of the entire image is invalid, thereby avoiding the occurrence of disastrous consequences.

[0012] In combination with the first aspect, in some implementations of the first aspect, the detection result is used to indicate that the disparity prediction result of the first image and / or the second image is not invalid, and the method further includes: determining a three-dimensional model or a map of the target scene according to the disparity prediction result of the first image and / or the second image.

[0013] In the embodiments of the present application, when the detection result is used to indicate that the disparity prediction result of the first image and / or the second image is not invalid, the three-dimensional model of the target scene can be determined according to the disparity prediction result and the uncertainty prediction result. In this way, in the process of constructing the three-dimensional model or the map, both the accuracy of the disparity prediction result and the rationality of the disparity prediction result are considered, and the three-dimensional reconstruction in the unmanned scene can be better completed.

[0014] In combination with the first aspect, in some implementations of the first aspect, the third neural network is obtained through sample data and data labels, the sample data includes: sample disparity prediction result, disparity ground truth, and sample uncertainty prediction result, and the sample label includes sample detection result.

[0015] Optionally, the sample uncertainty prediction result can be understood as a prediction result of uncertainty of a disparity prediction corresponding to each pixel in the sample.

[0016] Optionally, in the process of training the third neural network, the noise label can not be included in the training label, so that the third neural network with more accurate prediction can be trained.

[0017] Optionally, the training process of the third neural network can be: by calculating the variance between the sample disparity prediction result and the disparity ground truth, when the variance is greater than or equal to a preset threshold, outputting a detection result that the disparity prediction result of the first image and / or the second image is not invalid, and when the variance is less than the preset threshold, outputting a detection result that the disparity prediction result of the first image and / or the second image is invalid.

[0018] In the embodiments of the present application, the third neural network can be trained by the sample disparity prediction result, the disparity ground truth and the sample uncertainty prediction result, so as to judge whether the disparity prediction result of the first image and / or the second image is invalid based on the third neural network, thereby more efficiently completing the three-dimensional reconstruction in the unmanned aerial vehicle scene.

[0019] In combination with the first aspect, in some implementations of the first aspect, the matching cost detection on the first image and the second image to obtain a matching cost detection result comprises: inputting the first image and the second image into a convolutional neural network to obtain the matching cost detection result.

[0020] In the embodiments of the present application, the matching cost detection on the first image and the second image can be performed by the convolutional neural network, which can avoid manual design of features or similarity measurement criteria, making the matching cost detection more flexible and accurate, thereby better completing the three-dimensional reconstruction in the unmanned aerial vehicle scene.

[0021] In combination with the first aspect, in some implementations of the first aspect, the loss function corresponding to the training of the second neural network is determined according to an absolute error loss function, a disparity error distribution loss function and an uncertainty distribution loss function, and the disparity error distribution loss function and the uncertainty distribution loss function can be obtained by accumulating the weights corresponding to each pixel point in the first image and the second image.

[0022] In the embodiments of the present application, the disparity error distribution loss function and the uncertainty distribution loss function are introduced when the second neural network is trained, so that the above loss functions can guide the second neural network to not only consider the accuracy of prediction when predicting, but also reasonably estimate the uncertainty of prediction, thereby improving the robustness, reliability and generalization ability of the second neural network.

[0023] With reference to the first aspect, in some implementations of the first aspect, the loss function corresponding to the training of the second neural network is determined by the following formula:

[0024]

[0025] wherein N is the total number of pixel points in the first image and / or the second image, σ i is the uncertainty prediction result, d i is the disparity value of the i-th pixel, i≤N, H is the disparity ground truth value of the i-th pixel, i≤N, H ∈ (j) is the disparity error distribution loss function, H σ (j) is the uncertainty distribution loss function, and m is a preset constant capable of reflecting interval length.

[0026] In the embodiments of the present application, by introducing the disparity error distribution loss function and the uncertainty distribution loss function, the second neural network can take into account the accuracy and uncertainty of the prediction during the training process. Thus, since the training process of the second neural network is improved, the prediction result output by the second neural network in actual application can be used for three-dimensional reconstruction in the unmanned aerial vehicle scenario, thereby improving the accuracy and reliability of the final prediction result.

[0027] With reference to the first aspect, in some implementations of the first aspect, the disparity error distribution loss function can be determined by the following formula:

[0028]

[0029] wherein the disparity error ∈ (j) is determined by the following formula: j (j)

[0030]

[0031] wherein μ ∈ is the mean of the disparity error, b ∈ is the variance of the disparity error, λ1 and λ2 are constants, and α j is a preset constant capable of reflecting interval length.

[0032] With reference to the first aspect, in some implementations of the first aspect, the uncertainty distribution loss function can be determined by the following formula:

[0033]

[0034] wherein the uncertainty prediction result σ (i) ​​The weight w corresponding to the jth pixel j (σ (i) ) can be determined by the following formula:

[0035]

[0036] wherein μ ∈ is the mean of the disparity error, b ∈ is the variance of the disparity error, λ1 and λ2 are constants, and α j is a preset constant capable of reflecting the interval length.

[0037] In a second aspect, a detection apparatus is provided, which comprises: an acquisition unit configured to acquire a first image and a second image of a target scene; and a processing unit configured to: perform matching cost detection on the first image and the second image to obtain a matching cost detection result, the matching cost detection result being used to indicate the similarity of a pixel pair in the first image and the second image; input the matching cost detection result into a first neural network to obtain a disparity prediction result; input the matching cost detection result into a second neural network to obtain an uncertainty prediction result, the uncertainty prediction result being used to indicate the degree of confidence of disparity prediction according to the pixel pair; and input the disparity prediction result and the uncertainty prediction result into a third neural network to obtain a detection result, the detection result being used to indicate whether the disparity prediction result of the first image and / or the second image is invalid.

[0038] With reference to the second aspect, in some implementations of the second aspect, the detection result is used to indicate that the disparity prediction result of the first image and / or the second image is not invalid; and the processing unit is further configured to determine a three-dimensional model or a map of the target scene according to the disparity prediction result of the first image and / or the second image.

[0039] With reference to the second aspect, in some implementations of the second aspect, the third neural network is obtained through sample data and data labels, the sample data comprising: a sample disparity prediction result, a disparity ground truth and a sample uncertainty prediction result, and the sample labels comprising a sample detection result.

[0040] With reference to the second aspect, in some implementations of the second aspect, the processing unit is specifically configured to input the first image and the second image into a convolutional neural network to obtain the matching cost detection result.

[0041] With reference to the second aspect, in some implementations of the second aspect, the loss function corresponding to the training of the second neural network is determined according to an absolute error loss function, a disparity error distribution loss function, and an uncertainty distribution loss function, and the disparity error distribution loss function and the uncertainty distribution loss function can be obtained by accumulating the weights corresponding to each pixel point in the first image and the second image.

[0042] With reference to the second aspect, in some implementations of the second aspect, the loss function corresponding to the training of the second neural network is determined according to an absolute error loss function, a disparity error distribution loss function, and an uncertainty distribution loss function, and the disparity error distribution loss function and the uncertainty distribution loss function can be obtained by accumulating the weights corresponding to each pixel point in the first image and the second image.

[0043]

[0044] wherein N is the total number of pixel points in the first image and / or the second image, σ i is the uncertainty prediction result, d i is the disparity value of the i th pixel, i ≤ N, H is the disparity true value of the i th pixel, i ≤ N, H ∈ (j) is the disparity error distribution loss function, H σ (j) is the uncertainty distribution loss function, and m is a preset constant capable of reflecting interval length.

[0045] With reference to the second aspect, in some implementations of the second aspect, the loss function corresponding to the training of the second neural network is determined according to an absolute error loss function, a disparity error distribution loss function, and an uncertainty distribution loss function, and the disparity error distribution loss function and the uncertainty distribution loss function can be obtained by accumulating the weights corresponding to each pixel point in the first image and the second image.

[0046]

[0047] wherein the disparity error ∈ (i) is the weight w j (∈ (i) ) corresponding to the j th pixel can be determined according to the following formula:

[0048]

[0049] wherein μ ∈ is the mean of the disparity error, b ∈ is the variance of the disparity error, λ 1 and λ 2 are constants, and α j is a preset constant capable of reflecting interval length.

[0050] With reference to the second aspect, in some implementations of the second aspect, the loss function corresponding to the training of the second neural network is determined according to an absolute error loss function, a disparity error distribution loss function, and an uncertainty distribution loss function, and the disparity error distribution loss function and the uncertainty distribution loss function can be obtained by accumulating the weights corresponding to each pixel point in the first image and the second image.

[0051]

[0052] wherein the uncertainty prediction result σ (i) is the weight wj (σ (i) ) can be determined by the following formula:

[0053]

[0054] wherein μ ∈ is the mean of the disparity error, b ∈ is the variance of the disparity error, λ1 and λ2 are constants, and α j is a preset constant capable of reflecting the interval length.

[0055] In a third aspect, a detection device is provided, which includes at least one processor and a memory, the at least one processor being coupled to the memory and configured to read and execute instructions in the memory, so that the device implements the method in any one of the implementation manners of the first aspect.

[0056] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program code, when the computer program code is run on a computer, the computer is caused to execute the method in any one of the implementation manners of the first aspect.

[0057] In a fifth aspect, a chip is provided, which includes a circuit configured to execute the method in any one of the implementation manners of the first aspect.

[0058] In a sixth aspect, a computer program product is provided, which includes a computer program, when the computer program is run, the computer is caused to execute the method in any one of the implementation manners of the first aspect.

[0059] In a seventh aspect, a UAV is provided, which includes the detection device in any one of the implementation manners of the second aspect. BRIEF DESCRIPTION OF DRAWINGS

[0060] Figure 1 is a functional schematic diagram of a UAV provided by an embodiment of the present application;

[0061] Figure 2 is a problem of a disparity prediction result provided by an embodiment of the present application;

[0062] Figure 3 is a schematic flowchart of a detection method provided by an embodiment of the present application;

[0063] Figure 4 is a schematic flowchart of another detection method provided by an embodiment of the present application;

[0064] Figure 5 is a schematic diagram of a detection device provided by an embodiment of the present application;

[0065] Figure 6 is another detection device schematic diagram provided by an embodiment of the present application. DETAILED DESCRIPTION

[0066] The technical solutions in the embodiments of the present application will be described below with reference to the drawings.

[0067] The terms used in the following embodiments are only for the purpose of describing the specific embodiments and are not intended to be limiting of the present application. As used in the specification and the appended claims of the application, the singular forms “a,” “an” and “the” are intended to include both the singular and the plural forms, unless the context clearly indicates otherwise. It will be further understood that the terms “at least one” and “one or more” as used in the following embodiments refer to one, two or more than two. The term “and / or” is used to describe the association relationship of the associated objects, which means that there can be three relationships; for example, A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. The character “ / ” generally represents an “or” relationship between the associated objects before and after it.

[0068] In the present specification, the reference to “one embodiment” or “some embodiments” means that a particular feature, structure or characteristic described in connection with the embodiment is included in one or more embodiments of the present application. Therefore, the statements “in one embodiment”, “in some embodiments”, “in other some embodiments”, “in further some embodiments” and the like appearing in various places in the specification are not necessarily all referring to the same embodiment, but mean “one or more but not all embodiments”, unless otherwise specifically emphasized. The terms “include”, “contain”, “have” and their variants mean “include but not limited to”, unless otherwise specifically emphasized.

[0069] The scheme of the embodiments of the present application can be applied to a smart device. The smart device refers to any device, instrument or machine with computing processing capability. The smart device in the embodiments of the present application can be a robot, an autonomous vehicle, an intelligent assisted driving vehicle, an unmanned aerial vehicle, an intelligent assisted aircraft, a smart home device, etc. The present application does not make any limitation on the smart device. As long as the device can be installed with a camera or other sensors, it can be included in the scope of the smart device of the present application.

[0070] The method provided in this application can be applied to autonomous driving, drone navigation, agricultural drone operations, robot navigation, industrial non-contact inspection, 3D reconstruction, virtual reality, and other scenarios requiring 3D reconstruction. Specifically, the method in this application can be applied to agricultural drone operations, and a brief description of agricultural drone operations is provided below.

[0071] like Figure 1 As shown, the UAV 110 can be applied to various operational scenarios such as farmland monitoring, spraying and sowing, and plant disease and pest detection.

[0072] The unmanned aerial vehicle (UAV) 110 may include a sensor system 111. The sensor system 111 includes several sensors that sense information about the environment surrounding the UAV 110. For example, the sensor system 111 may include a positioning system (which may be GPS, BeiDou, or other positioning systems), an inertial measurement unit (IMU), radar, a laser rangefinder, and a camera. To obtain better perception results, the sensor system 111 can fuse information from multiple sensors. Specifically, the extrinsic parameters between the various sensors can be used to unify different sensors into the same coordinate system, thereby achieving the fusion of information from multiple sensors.

[0073] The 3D reconstruction module 112 can process and analyze the data acquired by the sensor system 111 using a specific algorithm to generate a 3D model or map of the target scene, and output the 3D model or map of the target scene to the upper-level functional module 113 and the display device 114.

[0074] The upper-layer functional module 113 can perform corresponding functions based on the 3D model or map of the target scene. In other words, the 3D reconstruction module 112 generates a 3D model or map of the target scene, which can be provided to the upper-layer operations of the UAV. For example, the positioning module can achieve more accurate UAV positioning based on the 3D model or map of the target scene. Furthermore, the obstacle avoidance module can combine the 3D model or map of the target scene to identify and assess potential obstacles in the UAV's environment.

[0075] Furthermore, the user can view the 3D model and map output by the drone on the display device 114, aiding in visualization, analysis, and decision-making for better drone control. It should be noted that... Figure 1 This is merely a schematic diagram of a system architecture provided in an embodiment of this application. The positional relationships between the devices, components, modules, etc. shown in the diagram do not constitute any limitation.

[0076] The technical problem to be solved and the technical solution adopted in this application are described below.

[0077] 3D reconstruction in drone scenarios refers to the process of using sensors and computing devices onboard the drone to collect large amounts of image or laser scan data, and then using computer vision and computer graphics techniques to transform this data into high-precision 3D models or maps. These sensors can include depth cameras, binocular cameras, multi-view cameras, and LiDAR sensors.

[0078] In 3D reconstruction, stereo matching algorithms are used. These algorithms infer the 3D positions of objects in a scene based on disparity information between multiple images. While stereo matching algorithms have achieved superior performance in obtaining disparity information, they still exhibit prediction errors due to the transfer problem between the real-world and training scenes, and the potential for image quality to be affected by external factors. For example... Figure 2 As shown, when performing 3D reconstruction of a sky photograph, the parallax prediction result differs significantly from the left eye image captured by the camera, indicating a prediction error.

[0079] Because it's impossible to determine whether a stereo matching algorithm has made a prediction error (e.g., whether the inaccurate parallax estimation of the entire image is due to backlighting, camera dirt, etc.), potentially leading to catastrophic consequences, determining whether the parallax prediction result has failed is a pressing issue that needs to be addressed.

[0080] This application provides a detection method, device, and drone that can detect whether the parallax prediction result of the entire image has failed, thereby better completing the 3D reconstruction in the drone scene and better performing visual obstacle avoidance in environmental perception.

[0081] Figure 3 This is a schematic flowchart of a detection method provided in an embodiment of this application. Figure 3 The detection method shown can be applied to Figure 1 In the drone, specifically, method 300 can be executed by the 3D reconstruction module 112, and method 300 can include steps S301 to S305.

[0082] S301, acquire the first and second images of the target scene.

[0083] Optionally, the first and second images can be acquired by a single camera deployed on the drone, or by multiple cameras deployed on the drone.

[0084] Optionally, when the first image and the second image are acquired by a binocular camera, the first image can be an image acquired by a left camera, and the second image can be an image acquired by a right camera, or the second image can be an image acquired by the left camera, and the first image can be an image acquired by the right camera.

[0085] S302, performing matching cost detection on the first image and the second image to obtain a matching cost detection result.

[0086] The matching cost detection result is used to indicate the similarity of the pixel pairs in the first image and the second image.

[0087] In one embodiment, step S302 can specifically include: inputting the first image and the second image into a convolutional neural network to obtain the matching cost detection result. In this way, manual design of features or similarity measurement criteria can be avoided, making the matching cost detection more flexible and accurate, and thus better completing three-dimensional reconstruction in an unmanned aerial vehicle environment.

[0088] S303, inputting the matching cost detection result into a first neural network to obtain a disparity prediction result.

[0089] Optionally, the input in the first neural network can be a cost volume in the matching cost detection result, which can be used to represent the pixel-level similarity between feature maps.

[0090] S304, inputting the matching cost detection result into a second neural network to obtain an uncertainty prediction result.

[0091] The uncertainty prediction result is used to indicate the degree of confidence of the disparity prediction according to the pixel pairs.

[0092] In one embodiment, the loss function corresponding to the training of the second neural network is determined according to an absolute error loss function, a disparity error distribution loss function, and an uncertainty distribution loss function. The disparity error distribution loss function and the uncertainty distribution loss function can be obtained by accumulating the weights corresponding to each pixel point in the first image and the second image. In this way, the above loss function can guide the second neural network to not only consider the accuracy of the prediction when making the prediction, but also reasonably estimate the uncertainty of the prediction, thereby improving the robustness, reliability, and generalization ability of the second neural network.

[0093] Optionally, the loss function corresponding to the training of the second neural network is specifically determined by the following formula:

[0094]

[0095] wherein N is the total number of pixel points in the first image and / or the second image, d iis the disparity value of the i-th pixel, σ i is the uncertainty prediction result, is the disparity ground truth of the i-th pixel, i≤N, H ∈ (j) is a disparity error distribution loss function, H σ (j) is an uncertainty distribution loss function, m is a preset constant capable of reflecting interval length.

[0096] Optionally, the disparity error distribution loss function can be determined by the following formula:

[0097]

[0098] wherein, disparity error ∈ (i) is the weight corresponding to the j-th pixel, w j (∈ (i) ) can be determined by the following formula:

[0099]

[0100] wherein, μ ∈ is the mean of disparity error, b ∈ is the variance of disparity error, λ1 and λ2 are constants, and α j is a preset constant capable of reflecting interval length.

[0101] Optionally, the uncertainty distribution loss function can be determined by the following formula:

[0102]

[0103] wherein, uncertainty prediction result σ (i) is the weight corresponding to the j-th pixel, w j (σ (i) ) can be determined by the following formula:

[0104]

[0105] wherein, μ ∈ is the mean of disparity error, b ∈ is the variance of disparity error, λ1 and λ2 are constants, and α j is a preset constant capable of reflecting interval length.

[0106] Optionally, the interval length in the above formula is obtained by dividing the calculable disparity range by the variance of disparity error, for example, the calculable disparity range is 0-192, and the variance is 8, and then the interval length is 0-24.

[0107] In the embodiment of the present application, the error between the disparity prediction result and the disparity ground truth and the uncertainty prediction result are modeled as a Gaussian distribution, and a negative log-likelihood loss function of the Gaussian distribution is constructed based on the distribution, so as to reduce the error of the disparity prediction in the training of the neural network and improve the accuracy of the uncertainty prediction.

[0108] S305, input the disparity prediction result and the uncertainty prediction result into the third neural network to obtain a detection result.

[0109] The detection result is used to indicate whether the disparity prediction result of the first image and / or the second image is invalid.

[0110] In one embodiment, in the case where the detection result indicates that the disparity prediction result is not invalid, after step S305, the method 300 can further include: determining a three-dimensional model or a map of the target scene according to the disparity prediction result and the uncertainty prediction result of the first image and / or the second image.

[0111] In one embodiment, the third neural network is obtained through sample data and data labels, the sample data includes: sample disparity prediction result, disparity ground truth and sample uncertainty prediction result, and the sample labels include sample detection results.

[0112] Optionally, in the process of training the third neural network, the noise label can not be included in the training label, so that a third neural network with more accurate prediction can be trained.

[0113] Optionally, the sample uncertainty prediction result can be understood as the prediction result of the uncertainty of the disparity prediction corresponding to each pixel in the sample.

[0114] Optionally, the training process of the third neural network can be: by calculating the variance between the sample disparity prediction result and the disparity ground truth, when the variance is greater than or equal to a preset threshold, outputting a detection result indicating that the disparity prediction result of the first image and / or the second image is not invalid, and when the variance is less than the preset threshold, outputting a detection result indicating that the disparity prediction result of the first image and / or the second image is invalid.

[0115] In the embodiment of the present application, by inputting the matching cost detection result into the first neural network and the second neural network, the disparity prediction result and the uncertainty prediction result are obtained, and then the disparity prediction result and the uncertainty prediction result are input into the third neural network to obtain a detection result indicating whether the disparity prediction result of the whole image (the first image and / or the second image) is invalid. In this way, it can be detected whether the disparity prediction result of the whole image is invalid, so as to avoid the occurrence of disastrous consequences.

[0116] Figure 4is a schematic flowchart of another detection method provided by an embodiment of the present application. The method 400 can be a more detailed description of steps S301-S305 in the method 300. The method 400 can include the following steps.

[0117] S401, performing feature extraction on the input images (e.g., the left image and the right image in FIG. 1) using a convolutional neural network. Figure 4

[0118] The left image and the right image can be the first image and the second image in the method 300.

[0119] Optionally, the left image and the right image can also be encoded into a low-resolution feature map in this step to facilitate processing by the neural network.

[0120] S402, performing matching cost calculation on the feature vectors extracted from the left image and the right image.

[0121] Specifically, the dot product between the feature vectors extracted from the left image and the right image can be used as a measure of similarity to calculate the similarity of all pixel pairs in the left image and the right image.

[0122] Optionally, a vector with a width x height x depth can be used to store the matching cost calculation results.

[0123] S403, inputting the matching cost calculation results into a disparity estimation subnetwork to obtain disparity prediction results at multiple resolutions.

[0124] Optionally, the disparity estimation subnetwork can be the first neural network in the method 300.

[0125] In one embodiment, when the disparity estimation subnetwork is trained, it can be optimized based on the following absolute error loss function.

[0126]

[0127] where N is the total number of pixel points in the first image and / or the second image, di is the disparity value of the i-th pixel, di true is the true value of the disparity of the i-th pixel, and i≤N. i

[0128] S404, inputting the cost volume in the matching cost calculation results into an uncertainty estimation subnetwork to obtain uncertainty prediction results.

[0129] Optionally, the uncertainty estimation subnetwork can be the second neural network in the method 300.

[0130] ​​​In one embodiment, when training the uncertainty estimation sub-network, the corresponding loss function can be determined based on the absolute error loss function, the disparity error distribution loss function, and the uncertainty distribution loss function. The disparity error distribution loss function and the uncertainty distribution loss function can be obtained by accumulating the weights corresponding to each pixel in the first and second images. Thus, by introducing the disparity error distribution loss function and the uncertainty distribution loss function, the second neural network can balance prediction accuracy and uncertainty during training. Because the training process of the second neural network is improved, its output prediction results can be used for 3D reconstruction in UAV scenarios, thereby improving the accuracy and reliability of the final prediction results.

[0131] Optionally, the loss function corresponding to training the uncertainty estimation subnetwork can be expressed as:

[0132]

[0133] Where N is the total number of pixels in the image, σ i For uncertain prediction results, d i Let be the disparity value of the i-th pixel. Let H be the true disparity value of the i-th pixel, i≤N, H ∈ (j) is the disparity error distribution loss function, H σ (j) is the uncertainty distribution loss function, and m is a preset constant that reflects the length of the interval.

[0134] H ∈ (j) and H σ (j) can be achieved by accumulating the weight w corresponding to each pixel. j The result is obtained after calculation using softmax, i.e.

[0135]

[0136]

[0137] Wherein, disparity error ∈ (i) The weight w corresponding to the j-th pixel j (∈ (i) ) and uncertainty prediction results σ (i) The weight w corresponding to the j-th pixel j (σ (i) It is calculated using the following formula:

[0138]

[0139]

[0140] Where, μ ∈, b ∈ respectively, are the mean and variance of the disparity error, and λ1 and λ2 are constants. j is a set of constants that can reflect the length of the interval.

[0141] S405, input the disparity prediction result and the uncertainty prediction result into the fusion network to obtain a detection result.

[0142] The detection result can be used to indicate whether the disparity prediction result of the left image and / or the right image is invalid.

[0143] Optionally, the fusion network can be a third neural network in the method 300, which can be used to judge whether the disparity prediction result of the whole image (left image and / or right image) is available by classifying the information of the fusion of the disparity prediction result and the uncertainty result. During the training process, if the noise label is directly added to the original disparity data set as the label of the fusion network, it may mislead the learning of the other two branch networks, therefore, when training the fusion network, the training label can not include the noise label.

[0144] Optionally, when training the fusion network, by calculating the variance between the sample disparity prediction result and the disparity ground truth, when the variance is greater than or equal to a preset threshold, the detection result that the disparity prediction result of the left image and / or the right image is not invalid is output, and when the variance is less than the preset threshold, the detection result that the disparity detection result of the left image and / or the right image is invalid is output. With the continuous iteration of network training, the uncertainty estimation branch and the disparity estimation branch in the fusion network gradually transit from underfitting to overfitting. In the early stage of training, the accuracy of the fusion network is low, and the error between the disparity prediction result and the disparity ground truth is large. With the improvement of the accuracy of the disparity estimation branch, the simple samples will be quickly mastered by the fusion network. Using the cycle to change the learning rate, switching the disparity estimation branch and the uncertainty estimation branch with high and low accuracy, can make the fusion network branch obtain a variety of samples, thereby enhancing the learning ability of the fusion network branch.

[0145] In the embodiments of the present application, by introducing the disparity error distribution loss function and the uncertainty distribution loss function, the neural network of the stereo matching algorithm can more comprehensively consider the accuracy and uncertainty of the prediction during the training process, and by introducing the fusion network, it can detect whether the disparity prediction result of the whole image (left image and / or right image) has been invalid, thereby better completing the three-dimensional reconstruction in the unmanned aerial vehicle scene, and better being used for visual obstacle avoidance in environment perception.

[0146] Figure 5FIG. 1 is a schematic diagram of a detection device 500 provided by an embodiment of the present application. The device 500 can include an acquisition unit 510, a storage unit 520, and a processing unit 530. The acquisition unit 510 is configured to acquire instructions and / or data. The acquisition unit 510 can also be referred to as a communication interface or a communication unit. The storage unit 520 is configured to implement a corresponding storage function and store corresponding instructions and / or data. The processing unit 530 can read instructions and / or data in the storage unit to enable the device 500 to implement the foregoing detection method.

[0147] The device 500 includes an acquisition unit 510 configured to acquire a first image and a second image of a target scene, and a processing unit 530 configured to: perform matching cost detection on the first image and the second image to obtain a matching cost detection result, the matching cost detection result being used to indicate a similarity of a pixel pair in the first image and the second image; input the matching cost detection result into a first neural network to obtain a disparity prediction result; input the matching cost detection result into a second neural network to obtain an uncertainty prediction result, the uncertainty prediction result being used to indicate a degree of credibility of disparity prediction according to the pixel pair; and input the disparity prediction result and the uncertainty prediction result into a third neural network to obtain a detection result, the detection result being used to indicate whether the disparity prediction result of the first image and / or the second image is invalid.

[0148] In a possible implementation, the detection result is used to indicate that the disparity prediction result is not invalid. The processing unit 530 is further configured to determine a three-dimensional model or a map of the target scene according to the disparity prediction result and the uncertainty prediction result of the first image and / or the second image.

[0149] In a possible implementation, the third neural network is obtained through sample data and a data label. The sample data includes a sample disparity prediction result, a disparity ground truth, and a sample uncertainty prediction result. The sample label is a sample detection result.

[0150] In a possible implementation, the processing unit 530 is specifically configured to input the first image and the second image into a convolutional neural network to obtain the matching cost detection result.

[0151] In a possible implementation, a loss function corresponding to the training of the second neural network is determined according to an absolute error loss function, a disparity error distribution loss function, and an uncertainty distribution loss function. The disparity error distribution loss function and the uncertainty distribution loss function can be obtained by accumulating weights corresponding to each pixel point in the first image and the second image.

[0152] In a possible implementation, the loss function corresponding to the training of the second neural network is specifically determined by the following formula:

[0153]

[0154] wherein, N is the total number of pixels in the first image and / or the second image, σ i is the uncertainty prediction result, d i is the disparity value of the i-th pixel, i≤N is the disparity true value of the i-th pixel, i≤N, H ∈ (j) is the disparity error distribution loss function, H σ (j) is the uncertainty distribution loss function, and m is a preset constant capable of reflecting interval length.

[0155] In a possible implementation, the disparity error distribution loss function can be determined by the following formula:

[0156]

[0157] wherein, disparity error ∈ (i) is the weight corresponding to the j-th pixel, w j (∈ (i) ) can be determined by the following formula:

[0158]

[0159] wherein, μ ∈ is the mean of disparity error, b ∈ is the variance of disparity error, λ1 and λ2 are constants, and α j is a preset constant capable of reflecting interval length.

[0160] In a possible implementation, the uncertainty distribution loss function can be determined by the following formula:

[0161]

[0162] wherein, uncertainty prediction result σ (i) is the weight corresponding to the j-th pixel, w j (σ (i) ) can be determined by the following formula:

[0163]

[0164] wherein, μ ∈ is the mean of disparity error, b ∈ is the variance of disparity error, λ1 and λ2 are constants, and α j is a preset constant capable of reflecting interval length.

[0165] Figure 6 is another detection device 600 provided by the embodiment of the application.

[0166] The apparatus 600 comprises a memory 610, a processor 620 and a communication interface 630. The memory 610, the processor 620 and the communication interface 630 are connected through internal connection paths. The memory 610 is configured to store instructions. The processor 620 is configured to execute the instructions stored in the memory 610 to control the communication interface 630 to acquire information, so that the apparatus 600 implements the foregoing detection method. Optionally, the memory 610 can be coupled to the processor 620 through an interface, or the memory 610 can be integrated with the processor 620.

[0167] It should be noted that the communication interface 630 uses a transceiving device such as but not limited to a transceiver. The communication interface 630 can also include an input / output interface.

[0168] The processor 620 stores one or more computer programs, which include instructions. When the instructions are executed by the processor 620, the apparatus 600 performs the three-dimensional reconstruction method in the foregoing embodiments.

[0169] It should be understood that, in the embodiments of the present application, the processor can be a central processing unit (CPU), and the processor can also be other general-purpose processors, digital signal processors (DSPs), ASICs, FPGAs or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0170] It should also be understood that, in the embodiments of the present application, the memory can include read-only memory and random access memory, and provide instructions and data for the processor. A part of the processor can also include non-volatile random access memory. For example, the processor can also store device type information.

[0171] In the implementation process, each step of the foregoing method can be completed by integrated logic circuits of hardware in the processor 620 or instructions in the form of software. The method disclosed in combination with the embodiments of the present application can be directly embodied as hardware processor execution completion, or executed by a combination of hardware and software modules in the processor. The software module can be located in a storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, register, etc. The storage medium is located in the memory 610, and the processor 620 reads the information in the memory 610, and combines the hardware to complete the steps of the foregoing method. To avoid repetition, it will not be described in detail here.

[0172] Optionally, Figure 6 The communication interface 630 in the middle can achieve Figure 5 The acquisition unit 510 in the middle, Figure 6 The memory 610 in the middle can achieve Figure 5 Storage unit 520 in the middle, Figure 6 The processor 620 in the middle can achieve Figure 5 The processing unit 530 in the middle.

[0173] Optionally, the device 500 or device 600 may be located in Figure 1 Among Chinese drones.

[0174] This application also provides a computer-readable storage medium storing program code that, when executed on a computer, causes the computer to perform the above-described... Figure 3 or Figure 4 Any of the methods mentioned above.

[0175] This application also provides a computer program product, which includes a computer program that, when run, causes the computer to perform the above-described actions. Figure 3 or Figure 4 Any of the methods mentioned above.

[0176] This application embodiment also provides a chip, including: a circuit, the circuit being used to perform the above... Figure 3 or Figure 4 Any of the methods mentioned above.

[0177] This application embodiment also provides a smart device, including: such as Figure 5 or Figure 6 Any of the detection devices shown.

[0178] Optionally, the aforementioned smart devices may be agricultural drones or vehicles.

[0179] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0180] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0181] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other manners. For example, the described device embodiments are merely schematic. The division of the units is merely logical function division. There can be other division manners in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0182] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0183] In addition, each functional unit in the various embodiments of the present application can be integrated into a processing unit, or each unit can be a physically separate unit, or two or more units can be integrated into one unit.

[0184] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, and the computer software product is stored in a storage medium, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0185] The above is merely specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method of detection, characterized in that, The method comprises: obtaining a first image and a second image of a target scene; performing matching cost detection on the first image and the second image to obtain a matching cost detection result, the matching cost detection result being used to indicate the similarity of a pixel pair in the first image and the second image; inputting the matching cost detection result into a first neural network to obtain a disparity prediction result; inputting the matching cost detection result into a second neural network to obtain an uncertainty prediction result, the uncertainty prediction result being used to indicate the reliability of disparity prediction according to the pixel pair, a loss function corresponding to the second neural network during training being determined according to an absolute error loss function, a disparity error distribution loss function and an uncertainty distribution loss function, the disparity error distribution loss function and the uncertainty distribution loss function being obtained by accumulating the weights corresponding to each pixel point in the first image and the second image; inputting the disparity prediction result and the uncertainty prediction result into a third neural network to obtain a detection result, the detection result being used to indicate whether the disparity prediction result of the first image and / or the second image is invalid, the third neural network being obtained through sample data and data labels, the sample data comprising a sample disparity prediction result, a disparity ground truth and a sample uncertainty prediction result, and the sample labels comprising a sample detection result; the loss function corresponding to the second neural network during training is determined by the following formula: wherein N is a total number of pixel points in the first image and / or the second image, is the uncertainty prediction result, is a disparity value of the i-th pixel, is a disparity true value of the i-th pixel, i≤N, is the disparity error distribution loss function, is the uncertainty distribution loss function, and m is a preset constant capable of reflecting interval length. the disparity error distribution loss function can be determined by the following formula: wherein the disparity error the weight corresponding to the jth pixel may be determined by the following equation: wherein, is a mean of the parallax error, is a variance of the parallax error, and is a constant, is a preset constant capable of reflecting the interval length.

2. The method of claim 1, wherein, the detection result is used to indicate that the disparity prediction result of the first image and / or the second image is not invalid, and the method further comprises: determining a three-dimensional model or a map of the target scene according to the disparity prediction result of the first image and / or the second image.

3. The method of claim 1 or 2, wherein, the matching cost detection on the first image and the second image to obtain a matching cost detection result comprises: inputting the first image and the second image into a convolutional neural network to obtain the matching cost detection result.

4. The method of claim 1 or 2, wherein, the uncertainty distribution loss function can be determined by the following formula: where the uncertainty prediction result the weight corresponding to the jth pixel may be determined by the following formula: wherein, is a mean of the parallax error, is a variance of the parallax error, and is a constant, is a preset constant capable of reflecting the interval length.

5. A detection device, characterized in that the apparatus comprises a unit or a module for performing the method of any one of claims 1 to 4.

6. A detection device, characterized in that comprises a processor and a memory, the processor being coupled to the memory, the memory being used to store a computer program or instructions, and the processor being used to execute the computer program or instructions in the memory, so that the method of any one of claims 1 to 4 is executed. 7.A computer readable storage medium, the computer readable storage medium storing program codes, when the computer program codes are run on a computer, causing the computer to execute the method of any one of claims 1 to 4. 8.An unmanned aerial vehicle, comprising the detection apparatus of claim 5 or 6.

Images