Three-dimensional reconstruction method, device and drone
By using neural networks on drones for parallax and uncertainty prediction, combined with matching cost detection results, the accuracy and reliability problems of the stereo matching algorithm in unmanned airport scenes are solved, and more efficient three-dimensional reconstruction is achieved.
Patent Information
- Application Number
- CN202311136109.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-04
- Publication Date
- 2024-12-27
- Estimated Expiration
- 2043-09-04
AI Technical Summary
In the unmanned airport scene, the parallax prediction results output by the stereo matching algorithm have problems such as unclear object edges, unclear internal textures of large objects, and inaccurate recognition of small objects, resulting in insufficient prediction accuracy and reliability.
By acquiring the first image and the second image of the target scene, input it into the first neural network for matching cost detection, then input it into the second neural network for parallax prediction, and combined with the uncertainty prediction results, the third neural network is used to use the third neural network Training is performed to improve the accuracy and credibility of parallax prediction to form a three-dimensional model or map.
The prediction accuracy and reliability of the stereo matching algorithm are improved, and the three-dimensional reconstruction in unmanned airport scenes can be better completed, and the accuracy and rationality of the parallax prediction results are enhanced.
Smart Images
Figure CN116977591B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of drone technology, and more specifically, to a three-dimensional reconstruction method, device and drone. Background Art
[0002] Three-dimensional reconstruction in drone scenarios refers to the process of using the sensors and computing equipment carried by the drone to collect a large amount of image or laser scanning data, and using computer vision and computer graphics techniques to convert this data into high-precision three-dimensional models or maps.
[0003] When performing 3D reconstruction, a stereo matching algorithm is required. The stereo matching algorithm can infer the 3D position of objects in the scene based on the disparity information between multiple images. Although the stereo matching algorithm has achieved relatively excellent performance when obtaining disparity information, the disparity prediction results output by the stereo matching algorithm still have problems such as unclear object edges, unclear internal textures of large objects, and inaccurate recognition of small objects. Therefore, how to improve the accuracy and reliability of stereo matching algorithm prediction is an urgent problem to be solved. Summary of the invention
[0004] The embodiments of the present application provide a three-dimensional reconstruction method, device and drone, which can improve the accuracy and reliability of stereo matching algorithm prediction, thereby better completing three-dimensional reconstruction in drone scenarios.
[0005] In a first aspect, a three-dimensional reconstruction method is provided, the method comprising: acquiring a first image and a second image of a target scene; inputting the first image and the second image into a first neural network to obtain a matching cost detection result, the matching cost detection result being used to indicate the similarity of pixel pairs in the first image and the second image; inputting the matching cost detection result into a second neural network to obtain a disparity prediction result; inputting the matching cost detection result into a third neural network to obtain an uncertainty prediction result, the uncertainty prediction result being used to indicate the degree of credibility of disparity prediction based on the pixel pairs; and determining a three-dimensional model or map of the target scene based on the disparity prediction result and the uncertainty prediction result.
[0006] Optionally, the first image and the second image may be acquired through one camera deployed on the drone, or may be acquired through multiple cameras deployed on the drone.
[0007] Optionally, a convolutional neural network may be used to extract features from the image taken by the camera to obtain feature vectors corresponding to the first image and the second image.
[0008] Optionally, the input to the second neural network may be a cost-volume in the matching cost detection result, which is used to represent the pixel-level similarity between feature maps to find the corresponding relationship.
[0009] In the embodiment of the present application, the matching cost detection result is input into the second neural network and the third neural network to obtain the disparity prediction result and the uncertainty prediction result, and then the three-dimensional model or map of the target scene is determined according to the disparity prediction result and the uncertainty prediction result. In this way, in the process of constructing the three-dimensional model or map, both the accuracy of the disparity prediction result and the rationality of the disparity prediction result are taken into account, thereby improving the accuracy and reliability of the stereo matching algorithm prediction, and being able to better complete the three-dimensional reconstruction in the drone scene.
[0010] In combination with the first aspect, in certain implementations of the first aspect, the third neural network is obtained through sample data and data labels, and the sample data includes: sample matching cost detection results; and the data labels are sample uncertainty prediction results.
[0011] In an embodiment of the present application, the third neural network can be trained using sample matching cost detection results and sample uncertainty prediction results, so that uncertainty prediction results can be subsequently obtained based on the third neural network, thereby improving the accuracy and reliability of the stereo matching algorithm prediction.
[0012] In combination with the first aspect, in certain implementations of the first aspect, the corresponding loss function during the training of the third neural network is determined based on the absolute error loss function, the disparity error distribution loss function and the uncertainty distribution loss function, and the disparity error distribution loss function and the uncertainty distribution loss function can be obtained by accumulating the weights corresponding to each pixel in the first image and the second image.
[0013] In an embodiment of the present application, a parallax error distribution loss function and an uncertainty distribution loss function are introduced when the third neural network is trained. In this way, the above-mentioned loss functions can guide the third neural network to not only consider the accuracy of the prediction when making predictions, but also to make a reasonable estimate of the uncertainty of the prediction, thereby improving the robustness, reliability and generalization ability of the third neural network.
[0014] In combination with the first aspect, in some implementations of the first aspect, the loss function corresponding to the training of the third neural network is specifically determined by the following formula:
[0015]
[0016] Wherein, N is the total number of pixels in the first image and / or the second image, σ iis the uncertainty prediction result, d i is the disparity value of the i-th pixel, is the true value of the disparity of the i-th pixel, i≤N, H ∈ (j) is the parallax error distribution loss function, H σ (j) is the uncertainty distribution loss function, and m is a preset constant that can reflect the length of the interval.
[0017] In the embodiment of the present application, by introducing the parallax error distribution loss function and the uncertainty distribution loss function, the third neural network can take into account both the accuracy and uncertainty of the prediction during the training process. In this way, since the training process of the third neural network is improved, the prediction results output by it in actual applications can be used for three-dimensional reconstruction in drone scenarios, thereby improving the accuracy and reliability of the stereo matching algorithm prediction.
[0018] In combination with the first aspect, in some implementations of the first aspect, the parallax error distribution loss function may be specifically determined by the following formula:
[0019]
[0020] Among them, the parallax error ∈ (i) The weight w corresponding to the jth pixel j (∈ (i) ) can be determined by the following formula:
[0021]
[0022] Among them, μ ∈ is the mean of the parallax error, b ∈ is the variance of the parallax error, λ1 and λ2 are constants, α j It is a preset constant that can reflect the length of the interval.
[0023] In conjunction with the first aspect, in some implementations of the first aspect, the uncertainty distribution loss function may be specifically determined by the following formula:
[0024]
[0025] Among them, the uncertainty prediction result σ (i) The weight w corresponding to the jth pixel j (σ (i) ) can be determined by the following formula:
[0026]
[0027] Among them, μ ∈ is the mean of the parallax error, b ∈is the variance of the parallax error, λ1 and λ2 are constants, α j It is a preset constant that can reflect the length of the interval.
[0028] In a second aspect, a three-dimensional reconstruction device is provided, which includes: an acquisition unit, used to acquire a first image and a second image of a target scene; a processing unit, used to: input the first image and the second image into a first neural network to obtain a matching cost detection result, wherein the matching cost detection result is used to indicate the similarity of pixel pairs in the first image and the second image; input the matching cost detection result into a second neural network to obtain a disparity prediction result; input the matching cost detection result into a third neural network to obtain an uncertainty prediction result, wherein the uncertainty prediction result is used to indicate the degree of credibility of disparity prediction based on the pixel pairs; and determine a three-dimensional model or map of the target scene based on the disparity prediction result and the uncertainty prediction result.
[0029] In combination with the second aspect, in certain implementations of the second aspect, the third neural network is obtained through sample data and data labels, and the sample data includes: sample matching cost detection results; and the data labels are sample uncertainty prediction results.
[0030] In combination with the second aspect, in certain implementations of the second aspect, the corresponding loss function during the training of the third neural network is determined based on the absolute error loss function, the disparity error distribution loss function and the uncertainty distribution loss function, and the disparity error distribution loss function and the uncertainty distribution loss function can be obtained by accumulating the weights corresponding to each pixel in the first image and the second image.
[0031] In conjunction with the second aspect, in some implementations of the second aspect, the loss function corresponding to the training of the third neural network is specifically determined by the following formula:
[0032]
[0033] Wherein, N is the total number of pixels in the first image and / or the second image, σ i is the uncertainty prediction result, d i is the disparity value of the i-th pixel, is the true value of the disparity of the i-th pixel, i≤N, H ∈ (j) is the parallax error distribution loss function, H σ (j) is the uncertainty distribution loss function, and m is a preset constant that can reflect the length of the interval.
[0034] In conjunction with the second aspect, in some implementations of the second aspect, the parallax error distribution loss function may be specifically determined by the following formula:
[0035]
[0036] Among them, the parallax error ∈ (i) The weight w corresponding to the jth pixel j (∈ (i) ) can be determined by the following formula:
[0037]
[0038] Among them, μ ∈ is the mean of the parallax error, b ∈ is the variance of the parallax error, λ1 and λ2 are constants, α j It is a preset constant that can reflect the length of the interval.
[0039] In conjunction with the second aspect, in some implementations of the second aspect, the uncertainty distribution loss function may be specifically determined by the following formula:
[0040]
[0041] Among them, the uncertainty prediction result σ (i) The weight w corresponding to the jth pixel j (σ (i) ) can be determined by the following formula:
[0042]
[0043] Among them, μ ∈ is the mean of the parallax error, b ∈ is the variance of the parallax error, λ1 and λ2 are constants, α j It is a preset constant that can reflect the length of the interval.
[0044] In a third aspect, a three-dimensional reconstruction device is provided, which includes: at least one processor and a memory, wherein the at least one processor is coupled to the memory and is used to read and execute instructions in the memory, so that the device implements the method in any one of the implementation modes of the first aspect above.
[0045] According to a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a program code, and when the computer program code is executed on a computer, the computer executes the method in any one of the implementations of the first aspect.
[0046] In a fifth aspect, a chip is provided, the chip comprising a circuit, the circuit being used to execute the method in any one of the implementations of the first aspect above.
[0047] According to a sixth aspect, a computer program product is provided. The computer product includes a computer program. When the computer program is executed, the computer executes the method in any one of the implementation modes of the first aspect.
[0048] In a seventh aspect, a drone is provided, comprising: a three-dimensional reconstruction device in any one of the implementations of the second aspect above. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] Figure 1 is a functional schematic diagram of a drone provided in an embodiment of the present application;
[0050] Figure 2 This is a problem with the disparity prediction result output by the stereo matching algorithm provided in the embodiment of the present application;
[0051] Figure 3 is a schematic flow chart of a three-dimensional reconstruction method provided in an embodiment of the present application;
[0052] Figure 4 is a schematic flow chart of another three-dimensional reconstruction method provided in an embodiment of the present application;
[0053] Figure 5 is a schematic diagram of a three-dimensional reconstruction device provided in an embodiment of the present application;
[0054] Figure 6 It is a schematic diagram of another three-dimensional reconstruction device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0055] The technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings.
[0056] The terms used in the following embodiments are only for the purpose of describing specific embodiments, and are not intended to be used as limitations on the present application. As used in the specification and the appended claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include expressions such as "one or more", unless there is a clear contrary indication in the context. It should also be understood that in the following embodiments of the present application, "at least one", "one or more" refer to one, two or more. The term "and / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist; for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.
[0057] References to "one embodiment" or "some embodiments" etc. described in this specification mean that a particular feature, structure or characteristic described in conjunction with the embodiment is included in one or more embodiments of the present application. Thus, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear at different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.
[0058] The solution of the embodiment of the present application can be applied to smart devices. Smart devices refer to any device, instrument or machine with computing and processing capabilities. The smart devices in the embodiments of the present application can be robots, autonomous vehicles, smart assisted driving vehicles, unmanned aerial vehicles, smart assisted aircraft, smart home devices, etc. The present application does not impose any restrictions on the smart devices. Any device that can be equipped with a camera or other sensor can be included in the scope of the smart devices of the present application.
[0059] The method provided in the embodiment of the present application can be applied to autonomous driving, drone navigation, agricultural drone operation, robot navigation, industrial non-contact detection, three-dimensional reconstruction, virtual reality and other scenarios requiring three-dimensional reconstruction. Specifically, the method provided in the embodiment of the present application can be applied to the scenario of agricultural drone operation, and the scenario of agricultural drone operation is briefly introduced below.
[0060] like Figure 1 As shown, the drone 110 can be used in operation scenarios such as farmland monitoring, spraying and sowing, and plant pest and disease detection.
[0061] Among them, the drone 110 may include a sensor system 111. The sensor system 111 includes several sensors that perceive information about the environment around the drone 110. For example, the sensor system 111 may include a positioning system (the positioning system may be a GPS system, or a Beidou system or other positioning systems), an inertial measurement unit (IMU), a radar, a laser rangefinder, and a camera. In order to obtain better perception results, the sensor system 111 can fuse information from multiple sensors. Specifically, different sensors can be unified into the same coordinate system through external parameters between each sensor, thereby realizing the fusion of multiple sensor information.
[0062] The 3D reconstruction module 112 can process and analyze the data acquired by the sensor system 111 using a specific algorithm to generate a 3D model or map of the target scene, and output the 3D model or map of the target scene to the upper-level functional module 113 and the display device 114.
[0063] The upper layer function module 113 can implement corresponding functions based on the three-dimensional model or map of the target scene. That is, the three-dimensional model or map of the target scene generated by the three-dimensional reconstruction module 112 can be provided to the upper layer business of the drone. For example, the positioning function module can realize more accurate drone positioning based on the three-dimensional model or map of the target scene. For another example, the obstacle avoidance function module can identify and evaluate potential obstacles in the drone's environment in combination with the three-dimensional model or map of the target scene.
[0064] Furthermore, the three-dimensional model and map output by the drone can be viewed on the display device 114 to help the user to visualize, analyze and make decisions to better control the drone. Figure 1 It is only a schematic diagram of a system architecture provided in an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation.
[0065] The following introduces the technical problems to be solved by this application and the technical solutions adopted.
[0066] 3D reconstruction in drone scenarios refers to the process of using the sensors and computing devices carried by drones to collect a large amount of image or laser scanning data and convert these data into high-precision 3D models or maps using computer vision and computer graphics technology. The above sensors can include depth cameras, binocular cameras, multi-view cameras, and lidar sensors.
[0067] When performing 3D reconstruction, a stereo matching algorithm is required. The stereo matching algorithm can infer the 3D position of objects in the scene based on the disparity information between multiple images. Although the stereo matching algorithm has achieved excellent performance in obtaining disparity information, Figure 2 As shown, the disparity prediction results output by the stereo matching algorithm still have problems such as unclear object edges, unclear internal textures of large objects, and inaccurate recognition of small objects. Therefore, how to improve the accuracy and reliability of the stereo matching algorithm prediction is an issue that needs to be solved urgently.
[0068] The embodiments of the present application provide a three-dimensional reconstruction method, device and drone, which can improve the accuracy and reliability of stereo matching algorithm prediction, thereby better completing three-dimensional reconstruction in drone scenarios.
[0069] Figure 3 is a schematic flow chart of a three-dimensional reconstruction method provided in an embodiment of the present application, Figure 3 The 3D reconstruction method shown can be applied to Figure 1 In the drone, specifically, the method 300 may be executed by the 3D reconstruction module 112, and the method 300 may include steps S301 to S305.
[0070] S301, acquiring a first image and a second image of a target scene.
[0071] Optionally, the first image and the second image may be acquired through one camera deployed on the drone, or may be acquired through multiple cameras deployed on the drone.
[0072] Optionally, a convolutional neural network may be used to extract features from the image taken by the camera to obtain feature vectors of the first image and the second image.
[0073] S302, inputting the first image and the second image into a first neural network to obtain a matching cost detection result.
[0074] The matching cost detection result is used to indicate the similarity between pixel pairs in the first image and the second image.
[0075] In one embodiment, when the first neural network is trained, the first neural network may be optimized by an absolute loss function as shown in the following formula:
[0076]
[0077] Wherein, N is the total number of pixels in the first image and / or the second image, d i is the disparity value of the i-th pixel, is the true value of the disparity of the i-th pixel, i≤N.
[0078] S303: Input the matching cost detection result into the second neural network to obtain a disparity prediction result.
[0079] Optionally, the input to the second neural network may be a cost volume in the matching cost detection result, which may be used to represent the pixel-level similarity between feature maps.
[0080] S304, inputting the matching cost detection result into the third neural network to obtain an uncertainty prediction result.
[0081] The uncertainty prediction result is used to indicate the reliability of disparity prediction based on pixel pairs.
[0082] In one embodiment, the third neural network is obtained through sample data and data labels, the sample data includes: sample matching cost detection results; the data label is the sample uncertainty prediction result. In this way, the third neural network can obtain uncertainty prediction results through the above model training method, thereby improving the accuracy and reliability of stereo matching algorithm prediction.
[0083] Specifically, the sample matching cost detection result can be input into the third neural network to obtain the uncertainty prediction result. The third neural network is trained by the uncertainty prediction result and the sample uncertainty prediction result. The training process can make the uncertainty prediction result output by the third neural network continuously solve the data label, that is, the process of minimizing the loss function.
[0084] In one embodiment, the corresponding loss function during the training of the third neural network is determined based on the absolute error loss function, the parallax error distribution loss function and the uncertainty distribution loss function, and the parallax error distribution loss function and the uncertainty distribution loss function can be obtained by accumulating the weights corresponding to each pixel in the first image and the second image. In this way, the above loss function can guide the third neural network to not only consider the accuracy of the prediction when making a prediction, but also to make a reasonable estimate of the uncertainty of the prediction, thereby improving the robustness, reliability and generalization ability of the third neural network.
[0085] Optionally, the corresponding loss function during the training of the third neural network is specifically determined by the following formula:
[0086]
[0087] Where N is the total number of pixels in the first image and / or the second image, σ i is the uncertainty prediction result, d i is the disparity value of the i-th pixel, is the true value of the disparity of the i-th pixel, i≤N, H ∈ (j) is the parallax error distribution loss function, H σ (j) is the uncertainty distribution loss function, and m is a preset constant that can reflect the length of the interval.
[0088] Optionally, the parallax error distribution loss function may be specifically determined by the following formula:
[0089]
[0090] Among them, the parallax error ∈ (i) The weight w corresponding to the jth pixel j (∈ (i) ) can be determined by the following formula:
[0091]
[0092] Among them, μ ∈ is the mean of the parallax error, b ∈ is the variance of the parallax error, λ1 and λ2 are constants, α j It is a preset constant that can reflect the length of the interval.
[0093] Optionally, the uncertainty distribution loss function can be specifically determined by the following formula:
[0094]
[0095] Among them, the uncertainty prediction result σ (i) The weight w corresponding to the jth pixel j (σ (i) ) can be determined by the following formula:
[0096]
[0097] Among them, μ ∈ is the mean of the parallax error, b ∈ is the variance of the parallax error, λ1 and λ2 are constants, α j It is a preset constant that can reflect the length of the interval.
[0098] Optionally, the interval length in the above formula is obtained by dividing the computable disparity range by the variance of the disparity error. For example, if the computable disparity range is 0-192 and the variance is 8, the interval length is 0-24.
[0099] In an embodiment of the present application, the error between the disparity prediction result and the true disparity value and the uncertainty prediction result are modeled as a Gaussian distribution, and a negative log-likelihood loss function of the Gaussian distribution is constructed based on the distribution, thereby reducing the error of the disparity prediction in the training of the neural network and improving the accuracy of the uncertainty prediction.
[0100] S305, determining a three-dimensional model or map of the target scene according to the disparity prediction result and the uncertainty prediction result.
[0101] In the embodiment of the present application, the matching cost detection result is input into the second neural network and the third neural network to obtain the disparity prediction result and the uncertainty prediction result, and then the three-dimensional model or map of the target scene is determined according to the disparity prediction result and the uncertainty prediction result. In this way, in the process of constructing the three-dimensional model or map, both the accuracy of the disparity prediction result and the rationality of the disparity prediction result are taken into account, thereby improving the accuracy and reliability of the stereo matching algorithm prediction, and being able to better complete the three-dimensional reconstruction in the drone scene.
[0102] Figure 4 It is a schematic flow chart of another three-dimensional reconstruction method provided in an embodiment of the present application. Method 400 may be a more detailed description of steps S301 to S305 in method 300. Method 400 may include the following steps.
[0103] S401, using a convolutional neural network to process an input image (e.g., Figure 4 The left and right images in the figure are used for feature extraction.
[0104] The left image and the right image may be the first image and the second image in method 300 .
[0105] Optionally, in this step, the left image and the right image can also be encoded into a low-resolution feature map to facilitate processing by the neural network.
[0106] S402, calculating the matching cost of the feature vectors extracted from the left image and the right image.
[0107] Specifically, the dot product between the feature vectors extracted from the left image and the right image can be used as a measure of similarity to calculate the similarity of all pixel pairs between the left image and the right image.
[0108] Optionally, the above matching cost calculation may be implemented by a neural network, and the neural network may be the first neural network in method 300 .
[0109] Optionally, a vector of width×height×depth may be used to store the above matching cost calculation result.
[0110] S403: Input the matching cost calculation result into the disparity estimation subnetwork to obtain disparity prediction results of multiple resolutions.
[0111] Optionally, the above disparity estimation subnetwork may be the second neural network in method 300 .
[0112] In one embodiment, when the disparity estimation subnetwork performs model training, optimization may be performed based on the following absolute error loss function.
[0113]
[0114] Wherein, N is the total number of pixels in the first image and / or the second image, d i is the disparity value of the i-th pixel, is the true value of the disparity of the i-th pixel, i≤N.
[0115] S404: Input the cost volume in the matching cost calculation result into the uncertainty estimation subnetwork to obtain an uncertainty prediction result.
[0116] Optionally, the uncertainty estimation subnetwork may be the third neural network in method 300 .
[0117] Optionally, the disparity prediction result obtained in the above step S403 and the uncertainty prediction result obtained in step S404 can be applied to the three-dimensional reconstruction process in the drone scene.
[0118] In one embodiment, the loss function used in the training process of the uncertainty estimation subnetwork can be based on the loss function of disparity calculation, and an additional loss function for measuring the similarity of error distribution and uncertainty distribution is added, that is, the overall loss function can be expressed as:
[0119]
[0120] Where N is the total number of pixels in the image, σ i is the uncertainty prediction result, d i is the disparity value of the i-th pixel, is the true value of the disparity of the i-th pixel, i≤N, H ∈ (j) is the parallax error distribution loss function, H σ (j) is the uncertainty distribution loss function, and m is a preset constant that can reflect the length of the interval.
[0121] H ∈ (j) and H σ (j) The weight w corresponding to each pixel can be accumulated j After softmax calculation, we get
[0122]
[0123]
[0124] Among them, the parallax error ∈ (i) The weight w corresponding to the jth pixel j (∈ (i) ) and the uncertainty prediction result σ (i) The weight w corresponding to the jth pixel j (σ (i) ) is calculated by the following formula:
[0125]
[0126]
[0127] Among them, μ ∈ , b ∈ are the mean and variance of the disparity error, respectively, and λ1 and λ2 are constants. jIt is a set of pre-set constants that can reflect the length of the interval.
[0128] In the embodiment of the present application, by introducing the parallax error distribution loss function and the uncertainty distribution loss function, the neural network of the stereo matching algorithm can more comprehensively consider the accuracy and uncertainty of the prediction during the training process. In this way, since the neural network of the stereo matching algorithm is improved during the training process, the prediction results output by it in actual applications can be used for three-dimensional reconstruction in drone scenarios, thereby improving the accuracy and reliability of the stereo matching algorithm's prediction.
[0129] Figure 5 is a schematic diagram of a three-dimensional reconstruction device 500 provided in an embodiment of the present application. The device 500 may include an acquisition unit 510, a storage unit 520, and a processing unit 530. The acquisition unit 510 is used to acquire instructions and / or data. The acquisition unit 510 may also be referred to as a communication interface or a communication unit. The storage unit 520 is used to implement a corresponding storage function and store corresponding instructions and / or data. The processing unit 530 may read the instructions and / or data in the storage unit so that the device 500 implements the aforementioned three-dimensional reconstruction method.
[0130] The device 500 includes: an acquisition unit 510, used to acquire a first image and a second image of a target scene; a processing unit 530, used to: input the first image and the second image into a first neural network to obtain a matching cost detection result, wherein the matching cost detection result is used to indicate the similarity of pixel pairs in the first image and the second image; input the matching cost detection result into a second neural network to obtain a disparity prediction result; input the matching cost detection result into a third neural network to obtain an uncertainty prediction result, wherein the uncertainty prediction result is used to indicate the credibility of disparity prediction based on pixel pairs; and determine a three-dimensional model or map of the target scene based on the disparity prediction result and the uncertainty prediction result.
[0131] In a possible implementation, the third neural network is obtained through sample data and data labels, where the sample data includes: sample matching cost detection results; and the data labels are sample uncertainty prediction results.
[0132] In one possible implementation, the corresponding loss function during the training of the third neural network is determined based on the absolute error loss function, the disparity error distribution loss function and the uncertainty distribution loss function, and the disparity error distribution loss function and the uncertainty distribution loss function can be obtained by accumulating the weights corresponding to each pixel in the first image and the second image.
[0133] In a possible implementation, the corresponding loss function during the training of the third neural network is specifically determined by the following formula:
[0134]
[0135] Where N is the total number of pixels in the first image and / or the second image, σ i is the uncertainty prediction result, d i is the disparity value of the i-th pixel, is the true value of the disparity of the i-th pixel, i≤N, H ∈ (j) is the parallax error distribution loss function, H σ (j) is the uncertainty distribution loss function, and m is a preset constant that can reflect the length of the interval.
[0136] In a possible implementation, the parallax error distribution loss function can be specifically determined by the following formula:
[0137]
[0138] Among them, the parallax error ∈ (i) The weight w corresponding to the jth pixel j (∈ (i) ) can be determined by the following formula:
[0139]
[0140] Among them, μ ∈ is the mean of the parallax error, b ∈ is the variance of the parallax error, λ1 and λ2 are constants, α j It is a preset constant that can reflect the length of the interval.
[0141] In a possible implementation, the uncertainty distribution loss function can be specifically determined by the following formula:
[0142]
[0143] Among them, the uncertainty prediction result σ (i) The weight w corresponding to the jth pixel j (σ (i) ) can be determined by the following formula:
[0144]
[0145] Among them, μ ∈ is the mean of the parallax error, b ∈ is the variance of the parallax error, λ1 and λ2 are constants, α j It is a preset constant that can reflect the length of the interval.
[0146] Figure 6 It is a schematic diagram of another three-dimensional reconstruction device 600 provided in an embodiment of the present application.
[0147] The device 600 includes: a memory 610, a processor 620, and a communication interface 630. The memory 610, the processor 620, and the communication interface 630 are connected through an internal connection path, the memory 610 is used to store instructions, and the processor 620 is used to execute the instructions stored in the memory 610 to control the communication interface 630 to obtain information, so that the device 600 implements the above-mentioned three-dimensional reconstruction method. Optionally, the memory 610 can be coupled to the processor 620 through an interface, or can be integrated with the processor 620.
[0148] It should be noted that the communication interface 630 uses a transceiver device such as, but not limited to, a transceiver. The communication interface 630 may also include an input / output interface.
[0149] The processor 620 stores one or more computer programs, which include instructions. When the instructions are executed by the processor 620, the device 600 executes the three-dimensional reconstruction method in the above-mentioned embodiments.
[0150] It should be understood that in the embodiments of the present application, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), ASICs, FPGAs or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.
[0151] It should also be understood that in the embodiment of the present application, the memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the processor may also include a non-volatile random access memory. For example, the processor may also store information about the device type.
[0152] In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor 620 or an instruction in the form of software. The method disclosed in conjunction with the embodiment of the present application can be directly embodied as a hardware processor for execution, or a combination of hardware and software modules in the processor for execution. The software module can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 610, and the processor 620 reads the information in the memory 610 and completes the steps of the above method in conjunction with its hardware. To avoid repetition, it is not described in detail here.
[0153] Optionally, Figure 6 The communication interface 630 in the embodiment can be implemented Figure 5 The acquisition unit 510 in Figure 6 The memory 610 in the embodiment can realize Figure 5 The storage unit 520, Figure 6 The processor 620 in the embodiment can implement Figure 5 The processing unit 530 in.
[0154] Optionally, the device 500 or the device 600 may be located at Figure 1 Among the drones.
[0155] The embodiment of the present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a program code, and when the computer program code is executed on a computer, the computer executes the above Figure 3 or Figure 4 Any of the methods in .
[0156] The present application also provides a computer program product, which includes a computer program. When the computer program is executed, the computer executes the above Figure 3 or Figure 4 Any of the methods in .
[0157] The present application also provides a chip, including: a circuit, the circuit is used to execute the above Figure 3 or Figure 4 Any of the methods in .
[0158] The present application also provides a smart device, including: Figure 5 or Figure 6 Any of the three-dimensional reconstruction devices shown.
[0159] Optionally, the above-mentioned intelligent device is an agricultural drone or vehicle.
[0160] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0161] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0162] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0163] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0164] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0165] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0166] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A three-dimensional reconstruction method, characterized in that: The method comprises: Acquire a first image and a second image of a target scene; Inputting the first image and the second image into a first neural network to obtain a matching cost detection result, wherein the matching cost detection result is used to indicate the similarity of pixel pairs in the first image and the second image; Inputting the matching cost detection result into a second neural network to obtain a disparity prediction result; Inputting the matching cost detection result into a third neural network to obtain an uncertainty prediction result, wherein the uncertainty prediction result is used to indicate the credibility of disparity prediction based on the pixel pair, wherein the third neural network is obtained through sample data and data labels, wherein the sample data includes: a sample matching cost detection result; and the data label is a sample uncertainty prediction result; Determining a three-dimensional model or map of the target scene according to the disparity prediction result and the uncertainty prediction result; The loss function corresponding to the training of the third neural network is determined according to an absolute error loss function, a disparity error distribution loss function and an uncertainty distribution loss function, and the disparity error distribution loss function and the uncertainty distribution loss function are obtained by accumulating the weights corresponding to each pixel point in the first image and the second image; The corresponding loss function during the training of the third neural network is specifically determined by the following formula: Wherein, N is the total number of pixels in the first image and / or the second image, σ i is the uncertainty prediction result, d i is the disparity value of the i-th pixel, is the true value of the disparity of the i-th pixel, i≤N, H ∈ (j) is the parallax error distribution loss function, H σ (j) is the uncertainty distribution loss function, and m is a preset constant that can reflect the interval length.
2. The method according to claim 1, characterized in that The parallax error distribution loss function is specifically determined by the following formula: Among them, the parallax error ∈ (i) The weight w corresponding to the jth pixel j (∈ (i) ) is determined by the following formula: Among them, μ ∈ is the mean of the parallax error, b ∈ is the variance of the parallax error, λ1 and λ2 are constants, α j It is a preset constant that can reflect the length of the interval.
3. The method according to claim 1 or 2, characterized in that The uncertainty distribution loss function is specifically determined by the following formula: Among them, the uncertainty prediction result σ (i) The weight w corresponding to the jth pixel j (σ (i) ) is determined by the following formula: Among them, μ ∈ is the mean of the parallax error, b ∈ is the variance of the parallax error, λ1 and λ2 are constants, α j It is a preset constant that can reflect the length of the interval.
4. A three-dimensional reconstruction device, characterized in that: The method comprises a unit or a module for executing the method as claimed in any one of claims 1 to 3.
5. A three-dimensional reconstruction device, characterized in that: The method comprises a processor and a memory, wherein the processor is coupled to the memory, the memory is used to store computer programs or instructions, and the processor is used to execute the computer program or instructions in the memory, so that the method according to any one of claims 1 to 3 is executed. 6 . A computer-readable storage medium storing a program code, wherein when the computer program code is executed on a computer, the computer is caused to execute the method according to claim 1 .
7. An unmanned aerial vehicle, comprising the three-dimensional reconstruction device according to claim 4 or 5.