Image depth information sensing method for fruit and vegetable picking robot

Abstract

The invention is suitable for the field of fruit tree picking robots, and particularly relates to an image depth information sensing method for a fruit and vegetable picking robot, and the method comprises the steps: collecting a multi-view image, extracting the color and density of a sampling point based on a multi-layer perceptron, and generating a left view, a middle view, a right view and a disparity map through rendering a three-dimensional scene; performing feature extraction to obtain a multi-scale feature map and a multi-scale context feature map; constructing a multi-view geometric cost body to obtain a correlation volume and a geometric coding volume; performing cost volume operation to obtain a weighted correlation volume; obtaining an initial disparity map by using a softmax function to obtain a final initial disparity map; and obtaining an updated disparity map through an iteration updater. The method not only overcomes the limitation of a binocular camera under the conditions of illumination variation and shielding, but also solves the problem of demand for a large amount of annotation data in a traditional depth estimation method, thereby providing more accurate and robust depth perception capability for the fruit and vegetable picking robot.

Description

Technical Field

[0001] The present invention belongs to the field of fruit picking robots, and in particular relates to an image depth information perception method for a fruit and vegetable picking robot. Background Art

[0002] As the global agricultural automation process accelerates, fruit and vegetable picking robots have become a hot topic in modern agricultural research. However, the efficient operation of fruit and vegetable picking robots is inseparable from accurate depth estimation, so that the robot can accurately identify and locate the position of the fruit. Although the mainstream depth cameras on the market can provide high-precision depth information, their high cost limits their popularization and application in large-scale farms.

[0003] Currently, binocular cameras have been widely used in many fields as a depth perception method. However, in the face of complex and changeable orchard environments, the performance of binocular cameras is slightly insufficient, especially in situations where natural lighting conditions are extremely changeable and trees frequently block the view. The accuracy and robustness of their parallax estimation are often greatly reduced, which is undoubtedly a major challenge for the pursuit of efficient and accurate fruit and vegetable harvesting automation. Summary of the invention

[0004] The purpose of the present invention is to provide an image depth information perception method for a fruit and vegetable picking robot, aiming to solve the problem that the accuracy and robustness of parallax estimation are often greatly reduced in situations where natural lighting conditions are extremely changeable and tree occlusions are frequent.

[0005] The present invention is implemented as follows: a method for perceiving image depth information of a fruit and vegetable picking robot, the method comprising:

[0006] Collect multi-view images, extract the color and density of the sampling points based on a multi-layer perceptron, and generate the left view, middle view, right view and disparity map by rendering the 3D scene;

[0007] Extract features from the left view and the middle view to obtain a multi-scale feature map and a multi-scale context feature map;

[0008] Construct a multi-view geometric cost volume based on the multi-scale feature map to obtain the correlation volume and geometric coding volume;

[0009] Perform cost volume operation according to the correlation volume and left view key information to obtain a weighted correlation volume;

[0010] The softmax function is used to obtain the initial disparity map through the weighted correlation volume, and the initial disparity map is optimized to obtain the final initial disparity map;

[0011] The input geometry encoding volume, correlation volume and final initial disparity map are passed through an iterative updater to obtain an updated disparity map.

[0012] Preferably, the steps of collecting multi-view images, extracting the color and density of sampling points based on a multi-layer perceptron, and generating a left view, a middle view, a right view and a disparity map by rendering a three-dimensional scene include:

[0013] Collect a multi-view image set, perform spatial point sampling and position encoding, and record the camera pose corresponding to each set of images, wherein the camera pose includes a spatial position and an observation direction;

[0014] The camera pose is used as the input of the multi-layer perceptron, and a nonlinear transformation is performed through the L-layer multi-layer perceptron to obtain the color and density of each sampling point;

[0015] The color and density of the sampling points are integrated into the color of each pixel on the ray through volume rendering, and a three-dimensional scene is rendered to generate a left view, a middle view, and a right view;

[0016] In the process of rendering a 3D scene, a disparity map is extracted based on the implicit depth map;

[0017] The steps of spatial point sampling and position encoding include encoding the input coordinates P through sine and cosine functions. Position encoding maps 3D coordinates to high-dimensional space vector representation. The position encoding E(P) is expressed as:

[0018] γ(p)=(sin(2 0 πp),cos(2 0 πp),sin(2 1 πp),cos(2 1 πp),...,sin(2 L-1 πp),cos(2 L-1 πp))

[0019] Where N represents the number of frequencies and P represents the 3D coordinates.

[0020] Preferably, the step of using the camera posture as the input of the multilayer perceptron, performing nonlinear transformation through the L-layer multilayer perceptron, and obtaining the color and density of each sampling point includes: using the encoded spatial position and observation direction as the input X0, performing nonlinear transformation through the L-layer multilayer perceptron, and obtaining the color and density of each sampling point, and the calculation formula of each layer is:

[0021] x l =σ(W l x l-1 +b l )

[0022] Among them, σ is the activation function, W and b are the weight matrix and bias vector.

[0023] Predict the color and density of each sample point through a multi-layer perceptron:

[0024] (c i ,σ i )=MLP(γ(x i ),γ(d))

[0025] Among them, c i is the sampling point color, σ i is the density, x i is the sampling point and d is the direction.

[0026] Preferably, the color and density of the sampling points are integrated into the color of each pixel on the ray by volume rendering, and a three-dimensional scene is rendered to generate a left view, a middle view, and a right view, including: defining camera parameters of the left viewpoint, the middle viewpoint, and the right viewpoint, the camera external parameters of the middle view are Ek=Rk|tk, Rk represents a rotation matrix, which is a 3×3 unit matrix, tk represents a translation matrix, and the external parameters of the right view camera are:

[0027]

[0028] The images corresponding to the three viewpoints Ek, EkR and EkL form a stereo triplet, namely the left view, the middle view and the right view, and the generated dataset is used as a training set to train the disparity network.

[0029] Preferably, in the process of rendering a three-dimensional scene, the step of extracting a disparity map according to an implicit depth map specifically includes: extracting disparity from the depth map implicit in the rendering process to obtain a disparity map, and aligning the central image of the triplet, the rendered depth map represents the distance from the camera to each pixel in the scene, the disparity map refers to the disparity of the corresponding pixel points in two images of different viewing angles after stereo correction, and the disparity map can be converted from the depth map, which is expressed as:

[0030]

[0031] Where b is the baseline distance, f is the focal length of the camera, and z is the depth.

[0032] Preferably, the step of extracting features from the left view and the middle view to obtain a multi-scale feature map and a multi-scale context feature map comprises: using a feature extractor and a context extractor to extract features to obtain a multi-scale feature map and a multi-scale context feature map respectively, wherein the feature extractor: given a left view image I l and the intermediate view image I c, with a size of 3×H×W. The image is reduced to 1 / 32 of the original size on the pre-trained MobileNetV2 model, and the image is restored to 1 / 4 of the original size by upsampling to obtain feature maps of different sizes. The context extractor consists of a residual block and a downsampling layer. The residual block allows the gradient to propagate in the network. The resolution of the feature map is gradually reduced during the downsampling process to provide the context features required for time series processing.

[0033] Preferably, the step of constructing a multi-view geometric cost volume based on the multi-scale feature map to obtain a correlation volume and a geometric coding volume comprises: l , Middle View I c and right view I r Extract respectively, the left feature f l,4 , intermediate feature f c,4 and right feature f r,4 , divide the number of channels of the feature map into g groups, calculate the relevant volume for each group, and calculate the correlation between the left and middle views Correlation between the middle and right views

[0034]

[0035] Among them, <·,·> represents the inner product, d represents the disparity, c is the number of feature layers, and the geometric encoding volume with global information is obtained through the 3D regularization network R

[0036] Preferably, the step of obtaining an initial disparity map by using a softmax function through a weighted correlation volume, and optimizing the initial disparity map to obtain a final initial disparity map includes:

[0037] Use the softmax function to calculate the corresponding cost volume value under different disparities d. The larger the cost volume corresponding to the disparity, the smaller the corresponding weight value. Multiply the different disparities with their corresponding weights and perform cumulative operations to obtain the initial disparity d. l and d c ,

[0038]

[0039] d l represents the disparity between the left view and the center view, d c represents the disparity between the center view and the right view, C G1 (d) indicates the corresponding C when the parallax is d. G1 The value of

[0040] The initial disparity is compared with the disparity d generated by NeRF NeRFPerform pixel-by-pixel comparison and take the disparity corresponding to the pixel with the smallest error as the final initial disparity map d0:

[0041] d0=min{|d NeRF -d l |,|d NeRF -d c |}.

[0042] Preferably, the step of inputting the geometric coding volume, the correlation volume and the final initial disparity map to obtain the updated disparity map through the iterative updater specifically includes: converting the depth map into the required disparity map, and the conversion formula is:

[0043]

[0044] Where Z is the depth, b is the camera baseline length, f is the camera focal length, and d is the parallax;

[0045] After obtaining the initial disparity map d0, extract the geometric features G k :

[0046]

[0047] Concat represents the concatenation of geometric coding volumes C from different disparity offsets along the channel direction. Gf and all pairs of correlation volumes C GA Features of and their pooled versions and r represents the maximum step size of the disparity search range, i.e., the maximum disparity value, and m represents the number of operations performed on each disparity step size or the length of the sliding window. G is processed through two encoder layers. k With the current disparity d n , and form the input x n , update the hidden state h of ConvGRU n-1 , to incrementally correct the disparity estimate;

[0048] The updated disparity map is:

[0049] d n+1 =d n +△d n .

[0050] Preferably, the method includes using triple photometric loss as a loss function, the formula is:

[0051]

[0052] I c The image representing the intermediate view, I r→cIt means matching the right view image to the middle view through parallax. SSIM is an indicator for evaluating the similarity of two images in terms of structure and content. The variable α is a weight parameter.

[0053] The image depth information perception method for a fruit and vegetable picking robot provided by the present invention not only overcomes the limitations of binocular cameras under conditions of lighting changes and occlusion, but also solves the problem of the need for a large amount of labeled data in traditional depth estimation methods, thereby providing the fruit and vegetable picking robot with a more accurate and robust depth perception capability. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 A schematic diagram of the overall process of the image depth information perception method for a fruit and vegetable picking robot provided by an embodiment of the present invention;

[0055] Figure 2 A flowchart of NeRF creating multi-view images and disparity supervision provided by an embodiment of the present invention;

[0056] Figure 3 A flowchart of self-supervised disparity network training provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0058] It is understood that the terms "first", "second", etc. used in this application may be used herein to describe various elements, but unless otherwise specified, these elements are not limited by these terms. These terms are only used to distinguish a first element from another element. For example, without departing from the scope of this application, a first xx script may be referred to as a second xx script, and similarly, a second xx script may be referred to as a first xx script.

[0059] In the present invention, the NeRF model is first used to generate high-quality multi-view images from the collected real orchard scenes. These images not only contain rich color and density information, but also provide depth clues under different perspectives. Then, the present invention uses these multi-view images to train the IGEV disparity network. The IGEV network fuses the information of multi-view images through geometric coding convolution to generate an accurate disparity map, and the disparity information generated by NeRF is used to supervise the network. Through this self-supervised training method, the present invention overcomes the challenges brought about by the lack of high-quality training samples in traditional methods, and significantly improves the performance of disparity estimation and depth estimation. Figure 1 shown.

[0060] When only the left view and the right view are used, due to the occlusion problem between branches and fruits, some areas may only be seen in one view but not in the other. To alleviate this problem, providing an intermediate perspective as an additional perspective can reduce the error caused by the perspective difference, which is conducive to better processing the occlusion information and enhancing the robustness of the model when facing complex and changing scenes.

[0061] The present invention comprises three parts: NeRF to create multi-view images and disparity supervision, self-supervised disparity network training and loss function design.

[0062] The core of NeRF technology is to use machine learning to infer and render three-dimensional scenes from given two-dimensional images taken at different positions and angles. Through inverse rendering, high-fidelity images from any new perspective can be synthesized. This paper uses NeRF to synthesize left, center, and right views for training the disparity network, and converts the implicit depth information of NeRF into disparity information as the supervision of the disparity network, thereby realizing self-supervised disparity network training. Figure 2 The following is a detailed explanation of this process:

[0063] like Figure 1 FIG. 1 is a schematic diagram of the overall process of a method for perceiving image depth information of a fruit and vegetable picking robot provided by an embodiment of the present invention. The method includes:

[0064] Multi-view images are collected, the color and density of the sampling points are extracted based on a multi-layer perceptron, and the left view, middle view, right view and disparity map are generated by rendering the three-dimensional scene.

[0065] In this embodiment, the steps of collecting multi-view images, extracting the color and density of sampling points based on a multi-layer perceptron, and generating a left view, a middle view, a right view and a disparity map by rendering a three-dimensional scene include:

[0066] A multi-view image set is collected, spatial point sampling and position encoding are performed, and the camera pose corresponding to each set of images is recorded, wherein the camera pose includes a spatial position and an observation direction.

[0067] In this step, if Figure 2 As shown, a handheld camera is used to collect a set of omnidirectional, multi-view images of the orchard scene I = {I1, I2, ..., I N}.

[0068] For spatial point sampling, each pixel in the image will have a ray emitted from the optical center of the camera. On each ray, sampling is performed at a fixed interval. The coordinates of the sampling points are as follows:

[0069] x i =o+t i d, ti ∈[t n , t f ]

[0070] Among them, tn and tf are the shortest and farthest distances of the ray, and xi is the coordinate of the sampling point.

[0071] Perform position encoding: record the camera pose corresponding to each image, including spatial position (x, y, z) and viewing direction (φ, θ), and encode the input coordinates P through sine and cosine functions. Position encoding maps 3D coordinates to high-dimensional space vector representation. This is conducive to model convergence of high-resolution images. Specifically, the position encoding E(P) can be expressed as

[0072] γ(p)=(sin(2 0 πp),cos(2 0 πp), sin(2 1 πp),cos(2 1 πp),...,sin(2 L-1 πp), cos(2 L-1 πp))

[0073] Where N represents the number of frequencies and P represents the 3D coordinates.

[0074] The camera pose is used as the input of the multi-layer perceptron, and a nonlinear transformation is performed through the L-layer multi-layer perceptron to obtain the color and density of each sampling point.

[0075] In this step, MLP (Multi-layer Perceptron) prediction is performed. The encoded spatial position and viewing direction are used as input X0, and nonlinear transformation is performed through the MLP of the L layer to obtain the color and density of each sampling point. The calculation formula for each layer is:

[0076] x l =σ(W l x l-1 +b l )

[0077] Among them, σ is the activation function, W and b are the weight matrix and bias vector.

[0078] Predict the color and density of each sample point through MLP:

[0079] (c i ,σ i )=MLP(γ(x i ),γ(d))

[0080] Among them, c i is the color of the sampling point, σi is the density, x i is the sampling point and d is the direction.

[0081] Volume rendering is used to integrate the color and density of the sampling points into the color of each pixel on the ray, and a three-dimensional scene is rendered to generate left, middle and right views.

[0082] In this step,

[0083] in,

[0084] δ i =t i+1 -t i , is the distance between adjacent sampling points.

[0085] In order to obtain multi-view images, the present invention defines the camera parameters of the left, middle and right viewpoints, wherein the camera external parameters of the middle view are Ek=Rk|tk, Rk represents the rotation matrix, which is a 3×3 unit matrix, and tk represents the translation matrix; then the external parameters of the right view camera are:

[0086]

[0087] Where b = (b, 0, 0) T is the baseline distance vector; the extrinsic parameters of the camera for the left view are:

[0088]

[0089] The images corresponding to these three viewpoints Ek, EkR and EkL form a stereo triplet, namely the left, center and right views. The generated large data sets can be used as training sets to train the disparity network.

[0090] In the process of rendering a 3D scene, a disparity map is extracted based on the implicit depth map;

[0091] In this step, the disparity is extracted from the depth map implicit in the rendering process to obtain the disparity map, and the central image of the triplet is aligned. The rendered depth map represents the distance from the camera to each pixel in the scene. The disparity map refers to the disparity of the corresponding pixel points in two different perspective images after stereo correction. The disparity map can be converted from the depth map as shown below

[0092] Where b is the baseline distance, f is the focal length of the camera, and z is the depth

[0093] like Figure 3As shown in the figure, self-supervised disparity network training: using binocular stereo matching method to estimate the depth of fruit and vegetable targets, the training mechanism of self-supervised disparity network is deeply explored. The present invention uses multi-view images generated by NeRF as training data, optimizes the disparity estimation network through self-supervised learning, and does not require external supervision labels, thereby achieving significant performance improvement in depth estimation and stereo matching tasks. Figure 3 The following is a detailed expansion of the process:

[0094] Feature extraction is performed on the left view and the middle view to obtain multi-scale feature maps and multi-scale context feature maps.

[0095] In this step, the left view image and the middle view image are input for feature extraction to obtain a multi-scale feature map and a multi-scale context feature map. The feature extraction part is composed of a feature extractor and a context extractor, which extract multi-scale features and multi-scale context features respectively.

[0096] Feature Extractor: Given a left view image I l and the intermediate view image I c (Middle view image I c With the right view image I r Similarly), their size is 3×H×W. First, the image is reduced to 1 / 32 of the original size on the pre-trained MobileNetV2 model. After the image size is reduced, each feature point covers a larger area of ​​the original image. This means that each feature point can aggregate more global information, which helps capture a wider range of context information. Then, the image is restored to 1 / 4 of the original size by upsampling, thereby restoring detail information and enhancing spatial resolution, and finally obtaining feature maps of different sizes. Where i represents different scales (i=4, 8, 16, 32). Feature map f l,4 and f r,4 (or c) is used to construct the multi-view geometric cost volume, and the feature maps of other scales are used to guide the 3D regularization network to improve the accuracy of disparity estimation.

[0097] Context extractor: It consists of residual blocks and downsampling layers. The residual blocks allow the gradient to propagate in the network, which can alleviate the problem of gradient disappearance and is conducive to the training of deep network models. In the downsampling process, the resolution of the feature map is gradually reduced (H / I, w / i, i=4, 8, 16), which provides the context features required for time series processing, which are used for initialization and updating of convolutional gated recurrent units and enhance the performance of disparity estimation. Effective feature updates and fusions are performed at different scales, ultimately enhancing the effect of stereo matching.

[0098] A multi-view geometric cost volume is constructed based on the multi-scale feature map to obtain the correlation volume and geometric coding volume.

[0099] In this step, a multi-scale feature map is input to construct a multi-view geometric cost volume, and a correlation volume and a geometric coding volume are obtained. l ,I c and I r Extract the left feature f l,4 , intermediate feature f c,4 and right feature f r,4 Then, the present invention constructs a correlation volume to calculate the feature correlation under different parallaxes. The present invention divides the number of channels of the above feature map into g (g=8) groups, and calculates the correlation volume for each group. The present invention first calculates the correlation between the left and middle views Correlation between the middle and right views As shown below

[0100]

[0101] Among them, <·,·> represents the inner product, d represents the disparity, and c is the number of feature layers. Although the constructed correlation volume can reflect the degree of matching between features, it has limitations in understanding the broad spatial layout and three-dimensional structure of the entire scene. To solve this problem, a geometric encoding volume with global information is obtained through a 3D regularized network R

[0102]

[0103]

[0104] The 3D regularization network here is essentially a 3-layer U-net network. U-Net is a convolutional neural network (CNN). The U-Net network structure is characterized by a U-shaped architecture consisting of two main parts: a downsampling path (encoder) and an upsampling path (decoder).

[0105] The cost volume operation is performed according to the correlation volume and the left view key information to obtain the weighted correlation volume.

[0106] In this step, the obtained correlation volume and the key information of the left view are subjected to the cost volume operation to obtain the weighted correlation volume; when looking for the corresponding point of a point in one image in another image, if each point is compared in turn, the process will become very complicated, especially when there are repeated patterns or lack of texture in the scene. Therefore, the present invention uses the cost volume excitation operation to use the key information provided by the left image to guide where it is more important and where it may be misleading. The same is true for using the middle image to excite the right view, as shown below

[0107] C′ i (c) = σ(f l,i )⊙Ci

[0108] σ represents the sigmoid function, whose value is between (0, 1), and ⊙ is the Hadamard function. With the key weights as a guide, the model can make more reasonable matching decisions by emphasizing or suppressing the contribution of different parts in the cost volume, especially in the boundary and detail areas, reducing edge blur and detail loss caused by over-smoothing.

[0109] The softmax function is used to obtain the initial disparity map through the weighted correlation volume, and the initial disparity map is optimized to obtain the final initial disparity map.

[0110] In this step, the softmax function is used to obtain the initial disparity map through the weighted correlation volume, and the initial disparity map is optimized to obtain the final initial disparity map.

[0111] In order to obtain the initial disparity map, the present invention uses the softmax function to calculate the corresponding costvolume values ​​under different disparities d. The larger the cost volume corresponding to the disparity, the more inaccurate the disparity estimation is, and the smaller the corresponding weight value is. The different disparities are multiplied by their corresponding weights, and then the accumulation operation is performed to obtain the initial disparity d. l and d c ,

[0112]

[0113] d l represents the disparity between the left view and the center view, d c represents the disparity between the center view and the right view, C G1 (d) indicates the corresponding C when the parallax is G1 The smaller the value, the more accurate the disparity estimation is, and the larger the corresponding weight value is. G2 (d) The same applies.

[0114] The above two initial disparity maps are obtained. These two disparity maps are compared pixel by pixel with the disparity map generated by NeRF, and the disparity corresponding to the pixel with the smallest error is taken as the final initial disparity map d0.

[0115] d0=min{|d NeRF -d l |,|d NeRF -d c |}.

[0116] The input geometry encoding volume, correlation volume and final initial disparity map are passed through an iterative updater to obtain an updated disparity map.

[0117] In this step, the input geometry coding volume, correlation volume and final initial disparity map are passed through an iterative updater to obtain an updated disparity map.

[0118] NeRF cannot directly generate a disparity map, but since it contains implicit depth information, the present invention can convert the depth map into the required disparity map. The conversion formula is as follows:

[0119]

[0120] Among them, Z is the depth, b is the camera baseline length, f is the camera focal length, and d is the parallax.

[0121] After obtaining the initial disparity map d0, iterative optimization is required to generate a more accurate disparity map. By combining the Geometry Encoding Volume and all-pairs correlations, the geometric features G are extracted from it. k

[0122]

[0123] Here Concat represents the concatenation of geometric coding volumes C from different disparity offsets along the channel direction. Gf and all pairs of correlation volumes C GA Features of and their pooled versions and Next, G is processed through two encoder layers k With the current disparity d n , and form the input x n , and then update the hidden state h of ConvGRU n-1 , to incrementally correct the disparity estimate. The update formula is as follows:

[0124] x n =[Encoder(G k ),Encoder(d n ),d n ]

[0125] z n =σ(Conv 3×3 ([h n-1 ,x n ],W z )+c k ),

[0126] r n =σ(Conv 3×3 ([h n-1 ,x n ],W r )+c r),

[0127]

[0128] Among them, x n represents the input at time step n, and Encoder() is a function that converts them into a representation suitable for use by a neural network. n 、r n is the state update weight calculated by the convolutional layer using the activation function σ, z n Represents the output value of the forget gate, r n Represents the output value of the reset gate, Conv 3×3 is the convolution operation, h n-1 、h n are the hidden state and the final state, σ and tanh are the activation functions sigmoid and hyperbolic tangent, respectively. W represents the convolution kernel weight, c is the context feature, ⊙ represents the Hadamard product, and Encoder represents the encoder for processing geometric features and disparity. Through the above iterative process, each update is based on the current disparity estimate and comprehensive geometric information, which effectively reduces the uncertainty of local matching and significantly improves the accuracy of the disparity map. Based on the updated hidden state hk, a residual disparity map △d is decoded through two consecutive convolutional layers. n , represents the relative disparity estimate d n The amount that needs to be adjusted, the updated disparity map is:

[0129] d n+1 =d n +△d n .

[0130] Loss function design module:

[0131] In order to ensure that the predicted disparity map is highly consistent with the actual scene depth, while ensuring the sensitivity to details and overall smoothness during the model learning process, the present invention combines photometric loss and disparity loss, which not only considers the accuracy of the initial disparity estimation, but also promotes the fine optimization of the disparity map during the iteration process.

[0132] Photometric loss: In traditional stereo matching tasks, the network estimates the disparity map by comparing the image features between two views. However, when there is occlusion, some areas in one view are not visible in the other view, which makes it difficult for the traditional two-view based photometric loss to provide effective supervision. To solve this problem, triple photometric loss uses the image data of three views (to form a "disparity triplet" (triplet) to enhance the reliability of matching. The formula is as follows

[0133]

[0134] The variable α is a weight parameter used to adjust the impact of the SSIM loss term and the pixel-level loss term on the total loss. When α is close to 0, the restoration process focuses more on maintaining the structural information of the original image; when α is close to 1, it focuses more on the consistency between the restoration result and the input image at the pixel level. c The image representing the intermediate view, I r→c It means matching the right view image to the middle view through parallax. SSIM is an indicator for evaluating the similarity of two images in terms of structural content. Different from the method of only comparing pixel values, SSIM takes into account the brightness, contrast and structural information of the image, so it is closer to the human eye's perception of image quality. In order to compensate for occlusion, the present invention also performs the same operation on the left view, and then selects the smallest loss as the final triple photometric loss.

[0135]

[0136] Disparity loss: Compare the initial disparity map d0 with the disparity map d obtained from NeRF implicit depth information through the L1 paradigm NeRF , get the initial disparity loss

[0137]

[0138] When calculating the total disparity loss, different predicted disparities are assigned different importance weights. The weights in the loss function increase exponentially with the increase of the disparity value. The purpose is to balance the problem that pixels farther away from the center of the image are more difficult to predict.

[0139]

[0140] The final loss is and The sum of , and give different weights to the two, η1, η2 are different weight values:

[0141]

[0142] In view of the above problems, the present invention uses the NeRF (neural radiant field) model to generate high-quality left, center and right view image sets and their corresponding disparity maps from real orchard scene images. These disparity maps are converted from the depth information implicit in NeRF and serve as supervisory signals for the disparity network. Then, a self-supervised disparity network is used to extract multi-scale features from the left, center and right view image sets, and a multi-view geometric cost volume is constructed to calculate the feature correlation under different disparities. In order to improve the accuracy, the cost volume excitation operation is used to optimize the cost volume, reduce edge blur and detail loss, and generate a more accurate disparity map through an iterative updater. In this process, the network also uses the disparity map generated by NeRF for guidance and selects the optimal disparity map. In order to further optimize the network performance, a comprehensive loss function including photometric loss and disparity loss is designed, in which the photometric loss enhances the reliability of matching, and the disparity loss compares the initial disparity map with the disparity map generated by NeRF through the L1 paradigm, and finally the weighted sum of these two losses is obtained to obtain the total loss, thereby guiding the training of the entire network.

[0143] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0144] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

[0145] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for perceiving image depth information of a fruit and vegetable picking robot, characterized in that: The method comprises: Collect multi-view images, extract the color and density of the sampling points based on a multi-layer perceptron, and generate the left view, middle view, right view and disparity map by rendering the 3D scene; Extract features from the left view and the middle view to obtain a multi-scale feature map and a multi-scale context feature map; Construct a multi-view geometric cost volume based on the multi-scale feature map to obtain the correlation volume and geometric coding volume; Perform cost volume operation according to the correlation volume and left view key information to obtain a weighted correlation volume; The softmax function is used to obtain the initial disparity map through the weighted correlation volume, and the initial disparity map is optimized to obtain the final initial disparity map; The input geometry encoding volume, correlation volume and final initial disparity map are passed through an iterative updater to obtain an updated disparity map.

2. The image depth information perception method for a fruit and vegetable picking robot according to claim 1, characterized in that: The steps of collecting multi-view images, extracting the color and density of sampling points based on a multi-layer perceptron, and generating a left view, a middle view, a right view, and a disparity map by rendering a three-dimensional scene include: Collect a multi-view image set, perform spatial point sampling and position encoding, and record the camera pose corresponding to each set of images, wherein the camera pose includes a spatial position and an observation direction; The camera pose is used as the input of the multi-layer perceptron, and a nonlinear transformation is performed through the L-layer multi-layer perceptron to obtain the color and density of each sampling point; The color and density of the sampling points are integrated into the color of each pixel on the ray through volume rendering, and a three-dimensional scene is rendered to generate a left view, a middle view, and a right view; In the process of rendering a 3D scene, a disparity map is extracted based on the implicit depth map; The steps of spatial point sampling and position encoding include encoding the input coordinates P through sine and cosine functions. Position encoding maps 3D coordinates to high-dimensional space vector representation. The position encoding E(P) is expressed as: γ(p)=(sin(2 0 πp),cos(2 0 πp),sin(2 1 πp),cos(2 1 πp),...,sin(2 L-1 πp),cos(2 L-1 πp)) Where N represents the number of frequencies and P represents the 3D coordinates.

3. The image depth information perception method for a fruit and vegetable picking robot according to claim 2, characterized in that: The camera posture is used as the input of the multi-layer perceptron, and a nonlinear transformation is performed through the L-layer multi-layer perceptron to obtain the color and density of each sampling point, including: using the encoded spatial position and observation direction as the input X0, and a nonlinear transformation is performed through the L-layer multi-layer perceptron to obtain the color and density of each sampling point. The calculation formula of each layer is: x l =σ(W l x l-1 +b l ) Among them, σ is the activation function, W and b are the weight matrix and bias vector. Predict the color and density of each sample point through a multi-layer perceptron: (c i ,s i )=MLP(γ(x i ),γ(d)) Among them, c i is the color of the sampling point, σi is the density, x i is the sampling point and d is the direction.

4. The image depth information perception method for a fruit and vegetable picking robot according to claim 2, characterized in that: The color and density of the sampling points are integrated into the color of each pixel on the ray through volume rendering, and the three-dimensional scene is rendered to generate the steps of left view, middle view and right view, including: defining the camera parameters of the left viewpoint, middle viewpoint and right viewpoint, the camera external parameters of the middle view are Ek=Rk|tk, Rk represents the rotation matrix, which is a 3×3 unit matrix, tk represents the translation matrix, and the external parameters of the right view camera are: The images corresponding to the three viewpoints Ek, EkR and EkL form a stereo triplet, namely the left view, the middle view and the right view, and the generated dataset is used as a training set to train the disparity network.

5. The image depth information perception method for a fruit and vegetable picking robot according to claim 2, characterized in that: In the process of rendering a three-dimensional scene, the step of extracting a disparity map according to an implicit depth map specifically includes: extracting disparity from the implicit depth map in the rendering process to obtain a disparity map, and aligning the central image of the triplet. The rendered depth map represents the distance from the camera to each pixel in the scene. The disparity map refers to the disparity of the corresponding pixel points in two images of different viewing angles after stereo correction. The disparity map can be converted from the depth map and is expressed as: Where b is the baseline distance, f is the focal length of the camera, and z is the depth.

6. The image depth information perception method for a fruit and vegetable picking robot according to claim 1, characterized in that: The step of extracting features from the left view and the middle view to obtain a multi-scale feature map and a multi-scale context feature map comprises: extracting features using a feature extractor and a context extractor to obtain a multi-scale feature map and a multi-scale context feature map respectively, wherein the feature extractor: given a left view image I l and the intermediate view image I c , with a size of 3×H×W. The image is reduced to 1 / 32 of the original size on the pre-trained MobileNetV2 model, and the image is restored to 1 / 4 of the original size by upsampling to obtain feature maps of different sizes. The context extractor consists of a residual block and a downsampling layer. The residual block allows the gradient to propagate in the network. The resolution of the feature map is gradually reduced during the downsampling process to provide the context features required for time series processing.

7. The image depth information perception method for a fruit and vegetable picking robot according to claim 1, characterized in that: The steps of constructing a multi-view geometric cost volume based on the multi-scale feature map and obtaining the correlation volume and the geometric coding volume include: l , Middle View I c and right view I r Extract respectively, the left feature f l,4 , intermediate feature f c,4 and right feature f r,4 , divide the number of channels of the feature map into g groups, calculate the relevant volume for each group, and calculate the correlation between the left and middle views Correlation between the middle and right views Among them, <·,·> represents the inner product, d represents the disparity, c is the number of feature layers, and the geometric encoding volume with global information is obtained through the 3D regularization network R 8. The image depth information perception method for a fruit and vegetable picking robot according to claim 1, characterized in that: The steps of obtaining an initial disparity map by using a softmax function through a weighted correlation volume, optimizing the initial disparity map, and obtaining a final initial disparity map include: Use the softmax function to calculate the corresponding cost volume value under different disparities d. The larger the cost volume corresponding to the disparity, the smaller the corresponding weight value. Multiply the different disparities with their corresponding weights and perform cumulative operations to obtain the initial disparity d. l and d c , d l represents the disparity between the left view and the center view, d c represents the disparity between the center view and the right view, C G1 (d) indicates the corresponding C when the parallax is d. G1 The value of The initial disparity is compared with the disparity d generated by NeRF NeRF Perform pixel-by-pixel comparison and take the disparity corresponding to the pixel with the smallest error as the final initial disparity map d0: d0=min{|d NeRF -d l |,|d NeRF -d c |}.

9. The image depth information perception method for a fruit and vegetable picking robot according to claim 1, characterized in that: The step of inputting the geometric coding volume, the correlation volume and the final initial disparity map through the iterative updater to obtain the updated disparity map specifically includes: converting the depth map into the required disparity map, and the conversion formula is: Where Z is the depth, b is the camera baseline length, f is the camera focal length, and d is the parallax; After obtaining the initial disparity map d0, extract the geometric features G k : Concat represents the concatenation of geometric coding volumes C from different disparity offsets along the channel direction. Gf and all pairs of correlation volumes C GA Features of and their pooled versions and r represents the maximum step size of the disparity search range, i.e., the maximum disparity value, and m represents the number of operations performed on each disparity step size or the length of the sliding window. G is processed through two encoder layers. k With the current disparity d n , and form the input x n , update the hidden state h of ConvGRU n-1 , to incrementally correct the disparity estimate; The updated disparity map is: d n+1 =d n +△d n 。 10. The image depth information perception method for a fruit and vegetable picking robot according to claim 1, characterized in that: The method includes using triple photometric loss as a loss function, and the formula is: I c The image representing the intermediate view, I r→c It means matching the right view image to the middle view through parallax. SSIM is an indicator for evaluating the similarity of two images in terms of structure and content. The variable α is a weight parameter.

Images