A speckle depth estimation method, system, electronic device and storage medium

Abstract

The present invention provides a speckle depth estimation method, system, electronic device and storage medium. The method includes collecting a plurality of template images with speckles, training sample images and environmental sample images without speckles; obtaining first depth data of the training sample images and normalizing it to obtain normalized depth data, and obtaining edge data of the environmental sample images and normalizing it to obtain normalized edge supervision data; inputting the template images, training sample images and environmental sample images into a training model to output estimated depth data, calculating a loss value of the training model based on the three depth data and continuously optimizing the training model to obtain a target model that is fully converged; and performing inference on the to-be-tested speckle image and the to-be-tested speckle-free image through the target model to obtain predicted depth information. Through the present application, the efficiency of reconstructing depth information through speckles can be effectively improved while ensuring depth accuracy, so that the speckle depth camera can be effectively applied in scenarios requiring high frame rates.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and particularly relates to a speckle depth estimation method, system, electronic device and storage medium. Background Art

[0002] Currently, the mainstream 3D cameras can be divided into two types: one is the 3D vision sensors represented by laser sensors, which are expensive and have good depth data quality, such as SICK, Enshape, and Ensenso. Their common feature is high precision, but they are slow and expensive, and are not suitable for large-scale promotion in the industrial field; the other is a series of consumer-grade RGB-D cameras, whose principles mainly include structured light, ToF, and binocular stereo vision, etc. Representative products include Microsoft's Kinect and Intel's Realsense. Their common feature is relatively low precision, but they are fast and relatively inexpensive. Consumer-grade RGB-D cameras are more suitable for industrial and practical applications and promotions due to their advantages in efficiency and price.

[0003] The consumer-grade RGB-D cameras sold on the market can generally reach the millimeter level in depth accuracy and can meet the needs of most production and life, but the quality of the depth data is often unsatisfactory and has a great impact on subsequent applications. Currently, the depth data of the speckle images of RGB-D cameras is mainly obtained by means of binocular depth estimation, speckle depth estimation, and TOF depth estimation. Among them, speckle depth estimation is mainly based on the principle of triangulation ranging, and the final depth value is obtained by matching the local small areas of the real-time speckle and the template speckle; however, this method has a large amount of calculation, resulting in a difficult improvement in the frame rate, making it difficult to meet the high frame rate requirements of VR / AR products, and is susceptible to environmental factors. There are also some who use the monocular depth estimation method based on deep learning to estimate the scene depth, but monocular depth is seriously affected by factors such as light, weather, and object color, and it is difficult to use in actual complex environments.

[0004] Therefore, how to optimize the deep learning algorithm of speckle images to effectively improve the efficiency of reconstructing depth information through speckles while ensuring depth accuracy, so that the speckle depth camera can be effectively applied in occasions with high frame rate requirements. Summary of the Invention

[0005] To solve the above technical problems, the present invention provides a speckle depth estimation method, system, electronic device and storage medium, which can effectively improve the efficiency of reconstructing depth information through speckles while ensuring accuracy, so that the speckle depth camera can be effectively applied in occasions with high frame rate requirements.

[0006] In a first aspect, the present application provides a speckle depth estimation method, which includes:

[0007] Collecting a number of template images, training sample images, and environment sample images; wherein the template image refers to a planar image with a white background and speckled at preset intervals within a depth reconstruction range; the training sample image refers to a speckle image collected in different environments within the same depth reconstruction range; and the environment sample image refers to a non-speckled image with the same environment and collection location as the training sample image;

[0008] Obtaining first depth data of the training sample image according to the template image, the training sample image, and a speckle template matching algorithm, and normalizing the first depth data to obtain normalized depth data;

[0009] Obtaining edge data of the environment sample graph according to the environment sample graph and the edge detection algorithm, and normalizing the edge data to obtain normalized edge supervision data;

[0010] Inputting the template image, the training sample image, and the environment sample image into a training model to output estimated depth data;

[0011] Calculating a loss value of the training model based on the normalized depth data, the normalized edge supervision data, and the estimated depth data, and continuously optimizing the training model according to the loss value to obtain a fully converged target model;

[0012] Normalized predicted depth data is obtained by inferring the speckle pattern to be measured and the speckle-free pattern to be measured using the target model, and actual estimated depth is calculated based on the normalized predicted depth data, so as to output predicted depth information of the speckle pattern to be measured and the speckle-free pattern to be measured in real time.

[0013] Preferably, the step of obtaining first depth data of the training sample image according to the template image, the training sample image, and the speckle template matching algorithm, and normalizing the first depth data to obtain normalized depth data specifically includes:

[0014] Using the template image as a matching template, obtaining first depth data of the training sample image through a speckle template matching algorithm;

[0015] Setting the depth data smaller than the minimum measurement range of the speckle camera in the first depth data to the minimum value, and setting the depth data larger than the maximum measurement range of the speckle camera to the maximum value, to obtain second depth data;

[0016] The second depth data is normalized to obtain normalized depth data; wherein the normalization formula is dep(x,y)=(depth(x,y)-d min ) / (d max -d min), where (x, y) represents the pixel position, depth(x, y) represents the edge data at the position (x, y), and d max is the depth data of the maximum measurement range of the speckle camera, and d min is the depth data of the minimum measurement range of the speckle camera, and dep(x, y) represents the normalized depth data at the position (x, y).

[0017] Preferably, the specific steps of obtaining the edge data of the environmental sample map according to the environmental sample map and the edge detection algorithm, and normalizing the edge data to obtain the normalized edge supervision data include:

[0018] Obtain the edge data of the environmental sample map through the edge detection algorithm;

[0019] Perform normalization processing on the edge data to obtain the normalized edge supervision data; where the formula for the normalization processing is edg(x, y) = (edge(x, y) - min(edge)) / (max(edge) - min(edge)), where (x, y) represents the pixel position, edge(x, y) represents the edge data at the position (x, y), min(edge) represents the minimum value in the edge data, and max(edge) represents the maximum value in the edge data.

[0020] Preferably, the steps of calculating the loss value of the training model based on the normalized depth data, the normalized edge supervision data, and the estimated depth data, and continuously optimizing the training model according to the loss value to obtain the target model with complete convergence processing specifically include:

[0021] Preprocess the template map, the training sample map, and the environmental sample map to filter out the environmental data in the template map, the training sample map, and the environmental sample map;

[0022] Merge and train the results of the preprocessed template map, training sample map, and environmental sample map to obtain S input samples;

[0023] Define each of the S input samples as a hyperparameter batch in the training model with a predetermined number of input samples, obtain a plurality of hyperparameter data, and input the plurality of hyperparameter data into the training model;

[0024] Take batch * h * w * (M + 2) as the input data dimension for training the training model, and take (2 * batch * h * w * 1, batch * h / 2 * w / 2 * 1, batch * h / 4 * w / 4 * 1) as the output data dimension for training the training model, and output the estimated depth data through the training model; where As the predicted edge estimate with an output dimension of batch*h*w*1; where h*w represents the resolution of the template image, the training sample image, and the environmental sample image, and M represents the number of acquisitions of the template image;

[0025] Calculate the loss value of the training model through a loss function based on the estimated depth data;

[0026] Perform backpropagation according to the loss value, and continuously optimize the parameters of the training model until it converges completely to obtain the target model.

[0027] Preferably, the step of preprocessing the template image, the training sample image, and the environmental sample image to filter out the environmental data in the template image, the training sample image, and the environmental sample image specifically includes:

[0028] Perform filtering and denoising on the training sample image to filter out salt-and-pepper noise and Gaussian noise in the training sample image; where the filtering and denoising include median filtering and Gaussian filtering;

[0029] Perform local preprocessing on the template image and the training sample image that has been filtered and denoised; where the local preprocessing specifically includes:

[0030] Pad the image I of the template image and the training sample image with a resolution of h*w with (n - 1) / 2 pixels of pixel value 0 around the perimeter to obtain F1 with a resolution of (h + n - 1)*(w + n - 1), and perform a two-dimensional convolution with a window size of n*n, a step size of 1, and all convolution kernel weights of 1 on F1 to obtain the feature matrix f avg , and obtain the mean feature matrix avg through the mean formula; where n represents the number of pixels, and the mean formula is:

[0031]

[0032] Based on the relationship of F2 = F1 2 obtain F2, and perform a two-dimensional convolution with a window size of n*n, a step size of 1, and all convolution kernel weights of 1 on F2 to obtain the feature matrix f std , and obtain the standard deviation matrix std through the standard deviation formula; where the standard deviation formula is:

[0033]

[0034] Based on the mean feature matrix avg and the standard deviation matrix std, filter out the environmental speckle matrix through the filtering formula to filter out the environmental data, and obtain the images of the template image and the training sample image after filtering out the environmental interference; where the filtering formula is:

[0035]

[0036] In the formula, (x, y) represents the pixel position, I(x, y) represents the pixel value at the position (x, y), avg(x, y) represents the pixel mean value at the position (x, y), and std(x, y) represents the covariance value at the position (x, y).

[0037] Preferably, the loss function is:

[0038]

[0039] In the formula, (x, y) represents the pixel position, h*w represents the resolution of the template image, the training sample image, and the environmental sample image. represents the estimated depth data of the pixel position (x, y), dep k (x, y) represents the depth data obtained by reducing the normalized depth data of the pixel position (x, y) by a multiple of k, and the value of K is 1, 2, or 4. represents the predicted edge estimation of the pixel position (x, y), and edge(x, y) represents the normalized edge supervision data at the position (x, y).

[0040] Preferably, the step of using the target model to perform inference on the to-be-detected speckle pattern and the to-be-detected speckle-free pattern to obtain the normalized predicted depth data, and calculating the actual estimated depth based on the normalized predicted depth data to output the predicted depth information of the to-be-detected speckle pattern in real time specifically includes:

[0041] The image f of the to-be-detected speckle pattern continuously collected from the speckle camera r and the image a of the to-be-detected speckle-free pattern r , through the processing of the preprocessing, the image f is obtained r The image after filtering out environmental interference and the image a r The image after filtering out environmental interference And splicing the image after filtering out environmental interference of the template image to obtain the target input sample, and inputting the target input sample into the target model for inference to obtain the normalized predicted depth data;

[0042] Based on the normalized predicted depth data, calculate the actual estimated depth by using a preset formula to output the predicted depth information of the to-be-detected speckle pattern in real time; wherein, the preset formula is:

[0043]

[0044] In the formula, d is the actual estimated depth, is the normalized predicted depth data, dmax The depth data d is the maximum measurement range of the speckle camera. min The depth data is the minimum measurement range of the speckle camera.

[0045] In a second aspect, the present application provides a speckle depth estimation system, including:

[0046] An acquisition module for acquiring a plurality of template images, training sample images, and environmental sample images; wherein, the template image is a white background planar image with speckles acquired at preset intervals within the depth reconstruction range, the training sample image is a speckle image acquired in different environments within the same depth reconstruction range, and the environmental sample image is a non-speckle image with the same environment and acquisition position as the training sample image;

[0047] A first processing module for obtaining first depth data of the training sample image according to the template image, the training sample image, and a speckle template matching algorithm, and normalizing the first depth data to obtain normalized depth data;

[0048] A second processing module for obtaining edge data of the environmental sample image according to the environmental sample image and an edge detection algorithm, and normalizing the edge data to obtain normalized edge supervision data;

[0049] A training module for inputting the template image, the training sample image, and the environmental sample image into a training model to output estimated depth data;

[0050] An iterative optimization module for calculating a loss value of the training model based on the normalized depth data, the normalized edge supervision data, and the estimated depth data, and continuously optimizing the training model according to the loss value to obtain a target model with complete convergence processing;

[0051] An operation module for inferring normalized predicted depth data for a to-be-measured speckle image and a to-be-measured non-speckle image through the target model, and calculating an actual estimated depth based on the normalized predicted depth data to output prediction depth information of the to-be-measured speckle image and the to-be-measured non-speckle image in real time.

[0052] Preferably, the first processing module includes:

[0053] A matching algorithm unit that uses the template image as a matching template to obtain first depth data of the training sample image through a speckle template matching algorithm;

[0054] A setting unit that sets the depth data in the first depth data that is less than the minimum measurement range of the speckle camera to the minimum value, and sets the depth data that is greater than the maximum measurement range of the speckle camera to the maximum value to obtain second depth data;

[0055] The first normalization unit normalizes the second depth data to obtain normalized depth data; wherein, the formula for the normalization process is dep(x,y)=(depth(x,y)-d min ) / (d max -d min ), in the formula, (x,y) represents the pixel position, depth(x,y) represents the edge data at the position (x,y), d max is the depth data of the maximum measurement range of the speckle camera, d min is the depth data of the minimum measurement range of the speckle camera, and dep(x,y) represents the normalized depth data at the position (x,y).

[0056] Preferably, the second processing module includes:

[0057] An edge algorithm unit for obtaining the edge data of the environmental sample map through an edge detection algorithm;

[0058] A second normalization unit for normalizing the edge data to obtain normalized edge supervision data; wherein, the formula for the normalization process is edg(x,y)=(edge(x,y)-min(edge)) / (max(edge)-min(edge)), in the formula, (x,y) represents the pixel position, edge(x,y) represents the edge data at the position (x,y), min(edge) represents the minimum value in the edge data, and max(edge) represents the maximum value in the edge data.

[0059] Preferably, the iterative optimization module includes:

[0060] A preprocessing unit for preprocessing the template map, the training sample map, and the environmental sample map to filter out the environmental data in the template map, the training sample map, and the environmental sample map;

[0061] A combined training unit for combining and training the results of the preprocessed template map, the training sample map, and the environmental sample map to obtain S input samples;

[0062] A definition unit for defining each of the S input samples as a hyperparameter batch in the training model with a predetermined number of input samples, obtaining a plurality of hyperparameter data, and inputting the plurality of hyperparameter data into the training model;

[0063] A regulation unit, which is used to take batch*h*w*(M + 2) as the input data dimension for training the training model, and take (2*batch*h*w*1, batch*h / 2*w / 2*1, batch*h / 4*w / 4*1) as the output data dimension for training the training model, and output estimated depth data through the training model; where, as the predicted edge estimation value with an output dimension of batch*h*w*1; in the formula, h*w represents the resolution of the template graph, the training sample graph, and the environmental sample graph, and M represents the number of acquisitions of the template graph;

[0064] A calculation unit, which is used to calculate the loss value of the training model through a loss function based on the estimated depth data;

[0065] An iterative optimization unit, which performs backpropagation according to the loss value and continuously optimizes the parameters of the training model until it converges completely to obtain a target model.

[0066] Preferably, the preprocessing unit is specifically used for:

[0067] Performing filtering and denoising on the training sample graph to filter out salt-and-pepper noise and Gaussian noise in the training sample graph; where the filtering and denoising include median filtering and Gaussian filtering;

[0068] Performing local preprocessing on the template graph and the training sample graph after filtering and denoising; where the local preprocessing specifically includes:

[0069] Padding (n - 1) / 2 pixels with pixel value 0 around the image I of the template graph and the training sample graph with a resolution of h*w to obtain F1 with a resolution of (h + n - 1)*(w + n - 1), and performing a two-dimensional convolution with a window size of n*n, a step size of 1, and all convolution kernel weights of 1 on F1 to obtain a feature matrix f avg , and obtaining a mean feature matrix avg through the mean formula; where n represents the number of pixels, and the mean formula is:

[0070]

[0071] Obtaining F2 based on the relationship of F2 = F1 2 , and performing a two-dimensional convolution with a window size of n*n, a step size of 1, and all convolution kernel weights of 1 on F2 to obtain a feature matrix f std , and obtaining a standard deviation matrix std through the standard deviation formula; where the standard deviation formula is:

[0072]

[0073] Based on the mean feature matrix avg and the standard deviation matrix std, the environmental speckle matrix is removed through a filtering formula to filter out environmental data, and the images of the template map and the training sample map after filtering out environmental interference are obtained; wherein, the filtering formula is:

[0074]

[0075] In the formula, (x, y) represents the pixel position, represents the pixel value at the position (x, y) after filtering out environmental interference, I(x, y) represents the pixel value at the position (x, y), avg(x, y) represents the pixel mean value at the position (x, y), and std(x, y) represents the covariance value at the position (x, y).

[0076] Preferably, the operation module includes:

[0077] An inference unit, configured to continuously acquire the image f of the speckle map to be measured and the image a of the speckle-free map to be measured from the speckle camera, r and obtain the image f r after filtering out environmental interference through the processing of the preprocessing, r and the image a after filtering out environmental interference, r and splice the images of the template map after filtering out environmental interference to obtain a target input sample, and input the target input sample into the target model for inference to obtain normalized predicted depth data; and splice the images of the template map after filtering out environmental interference to obtain a target input sample, and input the target input sample into the target model for inference to obtain normalized predicted depth data;

[0078] A real-time operation unit, configured to calculate an actual estimated depth based on the normalized predicted depth data by using a preset formula, so as to output the predicted depth information of the speckle map to be measured in real time; wherein, the preset formula is:

[0079]

[0080] In the formula, d is the actual estimated depth, is the normalized predicted depth data, d max is the depth data of the maximum measurement range of the speckle camera, d min is the depth data of the minimum measurement range of the speckle camera.

[0081] In a third aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where when the processor executes the computer program, the speckle depth estimation method described in the first aspect above is implemented.

[0082] Fourthly, an embodiment of the present application provides a storage medium, on which a computer program is stored, and when the program is executed by a processor, the speckle depth estimation method described in the first aspect above is implemented.

[0083] Compared with the prior art, a speckle depth estimation method, system, electronic device and storage medium provided by the present application normalize a plurality of template images and training sample images with speckles to obtain normalized depth data; normalize a plurality of environmental sample images without speckles to obtain normalized edge supervision data; input the collected template images, training sample images and environmental sample images into a training model to obtain predicted depth data; calculate the loss value of the training model based on the normalized depth data, normalized edge supervision data and predicted depth data, and continuously optimize the training model through the loss value to obtain a target model with complete convergence processing; then, perform inference on the to-be-tested speckle image through the target model to obtain normalized predicted depth data, and calculate the actual estimated depth based on the normalized predicted depth data, so as to achieve the purpose of real-time output of the predicted depth information of the to-be-tested speckle image. The present application collects speckle images as training data through a speckle RGB-D camera. Since the depth is only related to the speckle offset and has nothing to do with the specific environment, relatively few data samples are required for training; moreover, a deep learning algorithm is used instead of template matching in the speckle RGB-D camera, which can improve the frame rate of the RGB-D camera without reducing the accuracy, reduce the dizziness feeling of using the camera in VR / AR, and can be effectively applied in occasions requiring a high frame rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention, and those of ordinary skill in the art can also obtain other drawings without creative efforts based on these drawings.

[0085] Figure 1 It is a flowchart of the speckle depth estimation method provided in Embodiment 1 of the present invention;

[0086] Figure 2 It is a specific flowchart of step S102 of the speckle depth estimation method provided in Embodiment 1 of the present invention;

[0087] Figure 3 It is a specific flowchart of step S103 of the speckle depth estimation method provided in Embodiment 1 of the present invention;

[0088] Figure 4 It is a specific flowchart of step S105 of the speckle depth estimation method provided in Embodiment 1 of the present invention;

[0089] Figure 5It is the specific flowchart of step S106 of the speckle depth estimation method provided in Embodiment 1 of the present invention;

[0090] Figure 6 It is the structural block diagram of the speckle depth estimation corresponding to the method of Embodiment 1 provided in Embodiment 2 of the present invention;

[0091] Figure 7 It is the schematic hardware structure diagram of the electronic device provided in Embodiment 3 of the present invention.

[0092] Explanation of reference numerals:

[0093] 10 - Acquisition module;

[0094] 20 - First processing module, 21 - Matching algorithm unit, 22 - Fixing unit, 23 - First normalization unit;

[0095] 30 - Second processing module, 31 - Edge algorithm unit, 32 - Second normalization unit;

[0096] 40 - Training module;

[0097] 50 - Iterative optimization module, 51 - Preprocessing unit, 52 - Merged training unit, 53 - Definition unit, 54 - Regulation unit, 55 - Calculation unit, 56 - Iterative optimization unit;

[0098] 60 - Operation module, 61 - Reasoning unit, 62 - Real - time operation unit;

[0099] 70 - Bus, 71 - Processor, 72 - Memory, 73 - Communication interface. Detailed implementation manners

[0100] Now, the exemplary embodiments will be described more comprehensively with reference to the accompanying drawings. However, the exemplary embodiments can be implemented in various forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be more thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art.

[0101] In addition, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of the embodiments of the present disclosure. However, those skilled in the art will realize that the technical solutions of the present disclosure can be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, well - known methods, devices, implementations, or operations are not shown or described in detail to avoid obscuring aspects of the present disclosure.

[0102] The block diagrams shown in the drawings are only functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0103] The flowcharts shown in the drawings are only illustrative and do not necessarily include all contents and operations / steps, nor are they necessarily executed in the described order. For example, some operations / steps can be decomposed, while some operations / steps can be combined or partially combined. Therefore, the actual execution order may change according to the actual situation.

[0104] Embodiment 1

[0105] Specifically, Figure 1 The figure shows a schematic flowchart of a speckle depth estimation method provided in this embodiment.

[0106] As Figure 1 shown, the speckle depth estimation method of this embodiment includes the following steps:

[0107] S101, collect a plurality of template images, training sample images, and environmental sample images.

[0108] Among them, the template image refers to a white-background planar image with speckles collected at preset intervals within the depth reconstruction range. The training sample image refers to a speckle-bearing image collected in different environments within the same depth reconstruction range. The environmental sample image refers to a non-speckle-bearing image with the same environment and collection position as the training sample image.

[0109] Specifically, assume that the number of collected template images is m frames, that is, pattern M , where M = 1, 2, 3,..., m (note: generally, the number of template images is small, 3 - 10 frames). In this embodiment, the designed depth reconstruction range is 0.3m - 1.2m. The speckle RGB-D camera is aligned with the white wall, and the camera lens is perpendicular to the wall surface of the white wall. One frame of template image is collected at distances of 0.6m, 0.8m, and 1.0m within the construction depth range, and a total of 3 frames of template images are collected. The training sample images and environmental sample images are collected in different environments within the reconstruction range of 0.3m - 1.2m. The speckle-bearing sample image im S and the non-speckle-bearing sample image amb S, where S = 1, 2, 3, … s, and a total of 10,000 images are collected in this embodiment. It should be noted that generally, the more training sample images, the better, which can increase the diversity of training samples. However, a larger number of samples leads to a longer training time and more difficult data collection. Therefore, the number of training sample images needs to be determined according to specific circumstances. It should be noted that im S , amb S is one-to-one. For example, a frame of speckle-containing environmental image im i is collected, and the corresponding speckle-free amb i image is collected simultaneously while keeping the camera position and environment unchanged.

[0110] S102: Obtain the first depth data of the training sample image according to the template image, the training sample image, and the speckle template matching algorithm, and normalize the first depth data to obtain normalized depth data.

[0111] Among them, the data normalization problem is an important issue in the expression of feature vectors in data mining. When different features are listed together, due to the expression method of the features themselves, small data in absolute values may be "eaten" by large data. At this time, what needs to be done is to normalize the extracted features vector to ensure that each feature is treated equally by the classifier.

[0112] Further, as Figure 2 shown, the specific steps of step S102 include:

[0113] S1021: Use the template image as the matching template and obtain the first depth data of the training sample image through the speckle template matching algorithm.

[0114] Specifically, the template image in this embodiment is a known small image. Template matching is to extract several feature vectors from the image to be recognized and compare them with the corresponding feature vectors of the template, calculate the distance between the image and the template feature vectors, and determine the category by the minimum distance method. In short, template matching is to search for a target in a large image. It is known that there is a target to be found in the image, and the target has the same size, direction, and image as the template. Through a certain algorithm, the target can be found in the image and its coordinate position can be determined.

[0115] S1022: Set the depth data in the first depth data that is less than the minimum measurement range of the speckle camera to the minimum value, and set the depth data that is greater than the maximum measurement range of the speckle camera to the maximum value to obtain the second depth data.

[0116] S1023: Normalize the second depth data to obtain the normalized depth data.

[0117] Specifically, the formula for the normalization process is dep(x,y)=(depth(x,y)-d min ) / (d max -d min ), where (x,y) represents the pixel position, depth(x,y) represents the second depth data at the position (x,y), d max is the depth data of the maximum measurement range of the speckle camera, and d min is the depth data of the minimum measurement range of the speckle camera. dep(x,y) represents the normalized depth data at the position (x,y).

[0118] S103. Obtain the edge data of the environmental sample map according to the environmental sample map and the edge detection algorithm, and normalize the edge data to obtain the normalized edge supervision data.

[0119] Further, as Figure 3 shown, the specific steps of step S103 include:

[0120] S1031. Obtain the edge data of the environmental sample map through the edge detection algorithm.

[0121] Among them, the edge detection algorithm uses Sobel or canny. The Sobel operator is a discrete differentiation operator; it combines Gaussian smoothing and differential differentiation to calculate the approximate gradient of the image grayscale function; the pixel values of the image edges will change significantly, and one way to represent this change is to use derivatives; a large change in the gradient value indicates a significant change in the content of the image. The goal of Canny is to find an optimal edge detection algorithm. The meaning of optimal edge detection is that the algorithm can identify as many actual edges in the image as possible, the identified edges should be as close as possible to the actual edges in the actual image, the edges in the image can only be identified once, and the possible image noise should not be identified as an edge; Canny uses the calculus of variations, which is a method of finding a function that satisfies a specific function and is very similar to the first derivative of the Gaussian function. In this embodiment, the Sobel edge detection algorithm is adopted.

[0122] S1032. Normalize the edge data to obtain the normalized edge supervision data.

[0123] Among them, the formula for the normalization process is edg(x, y) = (edge(x, y) - min(edge)) / (max(edge) - min(edge)), where (x, y) represents the pixel position, edge(x, y) represents the edge data at the position (x, y), min(edge) represents the minimum value in the edge data, and max(edge) represents the maximum value in the edge data.

[0124] S104, input the template graph, the training sample graph, and the environmental sample graph into the training model to output the estimated depth data.

[0125] Among them, deep learning forms more abstract high-level representations of attribute categories or features by combining low-level features to discover the distributed feature representations of data. The training model in this embodiment is a deep learning neural network model, which calculates the solutions it needs by learning from the observed data.

[0126] S105, calculate the loss value of the training model based on the normalized depth data, the normalized edge supervision data, and the estimated depth data, and continuously optimize the training model according to the loss value to obtain a target model with complete convergence processing.

[0127] Further, as Figure 4 shown, the specific steps of step S105 include:

[0128] S1051, preprocess the template graph, the training sample graph, and the environmental sample graph to filter out the environmental data in the template graph, the training sample graph, and the environmental sample graph.

[0129] Specifically, step S1051 specifically includes:

[0130] S10511, perform filtering and denoising on the training sample graph to filter out the salt-and-pepper noise and Gaussian noise in the training sample graph;

[0131] Among them, salt-and-pepper noise is often caused by image cutting. The most commonly used algorithm for removing impulse interference and salt-and-pepper noise is median filtering; Gaussian noise is a kind of random noise, which is completely determined by its time-varying average value and the covariance function of two instants. If the noise is stationary, the average value is independent of time, and the covariance function becomes a correlation function that only depends on the difference between the two instants considered.

[0132] Specifically, the filtering and denoising includes median filtering and Gaussian filtering; median filtering is a non-linear signal processing technology based on the sorting statistics theory that can effectively suppress noise. The basic principle of median filtering is to replace the value of a point in a digital image or digital sequence with the median value of the points in a neighborhood of that point, making the surrounding pixel values closer to the true values and thus eliminating isolated noise points. Gaussian filtering is a process of weighted averaging the entire image, and the value of each pixel point is obtained by weighted averaging itself and other pixel values in its neighborhood.

[0133] S10512, perform local preprocessing on the template image and the training sample image that has undergone filtering and denoising processing;

[0134] Among them, the local preprocessing specifically includes:

[0135] 1. Pad (n - 1) / 2 pixels with pixel value 0 around the image I of the template image and the training sample image with a resolution of h*w to obtain F1 with a resolution of (h + n - 1)*(w + n - 1), and perform a two-dimensional convolution with a window size of n*n, a step size of 1, and all convolution kernel weights being 1 based on F1 to obtain the feature matrix f avg , and obtain the mean feature matrix avg through the mean formula; where n represents the number of pixels, and the mean formula is:

[0136]

[0137] Specifically, in this embodiment, the resolutions of the template image and the training sample image are 7*7, that is, the above h = 7, w = 7; use a window of n = 3 for convolution, and the resolution becomes 9*9 after padding; perform a convolution operation with a kernel of 3*3 after padding, and the resolution after convolution is still 7*7.

[0138] 2. Obtain F2 based on the relationship of F2 = F1 2 , and perform a two-dimensional convolution with a window size of n*n, a step size of 1, and all convolution kernel weights being 1 based on F2 to obtain the feature matrix f std , and obtain the standard deviation matrix std through the standard deviation formula; where the standard deviation formula is:

[0139]

[0140] Specifically, the convolutional layer network is local feature extraction based on two-dimensional convolutional neural network operations. Its input is a pair of speckle images (left speckle image and right speckle image), and the output is several first feature images corresponding to each speckle image. The convolutional layer (Convolution Layer) is used for feature extraction of the input data. It contains multiple convolutional kernels inside, and each element that makes up the convolutional kernel corresponds to a weight coefficient and a bias amount.

[0141] 3. Based on the mean feature matrix avg and the standard deviation matrix std, the environmental speckle matrix is removed through a filtering formula to filter out environmental data, and the images of the template map and the training sample map after filtering out environmental interference are obtained; wherein, the filtering formula is:

[0142]

[0143] In the formula, (x, y) represents the pixel position, represents the pixel value at the position (x, y) after filtering out environmental interference, I(x, y) represents the pixel value at the position (x, y), avg(x, y) represents the pixel mean at the position (x, y), and std(x, y) represents the covariance value at the position (x, y).

[0144] The results of the pre-processed template map, the training sample map and the environmental sample map are combined and trained to obtain S input samples;

[0145] S1052. The results of the pre-processed template map, the training sample map and the environmental sample map are combined and trained to obtain S input samples.

[0146] Among them, through step S1052, the template map pattern M After pre-processing, p1, p2,..., p m are obtained. The training sample map im S After pre-processing, and the environmental sample map amb S After pre-processing, edg1, edg2,..., edg s are obtained. The combined training image, that is, (im1, edg1, p1, p2,..., p m ) is the first training sample, (im i , edg i , p1, p2,..., p m ) is the i-th training sample, and so on. In this way, S training samples can be obtained.

[0147] S1053. Each of the S input samples is defined as a hyperparameter batch in the training model with a predetermined number of input samples, multiple hyperparameter data are obtained, and the multiple hyperparameter data are input into the training model;

[0148] Specifically, in this embodiment, the S input samples obtained above are made into a hyperparameter in the model training with a quantity of 16. A total of S / 16 hyperparameters can be obtained, and all the obtained hyperparameters are fed into the training model. Of course, 8 or 32 frames can also be selected as a hyperparameter during training in other embodiments.

[0149] S1054, Use batch*h*w*(M + 2) as the input data dimension for training the training model, and use (2*batch*h*w*1, batch*h / 2*w / 2*1, batch*h / 4*w / 4*1) as the output data dimension for training the training model, and output the estimated depth data through the training model;

[0150] Among them, As the predicted edge estimate with an output dimension of batch*h*w*1; in the formula, h*w represents the resolution of the template graph, the training sample graph, and the environmental sample graph, and M represents the number of acquisitions of the template graph. In this embodiment, the resolution of the template graph and the training sample graph is 7*7, that is, the above h = 7, w = 7, and M = 3.

[0151] S1055, Calculate the loss value of the training model through the loss function based on the estimated depth data;

[0152] Among them, the loss function refers to a function that maps an event to a real number representing the economic cost or opportunity cost associated with the event, and is used to assist in continuously reducing the variation of the target value through process improvement. In this embodiment, the loss function is:

[0153]

[0154] In the formula, (x, y) represents the pixel position, h*w represents the resolution of the template graph, the training sample graph, and the environmental sample graph, Represents the estimated depth data at the pixel position (x, y), dep k (x, y) represents the depth data obtained by shrinking the normalized depth data at the pixel position (x, y) by a multiple of k, and the value of K is 1, 2, or 4, Represents the predicted edge estimate at the pixel position (x, y), and edge(x, y) represents the normalized edge supervision data at the position (x, y).

[0155] S1056, Perform backpropagation according to the loss value, and continuously optimize the parameters of the training model until it converges completely to obtain the target model.

[0156] Among them, using the loss value for backpropagation aims to utilize the fact that there may be a large gap between the actual output value and the target value during neural network training. The loss function can intuitively show the gap between the target value and the actual output, so as to guide the output to approach the target value.

[0157] S106. Use the target model to perform inference on the to-be-tested speckle pattern and the to-be-tested speckle-free pattern to obtain normalized predicted depth data, and calculate the actual estimated depth based on the normalized predicted depth data to output the predicted depth information of the to-be-tested speckle pattern in real time.

[0158] Further, as Figure 5 shown, the specific steps of step S106 include:

[0159] S1061. Continuously collect the image f of the to-be-tested speckle pattern r and the image a of the to-be-tested speckle-free pattern r from the speckle camera, and obtain the image f r after filtering out environmental interference and the image a r after filtering out environmental interference through the preprocessing, and splice the image after filtering out environmental interference of the template image to obtain a target input sample, and input the target input sample into the target model for inference to obtain normalized predicted depth data;

[0160] S1062. Based on the normalized predicted depth data, use a preset formula to calculate the actual estimated depth to output the predicted depth information of the to-be-tested speckle pattern in real time.

[0161] Among them, the preset formula is:

[0162]

[0163] In the formula, d is the actual estimated depth, is the normalized predicted depth data, d max is the depth data of the maximum measurement range of the speckle camera, d min is the depth data of the minimum measurement range of the speckle camera.

[0164] In summary, in this embodiment, by collecting speckle data, inputting it into a deep learning neural network, designing a neural network loss function, finally the neural network model outputs normalized depth data, and then the actual depth information is obtained through post-processing. In this application, a speckle RGB-D camera is used to collect speckle images as training data. Since the depth is only related to the speckle offset and has nothing to do with the specific environment, relatively few data samples are required for training; moreover, using a deep learning algorithm instead of template matching in the speckle RGB-D camera can improve the frame rate of the RGB-D camera without reducing the accuracy, reduce the dizziness feeling when using the camera in VR / AR, and can be effectively applied in occasions with high frame rate requirements.

[0165] Embodiment 2

[0166] This embodiment provides a structural block diagram of a system corresponding to the method described in Embodiment 1. Figure 6 It is a structural block diagram of a speckle depth estimation system according to an embodiment of the present application. As Figure 6 shown, the system includes:

[0167] An acquisition module 10, configured to acquire a plurality of template images, training sample images, and environmental sample images; wherein, the template image refers to a white-background planar image with speckles acquired at preset intervals within the depth reconstruction range, the training sample image refers to a speckle-bearing image acquired in different environments within the same depth reconstruction range, and the environmental sample image refers to a non-speckle-bearing image with the same environment and acquisition position as the training sample image;

[0168] A first processing module 20, configured to obtain first depth data of the training sample image according to the template image, the training sample image, and a speckle template matching algorithm, and normalize the first depth data to obtain normalized depth data;

[0169] A second processing module 30, configured to obtain edge data of the environmental sample image according to the environmental sample image and an edge detection algorithm, and normalize the edge data to obtain normalized edge supervision data;

[0170] A training module 40, configured to input the template image, the training sample image, and the environmental sample image into a training model to output estimated depth data;

[0171] An iterative optimization module 50, configured to calculate a loss value of the training model based on the normalized depth data, the normalized edge supervision data, and the estimated depth data, and continuously optimize the training model according to the loss value to obtain a target model with complete convergence processing;

[0172] An operation module 60, configured to perform inference on a to-be-tested speckle image and a to-be-tested non-speckle image through the target model to obtain normalized predicted depth data, and calculate an actual estimated depth based on the normalized predicted depth data to real-time output predicted depth information of the to-be-tested speckle image and the to-be-tested non-speckle image.

[0173] Preferably, the first processing module 20 includes:

[0174] A matching algorithm unit 21, using the template image as a matching template, and obtaining first depth data of the training sample image through a speckle template matching algorithm;

[0175] A setting unit 22, setting depth data less than the minimum measurement range of the speckle camera to the minimum value and depth data greater than the maximum measurement range of the speckle camera to the maximum value in the first depth data to obtain second depth data;

[0176] The first normalization unit 23 normalizes the second depth data to obtain normalized depth data; wherein, the formula for the normalization process is dep(x,y)=(depth(x,y)-d min ) / (d max -d min ), in the formula, (x,y) represents the pixel position, depth(x,y) represents the edge data at the position (x,y), d max is the depth data of the maximum measurement range of the speckle camera, d min is the depth data of the minimum measurement range of the speckle camera, and dep(x,y) represents the normalized depth data at the position (x,y).

[0177] Preferably, the second processing module 30 includes:

[0178] An edge algorithm unit 31 for obtaining the edge data of the environmental sample map through an edge detection algorithm;

[0179] A second normalization unit 32 for normalizing the edge data to obtain normalized edge supervision data; wherein, the formula for the normalization process is edg(x,y)=(edge(x,y)-min(edge)) / (max(edge)-min(edge)), in the formula, (x,y) represents the pixel position, edge(x,y) represents the edge data at the position (x,y), min(edge) represents the minimum value in the edge data, and max(edge) represents the maximum value in the edge data.

[0180] Preferably, the iterative optimization module 50 includes:

[0181] A preprocessing unit 51 for preprocessing the template map, the training sample map, and the environmental sample map to filter out the environmental data in the template map, the training sample map, and the environmental sample map;

[0182] A combined training unit 52 for combining and training the results of the preprocessed template map, the training sample map, and the environmental sample map to obtain S input samples;

[0183] A definition unit 53 for defining each of the S input samples as a hyperparameter batch in the training model with a predetermined number of input samples, obtaining a plurality of hyperparameter data, and inputting the plurality of hyperparameter data into the training model;

[0184] The regulation unit 54 is configured to use batch*h*w*(M + 2) as the input data dimension for training the training model, and use (2*batch*h*w*1, batch*h / 2*w / 2*1, batch*h / 4*w / 4*1) as the output data dimension for training the training model, and output the estimated depth data through the training model; where as the predicted edge estimate with an output dimension of batch*h*w*1; where h*w represents the resolution of the template graph, the training sample graph, and the environmental sample graph, and M represents the number of acquisitions of the template graph;

[0185] The calculation unit 55 is configured to calculate the loss value of the training model through a loss function based on the estimated depth data;

[0186] The iterative optimization unit 56 performs backpropagation according to the loss value, and continuously optimizes the parameters of the training model until complete convergence to obtain the target model.

[0187] Preferably, the preprocessing unit 51 is specifically configured to:

[0188] Perform filtering and denoising on the training sample graph to filter out salt-and-pepper noise and Gaussian noise in the training sample graph; where the filtering and denoising include median filtering and Gaussian filtering;

[0189] Perform local preprocessing on the template graph and the training sample graph that has been filtered and denoised; where the local preprocessing specifically includes:

[0190] Pad the image I of the template graph and the training sample graph with a resolution of h*w with (n - 1) / 2 pixels of pixel value 0 around the perimeter to obtain F1 with a resolution of (h + n - 1)*(w + n - 1), and perform a two-dimensional convolution with a window size of n*n, a step size of 1, and all convolution kernel weights of 1 on F1 to obtain the feature matrix f avg , and obtain the mean feature matrix avg through the mean formula; where n represents the number of pixels, and the mean formula is:

[0191]

[0192] Based on the relationship of F2 = F1 2 obtain F2, and perform a two-dimensional convolution with a window size of n*n, a step size of 1, and all convolution kernel weights of 1 on F2 to obtain the feature matrix f std , and obtain the standard deviation matrix std through the standard deviation formula; where the standard deviation formula is:

[0193]

[0194] Based on the mean feature matrix avg and the standard deviation matrix std, the environmental speckle matrix is removed by a filtering formula to filter out environmental data, and the images of the template map and the training sample map after filtering out environmental interference are obtained; wherein, the filtering formula is:

[0195]

[0196] In the formula, (x, y) represents the pixel position, I(x, y) represents the pixel value at the position (x, y), avg(x, y) represents the pixel mean at the position (x, y), and std(x, y) represents the covariance value at the position (x, y).

[0197] Preferably, the operation module 60 includes:

[0198] An inference unit 61, configured to continuously acquire the image f of the speckle map to be measured and the image a of the speckle-free map to be measured from the speckle camera r and, through the processing of the preprocessing, obtain the image f r The image after filtering out environmental interference r and the image a The image after filtering out environmental interference r and splice the images of the template map after filtering out environmental interference to obtain a target input sample, and input the target input sample into the target model for inference to obtain normalized predicted depth data;

[0199] A real-time operation unit 62, configured to calculate an actual estimated depth based on the normalized predicted depth data by using a preset formula, so as to output the predicted depth information of the speckle map to be measured in real time; wherein, the preset formula is:

[0200]

[0201] In the formula, d is the actual estimated depth, is the normalized predicted depth data, d max is the depth data of the maximum measurement range of the speckle camera, d min is the depth data of the minimum measurement range of the speckle camera.

[0202] Example 3

[0203] Figure 1 Combined Figure 7 The speckle depth estimation method described can be implemented by an electronic device. It is a schematic diagram of the hardware structure of the electronic device according to this embodiment.

[0204] The electronic device may include a processor 71 and a memory 72 storing computer program instructions.

[0205] Specifically, the above-mentioned processor 71 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured as one or more integrated circuits for implementing the embodiments of the present application.

[0206] Among them, the memory 72 may include a mass memory for data or instructions. By way of example and not limitation, the memory 72 may include a hard disk drive (HDD), a floppy disk drive, a solid state drive (SSD), a flash memory, an optical disc, a magneto-optical disc, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 72 may include removable or non-removable (or fixed) media. Where appropriate, the memory 72 may be internal or external to the data processing device. In a particular embodiment, the memory 72 is non-volatile memory. In a particular embodiment, the memory 72 includes a read-only memory (ROM) and a random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically alterable read-only memory (EAROM), or a flash memory, or a combination of two or more of these. Where appropriate, the RAM may be a static random access memory (SRAM) or a dynamic random access memory (DRAM), where the DRAM may be a fast page mode dynamic random access memory (FPMDRAM), an extended date out dynamic random access memory (EDODRAM), a synchronous dynamic random access memory (SDRAM), etc.

[0207] The memory 72 can be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 71.

[0208] The processor 71 reads and executes the computer program instructions stored in the memory 72 to implement the speckle depth estimation method of the above-mentioned Embodiment 1.

[0209] In some of these embodiments, the electronic device may further include a communication interface 73 and a bus 70. Among them, as Figure 7 shown, the processor 71, the memory 72, and the communication interface 73 are connected through the bus 70 to complete communication with each other.

[0210] The communication interface 73 is used to implement communication between the various modules, devices, units, and / or devices in the embodiments of the present application. The communication interface 73 can also implement data communication with other components, such as external devices, image / data acquisition devices, databases, external storage, and image / data processing workstations.

[0211] The bus 70 includes hardware, software, or both, and couples components of the electronic device together. The bus 70 includes, but is not limited to, at least one of the following: Data Bus, Address Bus, Control Bus, Expansion Bus, Local Bus. By way of example and not limitation, the bus 70 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable bus or a combination of two or more of these. Where appropriate, the bus 70 may include one or more buses. Although the embodiments of the present application describe and illustrate specific buses, the present application contemplates any suitable bus or interconnect.

[0212] The electronic device may execute the speckle depth estimation method of Embodiment 1 of the present application based on the acquired speckle depth estimation system.

[0213] In addition, in combination with the speckle depth estimation method in Embodiment 1 above, the embodiments of the present application may be implemented by providing a storage medium. Computer program instructions are stored on the storage medium; when the computer program instructions are executed by a processor, the speckle depth estimation method of Embodiment 1 above is implemented.

[0214] The technical features of the above-described embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0215] The above-described embodiments merely represent several implementation manners of the present invention. The description is relatively specific and detailed, but it should not be construed as a limitation on the scope of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can still be made, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the appended claims.

Claims

1. A speckle depth estimation method, characterized in that, include: Collecting a number of template images, training sample images, and environment sample images; wherein the template image refers to a planar image with a white background and speckled at preset intervals within a depth reconstruction range; the training sample image refers to a speckle image collected in different environments within the same depth reconstruction range; and the environment sample image refers to a non-speckled image with the same environment and collection location as the training sample image; Obtaining first depth data of the training sample image according to the template image, the training sample image, and a speckle template matching algorithm, and normalizing the first depth data to obtain normalized depth data; Obtaining edge data of the environment sample graph according to the environment sample graph and the edge detection algorithm, and normalizing the edge data to obtain normalized edge supervision data; Inputting the template image, the training sample image, and the environment sample image into a training model to output estimated depth data; Calculating a loss value of the training model based on the normalized depth data, the normalized edge supervision data, and the estimated depth data, and continuously optimizing the training model according to the loss value to obtain a fully converged target model; Inferring the speckle pattern to be measured and the speckle-free pattern to be measured using the target model to obtain normalized predicted depth data, and calculating the actual estimated depth based on the normalized predicted depth data to output predicted depth information of the speckle pattern to be measured and the speckle-free pattern to be measured in real time; The step of calculating the loss value of the training model based on the normalized depth data, the normalized edge supervision data, and the estimated depth data, and continuously optimizing the training model according to the loss value to obtain a fully converged target model specifically includes: Preprocessing the template graph, the training sample graph, and the environment sample graph to filter out environmental data in the template graph, the training sample graph, and the environment sample graph; Merge the pre-processed template image, the training sample image, and the environment sample image to obtain S input samples; defining the S input samples one by one as a hyperparameter batch in the training model with a predetermined number of input samples, obtaining multiple hyperparameter data, and inputting the multiple hyperparameter data into the training model; The training model is trained with batch*h*w*(M+2) as the input data dimension, and with (2*batch*h*w*1, batch*h / 2*w / 2*1, batch*h / 4*w / 4*1) as the output data dimension, and the training model outputs estimated depth data; wherein h*w represents the resolution of the template image, the training sample image, and the environment sample image, and M represents the number of samples collected for the template image; The loss value of the training model is calculated based on the estimated depth data using a loss function, where the loss function is: where (x, y) represents the pixel position, represents the estimated depth data of the pixel position (x, y), dep k (x, y) represents the depth data obtained by reducing the normalized depth data of the pixel position (x, y) by a multiple of k, and the value of K is 1, 2, or 4. represents the predicted edge estimate of the pixel position (x, y), and edge(x, y) represents the normalized edge supervision data at the position (x, y); Back propagation is performed based on the loss value, and the parameters of the training model are continuously optimized until complete convergence to obtain the target model.

2. The speckle depth estimation method according to claim 1, wherein The step of obtaining first depth data of the training sample image according to the template image, the training sample image, and the speckle template matching algorithm, and normalizing the first depth data to obtain normalized depth data specifically includes: Using the template image as a matching template, obtaining first depth data of the training sample image through a speckle template matching algorithm; Setting the depth data smaller than the minimum measurement range of the speckle camera in the first depth data to the minimum value, and setting the depth data larger than the maximum measurement range of the speckle camera to the maximum value, to obtain second depth data; Normalize the second depth data to obtain normalized depth data; wherein, the formula for the normalization process is dep(x,y)=(depth(x,y)-d min ) / (d max -d min ), where (x,y) represents the pixel position, depth(x,y) represents the edge data at the position (x,y), d max is the depth data of the maximum measurement range of the speckle camera, and d min is the depth data of the minimum measurement range of the speckle camera, and dep(x,y) represents the normalized depth data at the position (x,y).

3. The speckle depth estimation method according to claim 1, wherein The specific steps of obtaining edge data of the environment sample graph according to the environment sample graph and the edge detection algorithm, and normalizing the edge data to obtain normalized edge supervision data include: Acquire edge data of the environment sample image by using an edge detection algorithm; The edge data is normalized to obtain normalized edge supervision data; wherein the normalization formula is edg(x,y)=(edge(x,y)-min(edge)) / (max(edge)-min(edge)), where (x,y) represents the pixel position, edge(x,y) represents the edge data at the (x,y) position, min(edge) represents the minimum value in the edge data, and max(edge) represents the maximum value in the edge data.

4. The speckle depth estimation method according to claim 3, characterized in that, The step of preprocessing the template graph, the training sample graph, and the environment sample graph to filter out the environmental data in the template graph, the training sample graph, and the environment sample graph specifically includes: Filtering and denoising the training sample image to filter out salt and pepper noise and Gaussian noise in the training sample image; wherein the filtering and denoising includes median filtering and Gaussian filtering; The template image and the training sample image subjected to filtering and denoising are subjected to local preprocessing; wherein the local preprocessing specifically includes: Pad the image I of the template image and the training sample image with a resolution of h*w with (n-1) / 2 pixels of pixel value 0 around it to obtain F1 with a resolution of (h+n-1)*(w+n-1), and perform a two-dimensional convolution with a window size of n*n, a step size of 1, and all convolution kernel weights of 1 based on F1 to obtain the feature matrix f avg , and obtain the average feature matrix avg through the average formula; where n represents the number of pixels, and the average formula is: Obtain F2 based on the relationship of F2 = F1 2 and perform a two-dimensional convolution with a window size of n*n, a step size of 1, and all convolution kernel weights being 1 on the basis of F2 to obtain the feature matrix f std , and obtain the standard deviation matrix std through the standard deviation formula; where the standard deviation formula is: Based on the mean feature matrix avg and the standard deviation matrix std, the environmental speckle matrix is removed by a filtering formula to filter out environmental data, thereby obtaining images of the template image and the training sample image after environmental interference is filtered out. The filtering formula is: Where (x, y) represents the pixel position, represents the pixel value at the (x, y) position after filtering out environmental interference, I(x, y) represents the pixel value at the (x, y) position, avg(x, y) represents the pixel mean value at the (x, y) position, and std(x, y) represents the covariance value at the (x, y) position.

5. The speckle depth estimation method according to claim 4, characterized in that The step of inferring the speckle pattern to be measured and the speckle-free pattern to be measured using the target model to obtain normalized predicted depth data, and calculating the actual estimated depth based on the normalized predicted depth data to output the predicted depth information of the speckle pattern to be measured in real time specifically includes: The image f of the speckle pattern to be measured continuously collected from the speckle camera r and the image a of the speckle-free pattern to be measured r , the image f is obtained through the processing of the said preprocessing r the image after filtering out environmental interference and the image a r the image after filtering out environmental interference and splice the image after filtering out environmental interference of the template image to obtain the target input sample, and input the target input sample into the target model for inference to obtain the normalized predicted depth data; Based on the normalized predicted depth data, a preset formula is used to calculate the actual estimated depth, so as to output the predicted depth information of the speckle pattern to be measured in real time; wherein the preset formula is: Where d is the actual estimated depth, is the normalized predicted depth data, d max is the depth data of the maximum measurement range of the speckle camera, d min is the depth data of the minimum measurement range of the speckle camera.

6. A speckle depth estimation system, characterized in that, include: An acquisition module, configured to acquire a plurality of template images, training sample images, and environment sample images; wherein the template image refers to a planar image with a white background and speckled at preset intervals within a depth reconstruction range; the training sample image refers to a speckle image acquired in different environments within the same depth reconstruction range; and the environment sample image refers to a non-speckle image with the same environment and acquisition location as the training sample image. The first processing module is used to obtain the first depth data of the training sample image according to the template image, the training sample image, and the speckle template matching algorithm, and normalize the first depth data to obtain normalized depth data; The second processing module is used to obtain the edge data of the environmental sample image according to the environmental sample image and the edge detection algorithm, and normalize the edge data to obtain normalized edge supervision data; The training module is used to input the template image, the training sample image, and the environmental sample image into the training model to output estimated depth data; The iterative optimization module is used to calculate the loss value of the training model based on the normalized depth data, the normalized edge supervision data, and the estimated depth data, and continuously optimize the training model according to the loss value to obtain a target model with complete convergence processing; The operation module is used to perform inference on the to-be-tested speckle image and the to-be-tested speckle-free image through the target model to obtain normalized predicted depth data, and calculate the actual estimated depth based on the normalized predicted depth data to real-time output the predicted depth information of the to-be-tested speckle image and the to-be-tested speckle-free image; Wherein, the iterative optimization module includes: The preprocessing unit is used to preprocess the template image, the training sample image, and the environmental sample image to filter out the environmental data in the template image, the training sample image, and the environmental sample image; The combined training unit is used to combine and train the results of the preprocessed template image, training sample image, and environmental sample image to obtain S input samples; The definition unit is used to define each of the S input samples as a hyperparameter batch in the training model with a predetermined number of input samples, obtain multiple hyperparameter data, and input the multiple hyperparameter data into the training model; The regulation unit is used to use batch*h*w*(M + 2) as the input data dimension for training the training model, and use (2*batch*h*w*1, batch*h / 2*w / 2*1, batch*h / 4*w / 4*1) as the output data dimension for training the training model, and output the estimated depth data through the training model; where h*w represents the resolution of the template image, the training sample image, and the environmental sample image, and M represents the acquisition quantity of the template image; The calculation unit is used to calculate the loss value of the training model based on the estimated depth data through a loss function, and the loss function is: Wherein, (x, y) represents the pixel position, represents the estimated depth data of the pixel position (x, y), dep k (x, y) represents the depth data obtained by reducing the normalized depth data of the pixel position (x, y) by a multiple of k, and the value of K is 1, 2, or 4. represents the predicted edge estimation of the pixel position (x, y), and edge(x, y) represents the normalized edge supervision data at the position (x, y); The iterative optimization unit performs backpropagation according to the loss value and continuously optimizes the parameters of the training model until complete convergence to obtain a target model.

7. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the speckle depth estimation method according to any one of claims 1 to 5 when executing the computer program.

8. A storage medium having a computer program stored thereon, characterized in that, The program implements the speckle depth estimation method according to any one of claims 1 to 5 when executed by the processor.

Images