Training Method and Calculation Method for Depth Calculation Model Applied to Monocular Structured Light

Abstract

The present invention relates to the field of structured light depth calculation, and specifically relates to a training method and a calculation method for a depth calculation model applied to monocular structured light. The present invention uses the same disparity generation module to update the disparity map and semantic information, so that the updates of the two share the same structure in the depth calculation model, and further enables the updates of the two to share the same set of parameters in the depth calculation model, realizing model parameter sharing. Realizing model parameter sharing can improve the generalization ability of the model after training, that is, any image obtained by structured light can be input into the model after training, and the depth information corresponding to the image can be obtained. By adopting the method of parameter sharing, the present invention can reduce the number of parameters and reduce the memory operation pressure.

Description

Technical Field

[0001] The present invention relates to the field of structured light depth calculation, and specifically to a training method and a calculation method for a depth calculation model applied to monocular structured light. Background Art

[0002] Depth cameras include monocular structured light cameras and binocular structured light cameras. Both monocular structured light cameras and binocular structured light cameras can calculate the distance (depth) between the scene in the image and the camera based on the images captured by the depth camera. The prior art trains a depth calculation model to obtain a trained depth calculation model, and then calculates the depth information of the image to be calculated through the trained depth calculation model. When training the depth calculation model in the prior art, synthetic data is used for fitting to obtain the fitted synthetic data, and then the synthetic data is used to train the depth calculation model. Since the prior art uses a complex self-supervised system to fit the data to obtain the fitted synthetic data, the trained depth calculation model is difficult to generalize to real scenes.

[0003] In summary, the generalization ability of the depth calculation model trained by the prior art is relatively low.

[0004] Therefore, the prior art still needs to be improved. Summary of the Invention

[0005] To solve the above technical problems, the present invention provides a training method and a calculation method for a depth calculation model applied to monocular structured light, which solves the problem that the generalization ability of the depth calculation model trained by the prior art is relatively low.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a training method for a depth calculation model applied to monocular structured light

[0008] A method, wherein the training method of the depth calculation model includes: obtaining a depth calculation model, the depth calculation model including a disparity generation module; obtaining a sample reference speckle pattern, a sample scene speckle pattern, and a reference disparity map composed of the sample reference speckle pattern and the sample scene speckle pattern; extracting the feature information of the sample reference speckle pattern and the sample scene speckle pattern, and performing calculations to obtain a cost space; extracting the initial semantic information and initial hidden information in the sample scene speckle pattern; using the initial semantic information, the cost space, and the initial hidden information to obtain a first disparity map and a first hidden information; calculating a first loss function value between the first disparity map and the reference disparity map; if the first loss function value is less than a predetermined threshold, it is confirmed that the training of the depth calculation model is completed; if the first loss function value is greater than or equal to the predetermined threshold, then fuse the first hidden information, the first disparity map, and the cost space to obtain a second disparity map; calculate a second loss function value between the second disparity map and the reference disparity map; calculate the weighted sum of the first loss function value and the second loss function value to obtain an accumulated loss function, wherein the weight of the second loss function value is greater than the weight of the first loss function value; if the accumulated loss function is less than the predetermined threshold, it is determined that the training of the depth calculation model is completed.

[0009] In a second aspect, the present invention provides a depth calculation method based on a trained depth calculation model. The depth calculation method includes: obtaining a to-be-measured scene speckle pattern and a speckle reference map; extracting the feature information of the to-be-measured scene speckle pattern and the speckle reference map and performing calculations to obtain a cost space; extracting the initial semantic information and initial hidden information of the speckle reference map; using the initial semantic information, the initial hidden information, and the cost space to generate a first target disparity map and a first hidden information; and obtaining the depth information corresponding to the to-be-measured scene speckle pattern based on the first target disparity map.

[0010] In one implementation, the obtaining the depth information corresponding to the to-be-measured scene speckle pattern based on the first target disparity map further includes: generating a second target disparity map according to a preset number of update times by using the first target disparity map, the first hidden information, and the cost space; and obtaining the depth information corresponding to the to-be-measured scene speckle pattern based on the second target disparity map and the depth information corresponding to the speckle reference map.

[0011] In one implementation, a preset resolution ratio between low resolution and full resolution is set, and the resolution of the speckle pattern of the scene to be measured is the full resolution; when extracting feature information from the speckle pattern of the scene to be measured and the speckle reference pattern according to the preset resolution ratio, downsampling processing is performed; when generating the second target disparity map by using the first target disparity map, the first hidden information, and the cost space, an upsampling mask is also obtained; according to the upsampling mask, the first target disparity map and / or the second target disparity map are processed to obtain the full-resolution disparity map corresponding to the first target disparity map and / or the full-resolution disparity map corresponding to the second target disparity map.

[0012] In a third aspect, an embodiment of the present invention further provides a training device for a depth calculation model applied to monocular structured light. The device includes the following components: a model input module, configured to obtain a depth calculation model, where the depth calculation model includes a disparity generation module; an image acquisition module, configured to obtain a sample reference speckle pattern, a sample scene speckle pattern, and a reference disparity map composed of the sample reference speckle pattern and the sample scene speckle pattern; a feature extraction module, configured to extract feature information covered by the sample reference speckle pattern and the sample scene speckle pattern, and calculate a cost space; an information extraction module, configured to extract initial semantic information and initial hidden information in the sample scene speckle pattern; a disparity generation module, configured to obtain a first disparity map by using the initial semantic information, the cost space, and the initial hidden information; a loss function calculation module, configured to calculate a first loss function value between the first disparity map and the reference disparity map; a training module, configured to confirm that the training of the depth calculation model is completed if the first loss function value is less than a predetermined threshold; and an upsampling module, configured to increase the resolution of the first disparity map to the same full resolution as the sample scene speckle pattern.

[0013] In a fourth aspect, an embodiment of the present invention further provides a terminal device. The terminal device includes a memory, a processor, and a training program for a depth calculation model applied to monocular structured light stored in the memory and executable on the processor. When the processor executes the training program for the depth calculation model applied to monocular structured light, the steps of the training method for the depth calculation model applied to monocular structured light described above are implemented.

[0014] In a fifth aspect, an embodiment of the present invention further provides a computer-readable storage medium. A training program for a depth calculation model applied to monocular structured light is stored on the computer-readable storage medium. When the training program for the depth calculation model applied to monocular structured light is executed by a processor, the steps of the training method for the depth calculation model applied to monocular structured light described above are implemented.

[0015] Beneficial effects: The present invention uses the same parallax generation module to update the parallax map and semantic information, so that the updates of both share the same structure in the depth calculation model, and further the updates of both share the same set of parameters in the depth calculation model, realizing model parameter sharing. Realizing model parameter sharing can improve the generalization ability of the model after training, that is, any image obtained by structured light can be input into the trained model to obtain the depth information corresponding to the image. In addition, since the present invention adopts the method of parameter sharing, the number of parameters can be reduced and the memory operation pressure can be reduced. Brief Description of the Drawings

[0016] Figure 1 is the overall flowchart of the present invention;

[0017] Figure 2 is the flowchart in the embodiment;

[0018] Figure 3 is the internal structure principle block diagram of the terminal device provided by the embodiment of the present invention. Detailed Embodiments

[0019] The following combines the embodiments and the drawings of the specification to clearly and completely describe the technical solutions in the present invention. Based on the embodiments in the present invention, all other embodiments obtained by those of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

[0020] It has been found through research that cameras include monocular structured light cameras and binocular structured light cameras. Both monocular structured light cameras and binocular structured light cameras can calculate the distance (depth) between the scene in the image and the camera based on the images captured by the camera. The prior art trains the depth calculation model to obtain the trained depth calculation model, and then calculates the depth information of the image to be calculated through the trained depth calculation model. When the prior art trains the depth calculation model, synthetic data is used for fitting to obtain the fitted synthetic data, and then the synthetic data is used to train the depth calculation model. Since the prior art uses a complex self-supervised system to fit the data to obtain the fitted synthetic data, the trained depth calculation model is difficult to generalize to the real scene.

[0021] To solve the above technical problems, the present invention provides a training method and a depth calculation method for a depth calculation model applied to monocular structured light, which solves the problem of low generalization ability of the depth calculation model obtained by the prior art training. Specifically in implementation, first, extract the features corresponding to the structured light in the sample reference speckle pattern and the sample scene speckle pattern, and use these features to construct a cost space. At the same time, the semantic information of the sample scene speckle pattern is also extracted. Use the cost space, the speech information, and the preset initial disparity to train the depth calculation model where the disparity generation module is located. The present invention can improve the generalization ability of the model after training through model parameter sharing. The depth calculation model (neural network model) includes a disparity generation module, and when outputting a disparity map, it also outputs updated hidden information, and can also perform iterative operations based on the updated hidden information and the disparity map to optimize the disparity and finally obtain a disparity map with higher accuracy. It can be used to update the preset initial disparity to obtain an updated disparity map. In this embodiment, the sample reference speckle pattern (the depth information of the sample reference speckle pattern is known) is obtained by simulation synthesis. The sample reference speckle pattern contains the texture information formed by the structured light pattern projected onto the target. Similarly, the sample scene speckle pattern also has the texture information formed by the structured light pattern projected onto the target. The texture information of both is used to construct the cost space.

[0022] Exemplary method

[0023] A training method for a depth calculation model applied to monocular structured light in this embodiment can be applied to a terminal device. The terminal device can be a terminal product with an image acquisition function, such as a mobile phone, a camera, etc. This embodiment includes two parts: one part is to train the depth calculation model, and the other part is to calculate the structured light depth based on the trained depth calculation model, as Figure 1 shown, the training method for the depth calculation model applied to monocular structured light specifically includes the following steps:

[0024] S100, obtain a depth calculation model, and the depth calculation model includes a disparity generation module.

[0025] The depth calculation model at this time is an untrained model. After being trained in subsequent steps S200 to S700, the trained depth calculation model is obtained, and then the trained depth calculation model is implanted into a monocular structured light camera for depth calculation.

[0026] S200, obtain a sample reference speckle pattern, a sample scene speckle pattern, and a reference disparity map composed of the sample reference speckle pattern and the sample scene speckle pattern.

[0027] The reference disparity map is used to reflect the gap between the depth information corresponding to the sample scene speckle pattern and the depth information corresponding to the sample reference speckle pattern.

[0028] In this embodiment, the depth information corresponding to the target image captured by the monocular structured light camera (i.e., the distance between the camera and the captured target) is calculated. The sample reference speckle pattern can be an image captured by the camera at a distance of forty centimeters from the reference target, or can be obtained by simulation synthesis. The sample disparity map between the sample reference speckle pattern and the sample scene speckle pattern is known. The sample disparity map is used to reflect the depth difference between the sample reference speckle pattern and the sample scene speckle pattern, and the sample disparity map is used as the subsequent training termination condition.

[0029] In addition, both the sample reference speckle pattern and the sample scene speckle pattern in this embodiment are low-resolution images. The resolution of the original image can be reduced to 1 / 2, 1 / 4, 1 / 8, 1 / 16, 1 / 32 to obtain the low-resolution sample reference speckle pattern and sample scene speckle pattern. The reason for using low-resolution images is to reduce the time required for model training, so as to quickly complete the training of the model.

[0030] S300, Extract the feature information covered by the sample reference speckle pattern and the sample scene speckle pattern, and calculate to obtain the cost space.

[0031] In this embodiment, the specific process of calculating the cost space is as follows:

[0032] Use the feature extraction module (a part of the neural network model) to extract the features on the low resolution of the sample reference speckle pattern and the sample scene speckle pattern respectively. This feature reflects the texture information contained in the speckle image, and then use matrix multiplication of these two features to calculate the cost space.

[0033] S400, Extract the initial semantic information and initial hidden information in the sample scene speckle pattern.

[0034] In this embodiment, the initial semantic information is used to characterize the scene features in the sample scene speckle pattern. Use the semantic extraction module (a part of the neural network model) to extract the initial semantic information of the sample scene speckle pattern. The initial semantic information is used to characterize the scene features in the sample scene speckle pattern. For example, the sample scene speckle pattern is a speckle pattern projected onto a tree.

[0035] In this embodiment, the convolutional layer structures corresponding to the feature extraction module and the semantic extraction module are the same. Therefore, the feature extraction module and the semantic extraction module share parameters, thereby reducing the parameters of the neural network model and thus reducing the amount of calculation.

[0036] S500, Use the initial semantic information, the cost space and the initial hidden information to obtain the first disparity map.

[0037] Input the initial semantic information, cost space, and initial hidden information into the disparity generation module located in the depth calculation model to generate a first disparity map.

[0038] In this embodiment, the disparity generation module outputs a first hidden information while outputting the first disparity map. The hidden information is the information in the sample scene speckle map other than the semantic information.

[0039] In this embodiment, the initial semantic information, the cost space, and a preset initial disparity are input into the disparity generation module to obtain an updated disparity map (the first disparity map) and updated semantic information output by the disparity generation module. The disparity generation module is located in the depth calculation model (i.e., the disparity generation module is a part of the depth calculation model).

[0040] In this embodiment, the initial semantic information can be directly input into the disparity generation module to obtain an updated disparity map, or the initial semantic information can be divided into initial first semantic information and initial second semantic information. The initial second semantic information is the semantic information to be updated. The second semantic information to be updated not only contains the information in the sample scene speckle map but also contains image-related information and is composite information.

[0041] In one embodiment, while generating the first disparity map using the initial semantic information, the cost space, and the initial hidden information, a first hidden information is also obtained. The disparity generation module can adopt convolutional, GRU, and LSTM methods. The initial semantic information includes initial first semantic information and initial second semantic information. When performing the update, S500 updates the disparity map (the first disparity map) through the following steps:

[0042] S501: According to the initial semantic information, obtain the initial first semantic information and the initial second semantic information in the initial semantic information.

[0043] S502: Input the initial first semantic information, the initial second semantic information, the cost space, and the preset initial disparity into the disparity generation module to obtain an updated disparity map (the updated disparity map for the initial disparity) and updated second semantic information output by the disparity generation module.

[0044] S600: Calculate the first loss function value between the first disparity map and the reference disparity map.

[0045] The specific process of step S600 is as follows: Calculate the difference between the first disparity map and the reference disparity map; when the absolute value of the difference is less than a set value, calculate the square value of the difference; multiply the square value of the difference by a set constant to obtain the first loss function value; or, when the absolute value of the difference is less than the set value, subtract the set constant from the absolute value of the difference to obtain the first loss function value.

[0046] Step S600 obtains the first loss function value based on the following formula:

[0047]

[0048] x1 is the difference between the first disparity map and the reference disparity map, and L1 is the first loss function value.

[0049] S700, determine whether the first loss function is less than a predetermined threshold.

[0050] If it is less, confirm that the training of the depth calculation model is completed; otherwise, adjust the parameters of the disparity generation module for iterative training.

[0051] In one embodiment, if the first loss function value is less than the predetermined threshold, confirm that the training of the depth calculation model is completed. Or, if the first loss function value is greater than or equal to the predetermined threshold, adjust the parameters related to the disparity generation module in the depth calculation model for iterative training. The predetermined threshold of this embodiment is set according to different sample scene speckle patterns.

[0052] In this embodiment, by continuously adjusting the parameters in the disparity generation module, collecting the disparity map output by the disparity generation module after adjusting the parameters, and then calculating the loss function value through this disparity map and the reference disparity map. When the loss function value is less than the predetermined threshold, it indicates that the training of the disparity generation module has been completed; otherwise, repeat the above process until the finally calculated loss function value is less than the predetermined threshold.

[0053] Step S700 further includes the following steps:

[0054] S701, fuse the first hidden information, the first disparity map, and the cost space to obtain a second disparity map.

[0055] Input the first hidden information, the first disparity map, and the cost space into the disparity generation module after adjusting the parameters to obtain the second disparity map output by the disparity generation module.

[0056] S702, calculate the second loss function value L2 of the second disparity map and the reference disparity map:

[0057]

[0058] x2 is the difference between the second parallax map and the reference parallax map.

[0059] S703. Calculate the weighted sum of the first loss function value and the second loss function value to obtain the cumulative loss function L:

[0060] L = aL1 + bL2

[0061] where b is greater than a.

[0062] S704. If the cumulative loss function is less than the predetermined threshold, determine that the training of the depth calculation model is completed.

[0063] If the cumulative loss function is still greater than the predetermined threshold, continue to calculate the next parallax map through the second parallax map until the loss function calculated based on the parallax map is less than the predetermined threshold, then the training of the depth calculation model is completed.

[0064] It can be understood that during the iterative training process of this embodiment, a preset number of iterations can also be selected, and when the update times reach the preset number of iterations, the iterative update training of the model ends.

[0065] This embodiment also provides a depth calculation method based on the depth calculation model after training. The depth calculation model trained by steps S100 to S700 is used to calculate the depth information of the to-be-measured speckle scene map. The depth calculation method includes the following steps S901, S902, S903, S904, S905, S906, S907:

[0066] S901. Obtain the to-be-measured scene speckle map and the speckle reference map.

[0067] The to-be-measured scene speckle map and the speckle reference map are obtained by the same structured light, and the depth information corresponding to the speckle reference map is known.

[0068] The to-be-measured scene speckle map is the real speckle map obtained by the module in the monocular structured light camera, and the speckle reference map is also the real speckle map obtained by the module, and the depth information of the speckle reference map is known.

[0069] S902. Extract the feature information of the to-be-measured scene speckle map and the speckle reference map and perform calculations to obtain the cost space.

[0070] In this embodiment, the cost space is used to record the feature information of the to-be-measured speckle scene map and the feature information of the speckle reference map.

[0071] S903. Extract the initial semantic information and the initial hidden information of the speckle reference map.

[0072] S904. Generate a first target disparity map by using the initial semantic information, the initial hidden information, and the cost space.

[0073] Input the initial semantic information, the initial hidden information, and the cost space into the disparity generation module in the trained depth calculation model, and obtain the first target disparity map and the first hidden information output by the disparity generation module.

[0074] S905. According to the preset number of updates, fuse the first target disparity map, the first hidden information, and the cost space through the disparity generation module in the trained depth calculation model to generate a second target disparity map.

[0075] The disparity generation module is used to output each target disparity map.

[0076] S906. Obtain the depth information corresponding to the speckle map of the to-be-measured scene according to the depth information corresponding to the second target disparity map and the speckle reference map.

[0077] The resolution of the original speckle map of the to-be-measured scene is the full resolution. If a map with the full resolution is input into the model, it will increase the computational amount of the model. Therefore, before inputting into the model, first reduce the resolution of the to-be-measured speckle scene map and the speckle reference map (perform downsampling processing when extracting feature information from the to-be-measured scene speckle map and the speckle reference map according to the preset resolution ratio) to improve the training speed of the model. The model generates a second target disparity map according to the first target disparity map and also generates an upsampling mask, and the upsampling mask is used to improve the resolution of the second target disparity map.

[0078] S907. Process the first target disparity map and / or the second target disparity map according to the upsampling mask to obtain the full-resolution disparity map corresponding to the first target disparity map and / or the full-resolution disparity map corresponding to the second target disparity map.

[0079] Take Figure 2 as an example to illustrate the overall process of the present invention. Figure 2 The update module in

[0080] a: Collect monocular structured light simulation synthesis data according to the 3D model.

[0081] The speckle reference map and the speckle scene map used for training the model are both simulation data made according to the real situation. The speckle reference map and the speckle scene map constitute the simulation synthesis data.

[0082] b: Construct a neural network model. The network inputs the speckle reference map of the fixed-distance plane target in the synthesis data and the collected speckle scene map, and the output is the disparity map aligned with the speckle scene map.

[0083] In this embodiment, the speckle reference map of the planar target is the speckle map captured by the camera, and the distance between the target corresponding to the speckle map and the camera is known.

[0084] c: Design a loss function to supervise the network.

[0085] In this embodiment, the design loss function L1 is obtained through the following formula:

[0086]

[0087] Where x is the difference between the reference disparity map and the updated disparity map. Each iteration corresponds to an L1. As the number of update iterations increases, the weight corresponding to L1 is continuously increasing. All L1s are weighted and summed as the loss value of the current iteration. When actually calculating the depth information of the speckle scene map, the sample reference speckle map and the sample scene speckle map are also input into the trained model. The loss function L1 is calculated for the updated disparity map output by the model and the sample disparity map, and the trained model is supervised according to L1 to determine whether it is in a normal working state.

[0088] d: Train the neural network model with synthetic data to obtain the trained model parameters.

[0089] The specific process of step d is as follows:

[0090] d1. Extract feature information from the speckle reference map and the speckle scene map at a low resolution through the feature extraction sub-module.

[0091] d2. Calculate the cost space by matrix multiplication according to the features of the speckle reference map and the speckle scene map at a low resolution obtained in step d1.

[0092] d3. Extract the semantic information at a low resolution from the speckle scene map through the semantic extraction module, and split it into hidden information (second semantic information) and semantic information (first semantic information).

[0093] d4. Use the hidden information and semantic information (first semantic information) obtained in step d3, the cost space obtained in step d2, and the initialized disparity as the first update input of the disparity generation module, and the output is the updated hidden information, the updated disparity map, and the upsampled convolutional mask.

[0094] In this embodiment, the initialized disparity map is set manually, usually set to zero.

[0095] d5. According to the preset number of updates, in subsequent update steps, use the cost space obtained in step d2, the semantic information obtained in step d3, the updated hidden information output by the previous disparity generation module, and the updated disparity map as the input of the current disparity generation module.

[0096] d6. Repeat step d5, calculate the loss function between the updated disparity map and the reference disparity map. When the loss function meets the preset cut-off condition, complete the training of the model to obtain the model parameters.

[0097] e: Load the model parameters obtained in step d into the model structure, input the real monocular structured light module, and according to the speckle reference map and the speckle scene map obtained by the structured light module, the network outputs a dense disparity map aligned with the speckle scene map.

[0098] After obtaining the disparity map, the depth information corresponding to the speckle scene map can be calculated based on the disparity map, where the calculation of the depth information is prior art.

[0099] In summary, the present invention uses the same disparity generation module to update the disparity map and the semantic information, so that the updates of the two share the same structure in the depth calculation model, and further the updates of the two share the same set of parameters in the depth calculation model, realizing model parameter sharing. Realizing model parameter sharing can improve the generalization ability of the model after training, that is, inputting any image obtained by structured light into the trained model can obtain the depth information corresponding to the image. In addition, since the present invention adopts the method of parameter sharing, it can reduce the number of parameters and reduce the memory operation pressure.

[0100] Exemplary device

[0101] This embodiment also provides a device for training a structured light depth calculation model, and the device includes the following components:

[0102] Model input module, used to obtain a depth calculation model, and the depth calculation model includes a disparity generation module;

[0103] Image acquisition module, used to obtain a sample reference speckle map, a sample scene speckle map, and a reference disparity map composed of the sample reference speckle map and the sample scene speckle map;

[0104] Feature extraction module, used to extract the feature information covered by the sample reference speckle map and the sample scene speckle map, and calculate to obtain a cost space;

[0105] Information extraction module, used to extract the initial semantic information and the initial hidden information in the sample scene speckle map;

[0106] A parallax generation module, configured to obtain a first parallax map by using the initial semantic information, the cost space, and the initial hidden information;

[0107] A loss function calculation module, configured to calculate a first loss function value of the first parallax map and the reference parallax map;

[0108] A training module, configured to confirm that the training of the depth calculation model is completed if the first loss function value is less than a predetermined threshold;

[0109] An upsampling module, configured to increase the resolution of the first parallax map to the full resolution same as that of the sample scene speckle map.

[0110] Based on the above embodiments, the present invention further provides a terminal device, and its principle block diagram can be as shown in the figure. The terminal device includes a processor, a memory, a network interface, a display screen, and a temperature sensor connected through a system bus. Among them, the processor of the terminal device is configured to provide computing and control capabilities. The memory of the terminal device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the terminal device is configured to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it implements a training method for a depth calculation model applied to monocular structured light. The display screen of the terminal device can be a liquid crystal display screen or an electronic ink display screen, and the temperature sensor of the terminal device is pre-set inside the terminal device for detecting the operating temperature of the internal device.

[0111] Those skilled in the art can understand that Figure 3 the principle block diagram shown in is only a block diagram of some structures related to the solution of the present invention, and does not constitute a limitation on the terminal device to which the solution of the present invention is applied. The specific terminal device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0112] In one embodiment, a terminal device is provided. The terminal device includes a memory, a processor, and a training method program for a depth calculation model applied to monocular structured light stored in the memory and executable on the processor. When the processor executes the training method program for the depth calculation model applied to monocular structured light, the following operation instructions are implemented:

[0113] Obtain a sample reference speckle map, a sample scene speckle map, and a reference parallax map composed of the sample reference speckle map and the sample scene speckle map;

[0114] Extract the feature information covered by the sample reference speckle pattern and the sample scene speckle pattern, and calculate to obtain a cost space, where the feature information corresponds to the monocular structured light;

[0115] Extract the initial semantic information and initial hidden information in the sample scene speckle pattern, where the initial semantic information is used to characterize the scene features in the sample scene speckle pattern;

[0116] Input the initial semantic information, the cost space, and the initial hidden information into the disparity generation module in the depth calculation model to obtain a first disparity map;

[0117] Calculate the first loss function value between the first disparity map and the reference disparity map;

[0118] If the first loss function value is less than a predetermined threshold, it is confirmed that the training of the depth calculation model is completed.

[0119] Those of ordinary skill in the art can understand that all or part of the processes of implementing the methods in the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the various embodiments provided by the present invention can include non-volatile and / or volatile memories. Non-volatile memories can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memories can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0120] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A training method for a depth calculation model applied to monocular structured light, characterized in that The training method of the depth calculation model includes: Obtain a depth calculation model, where the depth calculation model includes a disparity generation module; Obtain a sample reference speckle pattern, a sample scene speckle pattern, and a reference disparity map composed of the sample reference speckle pattern and the sample scene speckle pattern; Extract the feature information of the sample reference speckle pattern and the sample scene speckle pattern, and perform calculations to obtain a cost space; Extract the initial semantic information and initial hidden information in the sample scene speckle pattern, where the hidden information is the information other than the semantic information in the sample scene speckle pattern; Use the initial semantic information, the cost space, and the initial hidden information to obtain a first disparity map; Calculate the first loss function value between the first disparity map and the reference disparity map; If the first loss function value is less than a predetermined threshold, it is confirmed that the training of the depth calculation model is completed.

2. The training method of the depth calculation model applied to monocular structured light according to claim 1, characterized in that, While generating the first disparity map using the initial semantic information, the cost space, and the initial hidden information, a first hidden information is also obtained. The training method of the depth calculation model further includes: If the first loss function value is greater than or equal to the predetermined threshold, adjust the parameters of the disparity generation module for iterative training.

3. The training method of the depth calculation model applied to monocular structured light according to claim 2, characterized in that, It further includes: Fuse the first hidden information, the first disparity map, and the cost space to obtain a second disparity map; Calculate the second loss function value between the second disparity map and the reference disparity map; Calculate the weighted sum of the first loss function value and the second loss function value to obtain an accumulated loss function; If the accumulated loss function is less than the predetermined threshold, it is determined that the training of the depth calculation model is completed; Among them, the weight of the second loss function value is greater than the weight of the first loss function value.

4. The training method of the depth calculation model applied to monocular structured light according to claim 2, characterized in that, It further includes a preset number of iterations. When the updated training times reach the preset number of iterations, it is determined that the training of the depth calculation model is completed.

5. A deep computing method based on a trained deep computing model, characterized in that, The depth calculation method includes: Obtain a to-be-measured scene speckle pattern and a speckle reference map; Extract the feature information of the to-be-measured scene speckle pattern and the speckle reference map and perform calculations to obtain a cost space; Extract the initial semantic information and initial hidden information of the speckle reference map, where the hidden information is the information other than the semantic information in the sample scene speckle pattern; Use the initial semantic information, the initial hidden information, and the cost space to generate a first target disparity map; Based on the first target disparity map, obtain the depth information corresponding to the to-be-measured scene speckle pattern.

6. The depth calculation method based on the trained depth calculation model according to claim 5, wherein, While generating the first target disparity map using the initial semantic information, the initial hidden information, and the cost space, a first hidden information is also generated; The step of obtaining the depth information corresponding to the to-be-measured scene speckle pattern based on the first target disparity map further includes: Based on a preset number of update times, use the first target disparity map, the first hidden information, and the cost space to generate a second target disparity map; Based on the second target disparity map and the depth information corresponding to the speckle reference map, obtain the depth information corresponding to the to-be-measured scene speckle pattern.

7. The depth calculation method based on the trained depth calculation model according to claim 6, wherein, It further includes: Preset the resolution ratio of low resolution to full resolution, and the resolution of the to-be-measured scene speckle pattern is full resolution; When extracting feature information from the speckle pattern of the scene to be measured and the speckle reference pattern according to a preset resolution ratio, downsampling processing is performed; When generating the second target disparity map by using the first target disparity map, the first hidden information, and the cost space, an upsampling mask is also obtained; According to the upsampling mask, the first target disparity map and / or the second target disparity map are processed to obtain the full-resolution disparity map corresponding to the first target disparity map and / or the full-resolution disparity map corresponding to the second target disparity map.

8. A training device for a depth calculation model applied to monocular structured light, characterized in that, The device includes the following components: A model input module, configured to obtain a depth calculation model, where the depth calculation model includes a disparity generation module; An image acquisition module, configured to obtain a sample reference speckle pattern, a sample scene speckle pattern, and a reference disparity map composed of the sample reference speckle pattern and the sample scene speckle pattern; A feature extraction module, configured to extract the feature information covered by the sample reference speckle pattern and the sample scene speckle pattern, and calculate to obtain a cost space; An information extraction module, configured to extract the initial semantic information and the initial hidden information in the sample scene speckle pattern, where the hidden information is the information other than the semantic information in the sample scene speckle pattern; A disparity generation module, configured to obtain a first disparity map by using the initial semantic information, the cost space, and the initial hidden information; A loss function calculation module, configured to calculate a first loss function value between the first disparity map and the reference disparity map; A training module, configured to confirm that the training of the depth calculation model is completed if the first loss function value is less than a predetermined threshold; An upsampling module, configured to increase the resolution of the first disparity map to the same full resolution as the sample scene speckle pattern.

9. A terminal device, characterized in that, The terminal device includes a memory, a processor, and a training program for a depth calculation model applied to monocular structured light stored in the memory and executable on the processor. When the processor executes the training program for the depth calculation model applied to monocular structured light, the steps of the training method for the depth calculation model applied to monocular structured light according to any one of claims 1-4 are implemented.

10. A computer-readable storage medium, characterized in that, A training program for a depth calculation model applied to monocular structured light is stored on the computer-readable storage medium. When the training program for the depth calculation model applied to monocular structured light is executed by a processor, the steps of the training method for the depth calculation model applied to monocular structured light according to any one of claims 1-4 are implemented.

Images