Deep prediction model training methods, deep prediction methods and related devices

Abstract

An embodiment of the present invention discloses a depth prediction model training method, a depth prediction method and related devices, including: obtaining a training image set, dividing the depth prediction range into multiple continuous depth categories; for each training image, calculating the probability that the pixel point belongs to each depth category based on the depth value, and using the probability as the label value of the pixel point belonging to each depth category; using the training image and the label value to train the depth prediction model, the depth prediction model outputs the probability that the pixel point belongs to each depth category, and uses the probability as the confidence that the pixel point belongs to the depth category, and the probability is calculated by using the depth value as the label, thereby achieving smoothing of the label and more accurately marking the pixel points at the depth category boundary. The depth prediction model trained with the label has high classification accuracy for the pixel points at the depth category boundary, the classification result uses the probability as the confidence, the confidence is smooth, and can reflect the credibility of the classification result.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of artificial intelligence technology, and in particular to a deep prediction model training method, a deep prediction method, an apparatus, an electronic device, and a storage medium. Background Technology

[0002] Autonomous vehicles use various sensors such as cameras and radar to perceive the environment in order to make driving decisions based on the perceived environment. It is crucial to perform depth prediction on the images captured by the cameras to obtain the distance between obstacles and the autonomous vehicle.

[0003] Currently, when training depth prediction models, multiple depth categories are typically set. A pixel in an image is labeled as 1 if its depth value belongs to a certain depth category, and 0 otherwise. This leads to abrupt changes in the labels of pixels near the depth category boundaries. When a depth prediction model is trained using this image, the prediction results also use 0 or 1 to indicate whether a pixel belongs to any of the depth categories. This results in inaccurate depth classification results for pixels near the depth category boundaries, and abrupt changes in the confidence level when using 0 or 1 as the classification result, which fails to reflect the reliability of the classification results. Summary of the Invention

[0004] This invention provides a deep prediction model training method, a deep prediction method, an apparatus, an electronic device, and a storage medium to address the problems of inaccurate predictions, abrupt changes in confidence levels, and failure to reflect the reliability of classification results in existing deep prediction modules.

[0005] In a first aspect, embodiments of the present invention provide a deep prediction model training method, comprising:

[0006] Obtain a training image set, wherein each pixel in the training image set is associated with a depth value;

[0007] The preset depth prediction range is divided into multiple consecutive depth categories;

[0008] For each training image, the probability that the pixel belongs to each depth category is calculated based on the depth value, and the probability is used as the label value of the pixel belonging to each depth category;

[0009] A depth prediction model is trained using the training images and the label values. The depth prediction model is used to output the probability that the pixel belongs to each of the depth categories, wherein the probability corresponding to the depth category is the confidence level that the pixel belongs to the depth category.

[0010] Secondly, embodiments of the present invention provide a depth prediction method, including:

[0011] Obtain the image to be predicted;

[0012] The image to be predicted is input into a pre-trained depth prediction model to obtain the confidence score of each pixel in the image to be predicted belonging to each depth category.

[0013] Determine the maximum confidence level from among the various confidence levels;

[0014] The maximum confidence level and the depth category corresponding to the maximum confidence level are used as the final prediction result;

[0015] The depth prediction model is trained using the depth prediction model training method described in the first aspect.

[0016] Thirdly, embodiments of the present invention provide a deep prediction model training apparatus, comprising:

[0017] The training image set acquisition module is used to acquire a training image set, wherein each pixel of the training image set is associated with a depth value;

[0018] The depth category segmentation module is used to divide the preset depth prediction range into multiple consecutive depth categories;

[0019] The annotation module is used to calculate the probability that a pixel belongs to each depth category based on the depth value for each training image, and use the probability as the label value of the pixel belonging to each depth category.

[0020] The training module is used to train a depth prediction model using the training image and the label value. The depth prediction model is used to output the probability that the pixel belongs to each depth category, wherein the probability corresponding to the depth category is the confidence level that the pixel belongs to the depth category.

[0021] Fourthly, embodiments of the present invention provide a depth prediction device, comprising:

[0022] The image acquisition module is used to acquire the image to be predicted.

[0023] The prediction module is used to input the image to be predicted into a pre-trained depth prediction model to obtain the confidence level of each pixel in the image to be predicted belonging to each depth category.

[0024] The maximum confidence level determination module is used to determine the maximum confidence level from among the various confidence levels;

[0025] The prediction result determination module is used to take the maximum confidence score and the depth category corresponding to the maximum confidence score as the final prediction result;

[0026] The depth prediction model is trained using the depth prediction model training method described in the first aspect.

[0027] Fifthly, embodiments of the present invention also provide an electronic device, the electronic device comprising:

[0028] One or more processors;

[0029] Memory, used to store one or more computer programs;

[0030] When the one or more computer programs are executed by the one or more processors, the one or more processors implement the deep prediction model training method as described in the first aspect, and / or the deep prediction method as described in the second aspect.

[0031] In a sixth aspect, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the depth prediction model training method as described in the first aspect, and / or the depth prediction method as described in the second aspect.

[0032] In this embodiment of the invention, when training a depth prediction model, after dividing the preset depth prediction range into multiple consecutive depth categories, for each training image, the probability of each pixel in the training image belonging to each depth category is calculated using the depth value, and the probability is used as the label value of the pixel belonging to each depth category. The depth prediction model is trained using the training image and the label value. The depth prediction model outputs the probability of a pixel belonging to each depth category, and the probability is used as the confidence level of the pixel belonging to the depth category. On the one hand, compared with using the depth value of a pixel to belong to a certain depth category and using 0 or 1 as the label value, using the probability as the label value makes the label value smoother and more accurately marks the pixels at the depth category boundary. The depth prediction model trained with this label has high accuracy in classifying pixels at the depth category boundary. The confidence level of the depth classification result output by the depth prediction model using the probability is also smoother, avoiding the problem of sudden changes in confidence level. On the other hand, using the probability as the confidence level of the classification result, compared with using 0 or 1 as the confidence level, allows the selection of the classification result with high confidence as the final classification result, and the confidence level can more accurately reflect the credibility of the final classification result. Attached Figure Description

[0033] Figure 1 This is a flowchart illustrating the steps of a deep prediction model training method provided in Embodiment 1 of the present invention.

[0034] Figure 2 This is a flowchart of the steps of a deep prediction model training method provided in Embodiment 2 of the present invention;

[0035] Figure 3 This is a flowchart of the steps of a depth prediction method provided in Embodiment 3 of the present invention;

[0036] Figure 4 This is a schematic diagram of the structure of a deep prediction model training device provided in Embodiment 4 of the present invention;

[0037] Figure 5 This is a flowchart of a depth prediction device provided in Embodiment 5 of the present invention;

[0038] Figure 6 This is a schematic diagram of an electronic device structure provided in Embodiment Six of the present invention. Detailed Implementation

[0039] The present invention will be further described in detail below with reference to the accompanying drawings and examples. It will be understood that the specific embodiments described herein are intended only to illustrate the present invention and are not intended to limit the present invention. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions relevant to the present invention, not all structures.

[0040] The depth prediction method in this invention involves capturing images from cameras on an autonomous vehicle and inputting these images into a depth prediction model to predict the depth value of each pixel in the depth map and its corresponding depth category. In existing technologies, when training a depth prediction model, the predicted depth range is first divided into multiple depth categories that are consecutive in depth. For example, depth category 1 is 0-3 meters, depth category 2 is 3-6 meters, depth category 3 is 6-9 meters, and depth category 4 is 9-12 meters. If the depth value of pixel A in the training image is 3.01 meters, then the label of pixel A is (0, 1, 0, 0), which indicates that pixel A belongs to depth category 2. Therefore, if the label value in this label represents the depth of the pixel... The confidence scores of pixel A for each depth category are as follows: pixel A has a confidence score of 0 for depth category 1 and a confidence score of 1 for depth category 2. The confidence score changes abruptly from 0 to 1. Furthermore, for pixels at the depth category boundaries, such as pixel A (3.01 meters) and pixel B (4 meters), if 0 or 1 is used to represent the confidence score, the confidence scores of pixel A and pixel B for depth category 2 are both 1. In reality, the confidence score of pixel B for depth category 2 is higher than that of pixel A for depth category 2. This means that the label cannot accurately label pixels at the depth category boundaries, and the confidence score cannot accurately reflect the reliability of the depth prediction model's classification results for pixels at the depth category boundaries.

[0041] To address the problems existing in the current depth prediction models, the present invention proposes the following embodiments.

[0042] Example 1

[0043] Figure 1 This is a flowchart of a depth prediction model training method provided in Embodiment 1 of the present invention. This embodiment is applicable to training a depth prediction model to predict the depth category and confidence level of pixels in an image. The method can be executed by a depth prediction model training device, which can be implemented in software and / or hardware and can be configured in an electronic device. The method specifically includes the following steps:

[0044] S101. Obtain the training image set. Each pixel in the training image set is associated with a depth value.

[0045] In this embodiment of the invention, the training image set can be an image set including multiple depth maps, that is, each pixel in the depth map is associated with a depth value. In one example, multiple images can be captured by a binocular camera on an autonomous vehicle, and the distance from the real object corresponding to each pixel in the multiple images to the binocular camera can be calculated as the depth value using the binocular ranging principle. Of course, multiple depth maps can also be directly pulled from the existing depth map library of the network to obtain the training image set. This embodiment of the invention does not limit the method of obtaining the training image set.

[0046] S102. Divide the preset depth prediction range into multiple consecutive depth categories.

[0047] In the decision-making of autonomous vehicles, a depth prediction range can be set. This depth prediction range is related to the vehicle's speed, road conditions, vehicle performance, etc. For example, the depth prediction range can be set larger on highways and smaller on city streets. Alternatively, the depth prediction range can be determined based on the sensor's perception range.

[0048] Depth categories can be represented by depth ranges. In one example, the depth prediction range can be divided into multiple consecutive equidistant intervals to obtain multiple consecutive depth categories for the depth prediction range. For example, if the depth prediction range is 0-96 meters, then depth category 1 can be divided into 0-3 meters, depth category 2 into 3-6 meters, depth category 3 into 6-9 meters, and so on, dividing the depth prediction range into multiple consecutive depth categories. Of course, the above division of depth categories is only an example. In practical applications, those skilled in the art can divide depth categories according to the actual situation, such as dividing the depth prediction range into multiple depth categories in a non-equidistant manner. The embodiments of the present invention do not limit this.

[0049] S103. For each training image, calculate the probability of a pixel belonging to each depth category based on the depth value, and use the probability as the label value of the pixel belonging to each depth category.

[0050] In an optional embodiment, the center point depth value of each depth category can be calculated first. Then, the probability density is calculated using the center point depth value of each depth category and the depth value of each pixel in the training image as the probability that the pixel belongs to each depth category. This probability is then used as the label value of the pixel belonging to each depth category to obtain the label of the pixel. For example, assuming there are K depth categories and the probability that pixel A belongs to each depth category is Pi, then the label of pixel A is (P1, P2, P3, ..., P...). k ), where the probability can be the normal distribution probability of the depth value.

[0051] In this embodiment of the invention, the probability of a pixel belonging to each depth category is used as the label of the pixel. Since the probability calculation is continuous, on the one hand, the label of the pixel is smoothed, avoiding the problem of sudden changes in confidence caused by using the label as the confidence level. On the other hand, using probability as the label can more accurately label pixels near the boundary of the depth category. The depth prediction model trained with this label can accurately classify pixels near the boundary of the depth category. Furthermore, using probability as the confidence level makes the classification results with high confidence more accurate and better reflects the credibility of the classification results.

[0052] S104. Train the depth prediction model using training images and label values.

[0053] The depth prediction model implemented in this invention is used to output the probability of a pixel belonging to each depth category. The probability corresponding to the depth category is the confidence level of the pixel belonging to the depth category. That is, the depth prediction model outputs the depth category to which the pixel belongs and the confidence level of belonging to that depth category.

[0054] In an optional embodiment, one or a batch of training images can be randomly extracted and input into the initialized depth prediction model to obtain the predicted probability of each pixel belonging to each depth category. The loss rate is calculated using the label and prediction probability of each pixel. If the loss rate is less than a preset threshold or the number of iterations is greater than a preset number, it indicates that the prediction accuracy of the depth prediction model is high enough, and training of the depth prediction model can be stopped. If the loss rate is less than a preset threshold or the number of iterations is less than a preset number, it indicates that the prediction accuracy of the depth prediction model is low, and it is necessary to continue iterative training of the depth prediction model. In this case, after adjusting the model parameters of the depth prediction model, training images are extracted and input into the depth prediction model for further training.

[0055] In this embodiment of the invention, when training a depth prediction model, after dividing the preset depth prediction range into multiple consecutive depth categories, for each training image, the probability of each pixel in the training image belonging to each depth category is calculated using the depth value, and the probability is used as the label value of the pixel belonging to each depth category. The depth prediction model is trained using the training image and the label value. The depth prediction model outputs the probability of a pixel belonging to each depth category, and the probability is used as the confidence level of the pixel belonging to the depth category. On the one hand, compared with using the depth value of a pixel to belong to a certain depth category and using 0 or 1 as the label value, using the probability as the label value makes the label value smoother and more accurately marks the pixels at the depth category boundary. The depth prediction model trained with this label has high accuracy in classifying pixels at the depth category boundary. The confidence level of the depth classification result output by the depth prediction model using the probability is also smoother, avoiding the problem of sudden changes in confidence level. On the other hand, using the probability as the confidence level of the classification result, compared with using 0 or 1 as the confidence level, allows the selection of the classification result with high confidence as the final classification result, and the confidence level can more accurately reflect the credibility of the final classification result.

[0056] Example 2

[0057] Figure 2 This is a flowchart of a deep prediction model training method provided in Embodiment 2 of the present invention. The present invention is an optimization based on the foregoing embodiments, and the method specifically includes the following steps:

[0058] S201. Obtain the training image set. Each pixel in the training image set is associated with a depth value.

[0059] In this embodiment of the invention, the training image set is a collection of multiple depth maps, that is, each pixel of each training image in the training image set is associated with a depth value. In practical applications, the training image set can be obtained by calculating the depth values ​​after taking multiple images with a binocular camera on an autonomous vehicle.

[0060] S202. Divide the depth prediction range into multiple continuous, equidistant depth categories according to the preset distance.

[0061] In one example, taking a depth prediction range of 0-96 meters and a preset distance of 3 meters as an example, the depth prediction range of 0-96 meters can be divided into K=32 consecutive depth categories, where depth category 1 is 0-3 meters, depth category 2 is 3-6 meters, depth category 3 is 6-9 meters, and so on to obtain 32 depth categories.

[0062] S203. Calculate the center point depth value for each depth category.

[0063] In an optional embodiment, a depth category is a depth interval, and the center point depth value can be the depth value at the center of the depth interval. For each depth category, the starting depth value and the ending depth value of the depth category can be obtained, the difference between the ending depth value and the starting depth value can be calculated, the product of the difference and 0.5 can be calculated, and finally the sum of the starting depth value and the product can be calculated to obtain the center point depth value of the depth category. The specific calculation formula is as follows:

[0064]

[0065] In the above formula, M i The depth value of the center point of depth category i. Let i be the starting depth value for depth category i. Let be the endpoint depth value of depth category i. Taking depth category 1 as 0-3 meters, depth category 2 as 3-6 meters, and depth category 3 as 6-9 meters as examples, the center point depth value of depth category 1 is 1.5 meters, the center point depth value of depth category 2 is 4.5 meters, and the center point depth value of depth category 3 is 7.5 meters.

[0066] S204. For each pixel in the training image, the probability of each pixel belonging to the depth category is calculated using the pixel's depth value, the center point's depth value, and preset hyperparameters as the pixel's label value.

[0067] Specifically, the probability of a pixel belonging to the depth category can be calculated using the following formula:

[0068]

[0069] Among them, P ij M represents the probability that the depth value dj of pixel j belongs to depth category i. i σ is the center point depth value of depth category i, and σ is a pre-set hyperparameter.

[0070] From the above formula, we can see that for a pixel j, with its depth value dj remaining constant, the probability P ij The depth value M is the center point depth value of each depth category i. iThe distance between the depth value of a pixel A and the center depth value of a certain depth category is a function of the independent variable. This means that the smaller the distance between the depth value of a pixel A and the center depth value of each depth category, the greater the probability that it belongs to a higher depth category. In other words, the distance from the depth value of a pixel to the center depth value of each depth category is smooth. For example, the probabilities that pixel A with a depth value of 3.99 belongs to depth category 1, depth category 2, depth category 3, depth category 4, and depth category 5 are 0.2, 0.5, 0.2, 0.07, and 0.03, respectively. Therefore, the label of pixel A is (0.2, 0.5, 0.03). The label (0.2, 0.5, 0.2, 0.07, 0.03) indicates that the depth value of pixel A is within the range of depth category 2. Compared to the label (0, 1, 0, 0, 0) obtained directly from the depth value of pixel A, the label (0.2, 0.5, 0.2, 0.07, 0.03) for pixel A is smoother and does not have the abrupt changes from 0 to 1 or 1 to 0 in the label (0, 1, 0, 0, 0). Compared to pixels that are labeled with 0 or 1 to mark the depth category boundary, the label of pixels that are labeled with the probability more accurately marks the depth category boundary.

[0071] S205. Randomly extract training images and input them into the depth prediction model to obtain the predicted probability of each pixel belonging to each depth category.

[0072] Specifically, the depth prediction model can be initialized first, for example, by constructing the network structure of the depth prediction model, which can be a neural network such as CNN, DNN, or RNN. After constructing the network structure of the depth prediction model, the network parameters can be initialized. Then, at least one training image can be randomly extracted from the labeled training image set and input into the initialized depth prediction model to obtain the predicted probability of each pixel in the training image belonging to each depth category.

[0073] S206. Calculate the loss rate using the predicted probability and the label value of the pixel.

[0074] Specifically, the loss rate can be calculated using the following loss function:

[0075]

[0076] Among them, pred j The label represents the predicted probability of pixel j. j Let be the label value of pixel j, and N be the number of pixels in the training image.

[0077] Of course, in practical applications, the loss rate can also be calculated using the mean squared error loss function, the squared error loss function, the L2 distance, etc. This embodiment of the invention does not limit the method of calculating the loss rate.

[0078] S207. Determine whether the loss rate is less than the preset threshold.

[0079] Specifically, the preset threshold can be an indicator to measure the prediction accuracy of the depth prediction model. In practical applications, it can be preset according to the situation. For example, a smaller threshold can be set when the depth prediction accuracy requirement is high, and a larger threshold can be set otherwise. After calculating the loss rate, the loss rate is compared with the preset threshold. If the loss rate is less than the preset threshold, S208 is executed, otherwise S209 is executed.

[0080] S208. Stop training the deep prediction model and obtain the trained deep prediction model.

[0081] If the loss rate calculated after each round of training iterations is less than a preset threshold, training of the deep prediction model can be stopped, the model parameters can be saved to obtain the trained deep prediction model, and this deep prediction model can be ported to the on-board computer of an autonomous vehicle for depth prediction.

[0082] S209. Adjust the model parameters of the depth prediction model using the loss rate.

[0083] If the calculated loss rate is less than the preset threshold after each round of training iterations, the gradient can be calculated using the loss rate. The calculated gradient and the preset learning rate are then used to adjust the model parameters of the deep prediction model. The process then returns to S205 to continue training the deep prediction model until the loss rate is less than the preset threshold or the number of training iterations exceeds the preset number.

[0084] This invention acquires a training image set, divides the depth prediction range into multiple continuous, equidistant depth categories according to a preset distance, and calculates the center point depth value for each depth category. For each pixel in the training image, the probability of each pixel belonging to a depth category is calculated using the pixel's depth value, the center point depth value, and preset hyperparameters as the pixel's label value. After randomly extracting training images and inputting them into the depth prediction model to obtain the predicted probability of each pixel belonging to each depth category, the loss rate is calculated using the predicted probability and the pixel's label value. Training of the depth prediction model is stopped when the loss rate is less than a preset threshold; when the loss rate is greater than the preset threshold, the model parameters are adjusted and training continues. The depth prediction model outputs that each pixel belongs to a different depth category. The probability of a pixel belonging to a depth category is used as the confidence level. On the one hand, compared to using the depth value of a pixel to belong to a certain depth category and using 0 or 1 as the label value, using probability as the label value makes the label value smoother and more accurately marks the pixels at the depth category boundary. The depth prediction model trained with this label has high accuracy in classifying pixels at the depth category boundary. The confidence level of the depth classification result output by the depth prediction model using this probability is also smoother, avoiding the problem of sudden changes in confidence level. On the other hand, using probability as the confidence level of the classification result, compared to using 0 or 1 as the confidence level, allows the selection of the classification result with higher confidence as the final classification result. The confidence level can also more accurately reflect the credibility of the final classification result.

[0085] Example 3

[0086] Figure 3 This is a flowchart of a depth prediction method provided in Embodiment 3 of the present invention. This embodiment of the invention is applicable to predicting the depth category and confidence level of each pixel in an image. The method can be executed by a depth prediction device, which can be implemented by software and / or hardware. The depth prediction device can be configured in an electronic device, such as in the onboard computer of an autonomous vehicle or in a backend server. The method specifically includes the following steps:

[0087] S301. Obtain the image to be predicted.

[0088] In this embodiment of the invention, the image to be predicted can be an image captured in real time by a camera on an autonomous vehicle. For example, the camera on the autonomous vehicle captures images by exposing at a preset period and transmits the images to the vehicle's onboard computer. The onboard computer can receive the image and identify it as the image to be predicted in order to predict the depth category and confidence level of each pixel in the image.

[0089] S302. Input the image to be predicted into the pre-trained depth prediction model to obtain the confidence score of each pixel in the image to be predicted belonging to each depth category.

[0090] In this embodiment of the invention, the depth prediction model can be trained using the depth prediction model training method of Embodiment 1 or Embodiment 2. The specific training process can be found in Embodiment 1 or Embodiment 2, and will not be detailed here. After inputting the image to be predicted, the trained depth prediction model can output the probability that each pixel in the image belongs to each depth category, and use this probability as the confidence level of each pixel belonging to each depth category.

[0091] S303. Determine the maximum confidence level from among the various confidence levels.

[0092] Since the depth prediction model predicts the confidence level of each pixel belonging to each depth category, meaning a pixel has multiple confidence levels, the maximum confidence level can be determined from these multiple confidence levels.

[0093] S304. The maximum confidence level and the depth category corresponding to the maximum confidence level are used as the final prediction results.

[0094] The higher the confidence level, the greater the probability that the depth category belongs to that depth category, and the more accurate the classification of the pixel in that depth category. Therefore, the depth category corresponding to the maximum confidence level can be used as the final depth category of each pixel, and the maximum confidence level can be used as the confidence level of the final depth category. Finally, the maximum confidence level and the depth category corresponding to the maximum confidence level are output as the final prediction result to the decision module, so that the decision module can make a decision based on the depth category and the confidence level.

[0095] In this embodiment of the invention, after acquiring the image to be predicted, the image is input into a pre-trained depth prediction model to obtain the confidence score of each pixel in the image belonging to each depth category. The maximum confidence score is determined from these scores, and the maximum confidence score and its corresponding depth category are used as the final prediction result. Since the depth prediction model calculates the probability of each pixel in the training image belonging to each depth category using the depth value of the pixel during training, and uses this probability as the label value of each depth category, after training with the training image and label values, the depth prediction model outputs the probability of a pixel belonging to each depth category, and uses this probability as the confidence score of the pixel belonging to the depth category. On the one hand, compared to using pixel-based methods... The depth value of a point belongs to a certain depth category and is labeled with 0 or 1. Using probability as the label value makes the label value smoother and more accurately marks the pixels at the depth category boundary. The depth prediction model trained with this label has high accuracy in classifying pixels at the depth category boundary. The confidence of the depth classification result output by the depth prediction model using this probability is also smoother, avoiding the problem of sudden changes in confidence. On the other hand, using probability as the confidence of the classification result, compared with using 0 or 1 as the confidence, allows the selection of the classification result with higher confidence as the final classification result. The confidence is also more accurate in reflecting the credibility of the final classification result, ultimately improving the accuracy of the decision-making module and ensuring the safe driving of autonomous vehicles.

[0096] Example 4

[0097] Figure 4 This is a schematic diagram of a deep prediction model training device provided in Embodiment 4 of the present invention. The deep prediction model training device of the present invention may specifically include the following modules:

[0098] The training image set acquisition module 401 is used to acquire a training image set, wherein each pixel of the training image set is associated with a depth value.

[0099] The depth category segmentation module 402 is used to divide the preset depth prediction range into multiple consecutive depth categories;

[0100] The annotation module 403 is used to calculate the probability that the pixel belongs to each depth category based on the depth value for each training image, and use the probability as the label value of the pixel belonging to each depth category.

[0101] The training module 404 is used to train a depth prediction model using the training image and the label value. The depth prediction model is used to output the probability that the pixel belongs to each of the depth categories, wherein the probability corresponding to the depth category is the confidence level that the pixel belongs to the depth category.

[0102] Optionally, the depth category segmentation module 402 includes:

[0103] A segmentation unit is used to divide the depth prediction range into multiple consecutive, equidistant depth categories according to a preset distance.

[0104] Optionally, the annotation module 403 includes:

[0105] The center point depth value calculation module is used to calculate the center point depth value for each of the depth categories;

[0106] The probability calculation module is used to calculate the probability of each pixel belonging to the depth category for each pixel in the training image, using the pixel's depth value, the center point's depth value, and preset hyperparameters, as the pixel's label value.

[0107] Optionally, the center point depth value calculation module includes:

[0108] A depth value acquisition unit is used to acquire the starting depth value and the ending depth value of the depth category;

[0109] A depth difference calculation unit is used to calculate the difference between the endpoint depth value and the starting depth value;

[0110] A product calculation unit is used to calculate the product of the difference and 0.5;

[0111] The center point depth value calculation unit is used to calculate the sum of the product of the starting depth value and the product to obtain the center point depth value of the depth category.

[0112] Optionally, the probability calculation module includes:

[0113] The probability calculation unit is used to calculate the probability that the pixel belongs to the depth category using the following formula:

[0114]

[0115] Among them, P ij M represents the probability that the depth value dj of pixel j belongs to depth category i. i σ is the center point depth value of depth category i, and σ is a pre-set hyperparameter.

[0116] Optionally, the training module 404 includes:

[0117] The image input module is used to randomly extract the predicted probability of each pixel belonging to each depth category from the training image input depth prediction model;

[0118] The loss rate calculation module is used to calculate the loss rate using the predicted probability and the label value of the pixel.

[0119] The judgment module is used to determine whether the loss rate is less than a preset threshold;

[0120] The stop training module is used to stop training the deep prediction model and obtain the trained deep prediction model.

[0121] The parameter adjustment module is used to adjust the model parameters of the depth prediction model using the loss rate and then return the result to the image input module.

[0122] Optionally, the loss rate calculation module includes:

[0123] The loss rate calculation unit is used to calculate the loss rate using the following loss function:

[0124] The loss rate is calculated using the following loss function:

[0125]

[0126] Among them, pred j The label represents the predicted probability of pixel j. j Let be the label value of pixel j, and N be the number of pixels in the training image.

[0127] The deep prediction model training device provided in this embodiment of the invention can execute the deep prediction model training method provided in Embodiment 1 or Embodiment 2 of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0128] Example 5

[0129] Figure 5 This is a schematic diagram of a depth prediction device provided in Embodiment 5 of the present invention. The depth prediction device of this embodiment may specifically include the following modules:

[0130] The image acquisition module 501 is used to acquire the image to be predicted.

[0131] The prediction module 502 is used to input the image to be predicted into a pre-trained depth prediction model to obtain the confidence level of each pixel in the image to be predicted belonging to each depth category.

[0132] Maximum confidence level determination module 503 is used to determine the maximum confidence level from each confidence level;

[0133] The prediction result determination module 504 is used to take the maximum confidence score and the depth category corresponding to the maximum confidence score as the final prediction result;

[0134] The depth prediction model is trained using the depth prediction model training method described in Example 1 or Example 2.

[0135] The depth prediction device provided in this embodiment of the invention can execute the depth prediction method provided in Embodiment 3 of the invention, and has the corresponding functional modules and beneficial effects of the method.

[0136] Example 6

[0137] Figure 6 A schematic diagram of an electronic device 60 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0138] like Figure 6 As shown, the electronic device 60 includes at least one processor 61 and a memory, such as a read-only memory (ROM) 62 and a random access memory (RAM) 63, communicatively connected to the at least one processor 61. The memory stores computer programs executable by the at least one processor. The processor 61 can perform various appropriate actions and processes based on the computer program stored in the ROM 62 or loaded into the RAM 63 from storage unit 68. The RAM 63 may also store various programs and data required for the operation of the electronic device 60. The processor 61, ROM 62, and RAM 63 are interconnected via a bus 64. An input / output (I / O) interface 65 is also connected to the bus 64.

[0139] Multiple components in electronic device 60 are connected to I / O interface 65, including: input unit 66, such as keyboard, mouse, etc.; output unit 67, such as various types of monitors, speakers, etc.; storage unit 68, such as disk, optical disk, etc.; and communication unit 69, such as network card, modem, wireless transceiver, etc. Communication unit 69 allows electronic device 60 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0140] Processor 61 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 61 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 61 performs the various methods and processes described above, such as deep prediction model training methods, and / or, deep prediction methods.

[0141] In some embodiments, the depth prediction model training method, and / or the depth prediction method, may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 68. In some embodiments, part or all of the computer program may be loaded into and / or installed on electronic device 60 via ROM 62 and / or communication unit 69. When the computer program is loaded into RAM 63 and executed by processor 61, one or more steps of the depth prediction model training method, and / or the depth prediction method described above may be performed. Alternatively, in other embodiments, processor 61 may be configured to execute the depth prediction model training method, and / or the depth prediction method, by any other suitable means (e.g., by means of firmware).

[0142] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0143] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0144] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0145] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0146] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0147] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0148] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0149] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A deep prediction model training method, characterized in that, include: Obtain a training image set, wherein each pixel in the training image set is associated with a depth value; The preset depth prediction range is divided into multiple consecutive depth categories; For each training image, the probability that the pixel belongs to each depth category is calculated based on the depth value, and the probability is used as the label value of the pixel belonging to each depth category; A depth prediction model is trained using the training images and the label values. The depth prediction model is used to output the probability that the pixel belongs to each of the depth categories, wherein the probability corresponding to the depth category is the confidence level that the pixel belongs to the depth category. For each training image, calculating the probability that a pixel belongs to each depth category based on the depth value, and using the probability as the label value of the pixel belonging to each depth category, includes: Calculate the center point depth value for each of the depth categories; The probability that a pixel belongs to the depth category is calculated using the following formula: Among them, P ij M represents the probability that the depth value dj of pixel j belongs to depth category i. i σ is the center point depth value of depth category i, and σ is a pre-set hyperparameter.

2. The deep prediction model training method as described in claim 1, characterized in that, The step of dividing the preset depth prediction range into multiple consecutive depth categories includes: The depth prediction range is divided into multiple consecutive, equidistant depth categories according to a preset distance.

3. The deep prediction model training method as described in claim 1, characterized in that, The calculation of the center point depth value for each depth category includes: Obtain the starting depth value and ending depth value of the depth category; Calculate the difference between the endpoint depth value and the starting depth value; Calculate the product of the difference and 0.5; The center point depth value of the depth category is obtained by summing the product of the initial depth value and the product.

4. The deep prediction model training method according to any one of claims 1-3, characterized in that, The step of training a depth prediction model using the training images and the label values ​​includes: The training images are randomly extracted and input into the depth prediction model to obtain the predicted probability of each pixel belonging to each depth category; The loss rate is calculated using the predicted probability and the label value of the pixel. Determine whether the loss rate is less than a preset threshold; If so, stop training the depth prediction model and obtain the trained depth prediction model; If not, adjust the model parameters of the depth prediction model using the loss rate, and return to the step of randomly extracting the training image and inputting it into the depth prediction model to obtain the predicted probability of each pixel belonging to each depth category.

5. The deep prediction model training method as described in claim 4, characterized in that, The calculation of the loss rate using the predicted probability and the label value of the pixel includes: The loss rate is calculated using the following loss function: Among them, pred j The label represents the predicted probability of pixel j. j Let be the label value of pixel j, and N be the number of pixels in the training image.

6. A depth prediction method, characterized in that, include: Obtain the image to be predicted; The image to be predicted is input into a pre-trained depth prediction model to obtain the confidence score of each pixel in the image to be predicted belonging to each depth category. Determine the maximum confidence level from among the various confidence levels; The maximum confidence level and the depth category corresponding to the maximum confidence level are used as the final prediction result; The depth prediction model is trained using the depth prediction model training method described in any one of claims 1-5.

7. A deep prediction model training device, characterized in that, include: The training image set acquisition module is used to acquire a training image set, wherein each pixel of the training image set is associated with a depth value; The depth category segmentation module is used to divide the preset depth prediction range into multiple consecutive depth categories; The annotation module is used to calculate the probability that a pixel belongs to each depth category based on the depth value for each training image, and use the probability as the label value of the pixel belonging to each depth category. The training module is used to train a depth prediction model using the training image and the label value. The depth prediction model is used to output the probability that the pixel belongs to each depth category, wherein the probability corresponding to the depth category is the confidence level that the pixel belongs to the depth category. The annotation module includes: The center point depth value calculation module is used to calculate the center point depth value for each of the depth categories; The probability calculation module is used to calculate the probability of each pixel belonging to the depth category for each pixel in each training image, using the pixel's depth value, the center point's depth value, and preset hyperparameters, as the pixel's label value. The probability calculation module includes: The probability calculation unit is used to calculate the probability that the pixel belongs to the depth category using the following formula: Among them, P ij M represents the probability that the depth value dj of pixel j belongs to depth category i. i σ is the center point depth value of depth category i, and σ is a pre-set hyperparameter.

8. A depth prediction device, characterized in that, include: The image acquisition module is used to acquire the image to be predicted. The prediction module is used to input the image to be predicted into a pre-trained depth prediction model to obtain the confidence level of each pixel in the image to be predicted belonging to each depth category. The maximum confidence level determination module is used to determine the maximum confidence level from among the various confidence levels; The prediction result determination module is used to take the maximum confidence score and the depth category corresponding to the maximum confidence score as the final prediction result; The depth prediction model is trained using the depth prediction model training method described in any one of claims 1-5.

9. An electronic device, characterized in that, The electronic device includes: One or more processors; Memory, used to store one or more computer programs; When the one or more computer programs are executed by the one or more processors, the one or more processors implement the deep prediction model training method as described in any one of claims 1-5, and / or the deep prediction method as described in claim 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the deep prediction model training method as described in any one of claims 1-5, and / or the deep prediction method as described in claim 6.

Images