Depth prediction method and system, electronic equipment and vehicle

Abstract

The invention provides a depth prediction method and system, electronic equipment and a vehicle, and the method comprises the steps: determining a first probability value and a first offset of a first target in a first depth interval based on a pre-trained depth prediction model according to a first image, shot by a camera device, of the first target around the vehicle, and a second probability value and a second offset of the first target in a second depth interval; determining a first depth value of the first target according to the first offset and the distance between the first depth interval and the camera device; determining a second depth value of the first target according to the second offset and the distance between the second depth interval and the camera device; and determining the depth value of the first target based on the first probability value, the second probability value, the first depth value and the second depth value. The accuracy of determining the distance between the vehicle and the surrounding target can be improved.

Description

Technical Field

[0001] This application relates to the field of depth prediction technology, and in particular to a depth prediction method, system, electronic device, and vehicle. Background Technology

[0002] With the continuous growth in the number of motor vehicles, issues related to driving safety and parking safety have received widespread attention. During vehicle operation, accidents can occur due to vehicles being too close to surrounding objects. For example, if a pedestrian is in the driver's blind spot, the vehicle may collide with the pedestrian if it fails to stop in time; similarly, if a vehicle is too close to an object ahead and fails to stop in time, a rear-end collision may occur. Currently, the distance between a vehicle and surrounding objects is typically predicted using images captured by onboard cameras. However, this method of predicting the distance from a target to a vehicle does not provide fine-grained segmentation of the distance between the target and the vehicle, resulting in low accuracy in predicting the distance from the target to the vehicle. Summary of the Invention

[0003] In view of the above, it is necessary to propose a depth prediction method, system, electronic device and vehicle to solve the technical problem of low accuracy in predicting the distance between the vehicle and surrounding targets.

[0004] This application provides a depth prediction method applied to an electronic device communicatively connected to a camera device in a vehicle. The method includes: based on a first image of a first target surrounding the vehicle captured by the camera device, determining a first probability value and a first offset of the first target being in a first depth interval, and a second probability value and a second offset of the first target being in a second depth interval, based on a pre-trained depth prediction model; determining a first depth value of the first target based on the first offset, the distance between the first depth interval and the camera device; determining a second depth value of the first target based on the second offset, the distance between the second depth interval and the camera device; and determining a depth value of the first target based on the first probability value, the second probability value, the first depth value, and the second depth value.

[0005] In some embodiments, determining the depth value of the first target based on the first probability value, the second probability value, the first depth value, and the second depth value includes: determining the absolute value of the difference between the first probability value and the second probability value; if the absolute value is less than a preset threshold, determining a first weight for the first depth value and a second weight for the second depth value based on the first probability value and the second probability value; performing a weighted summation calculation on the first depth value and the second depth value based on the first weight and the second weight to obtain the depth value of the first target; if the absolute value is greater than the preset threshold and the first probability value is greater than the second probability value, determining the first depth value as the depth value of the first target; and if the absolute value is greater than the preset threshold and the second probability value is greater than the first probability value, determining the second depth value as the depth value of the first target.

[0006] In some embodiments, the method further includes determining the first depth interval and the second depth interval, wherein determining the first depth interval and the second depth interval includes: determining, based on the first image and a pre-trained depth prediction model, the probability value of the first target being located in a plurality of preset depth intervals; and determining the first depth interval and the second depth interval from the plurality of depth intervals based on the probability values ​​in descending order.

[0007] In some embodiments, determining the first depth value of the first target based on the first offset, the distance between the first depth interval and the camera device includes: determining the first predicted depth of the first target based on the offset of the first target in the first depth interval and the distance between the first depth interval and the camera device; updating the first predicted depth based on the intrinsic parameters of the camera device to obtain the first depth value.

[0008] In some embodiments, determining the second depth value of the first target based on the second offset, the distance between the second depth interval and the camera device includes: determining the second predicted depth of the first target based on the offset of the first target in the second depth interval and the distance between the second depth interval and the camera device; updating the second predicted depth based on the intrinsic parameters of the camera device to obtain the second depth value.

[0009] In some embodiments, the method further includes training the depth prediction model, wherein training the depth prediction model includes: constructing labeled data based on pre-acquired sample images; wherein the sample images include a second target; determining, based on the sample images and a pre-constructed first model, the predicted probability value and predicted offset of the second target in multiple depth intervals; determining the loss value of the first model based on the labeled data, the predicted probability value, and the predicted offset; and obtaining a depth prediction model trained to a convergent state when the loss value satisfies a preset condition.

[0010] In some embodiments, constructing labeled data based on pre-acquired sample images includes: determining multiple depth intervals within the imaging range of the camera device according to a preset interval distance; determining the true offset of the second target relative to the midpoint of any one of the multiple depth intervals based on the distance between the second target and the camera device; determining the labeling probability value of the second target being located in any one of the depth intervals based on the true offset; and determining the labeled data based on the true offset and the labeling probability value.

[0011] In some embodiments, determining the loss value of the first model based on the labeled data, the predicted probability value, and the predicted offset includes: determining a first loss value of the first model based on the cross-entropy between the predicted probability value and the labeled probability value, and based on the difference between the predicted offset and the true offset; determining a third depth interval and a fourth depth interval from the plurality of depth intervals based on the predicted probability values ​​in descending order; determining a second loss value of the first model based on the third predicted offset of the second target in the third depth interval and the fourth predicted offset of the second target in the fourth depth interval; determining a third loss value of the first model based on the third predicted offset, the fourth predicted offset, the intrinsic parameters of the camera device, and the pre-acquired morphological information of the second target; and determining the loss value of the first model based on the first loss value, the second loss value, and the third loss value.

[0012] In some embodiments, determining the second loss value of the first model based on the third predicted offset of the second target in the third depth interval and the fourth predicted offset of the second target in the fourth depth interval includes: normalizing the distance between the second target and the camera device based on the intrinsic parameters of the camera device and preset normalization parameters to obtain a normalized depth value of the second target; determining the third predicted depth of the second target based on the third predicted offset, the distance between the third depth interval and the camera device; determining the fourth predicted depth of the second target based on the fourth predicted offset, the distance between the fourth interval and the camera device; and determining the second loss value based on the difference between the normalized depth value and the third predicted depth, and the difference between the normalized depth value and the fourth predicted depth.

[0013] In some embodiments, determining the third loss value of the first model based on the third predicted offset, the fourth predicted offset, the intrinsic parameters of the camera device, and the pre-acquired morphological information of the second target includes: updating the third predicted depth and the fourth predicted depth based on the intrinsic parameters to obtain a third depth value corresponding to the third predicted depth and a fourth depth value corresponding to the fourth predicted depth; determining the first three-dimensional coordinates of the center point of the second target in the three-dimensional coordinate system corresponding to the camera device based on the third depth value, the fourth depth value, the first two-dimensional coordinates of the center point of the second target in the sample image, and the intrinsic parameters; determining the second three-dimensional coordinates of the vertex of the second target in the three-dimensional coordinate system based on the morphological information and the first three-dimensional coordinates; determining the projected coordinates of the vertex of the second target in the sample image based on the intrinsic parameters and the second three-dimensional coordinates; and determining the third loss value based on the projected coordinates and the second two-dimensional coordinates of the second target in the sample image.

[0014] This application embodiment also provides a depth prediction system, the system including an electronic device and a camera device, the electronic device being communicatively connected to the camera device, the camera device being used to capture a first target around a vehicle; the electronic device being used to determine, based on a pre-trained depth prediction model, a first probability value and a first offset of the first target being in a first depth interval, and a second probability value and a second offset of the first target being in a second depth interval, based on a first image of the first target captured by the camera device; a first depth value of the first target being determined based on the first offset, the distance between the first depth interval and the camera device; and a second depth value of the first target being determined based on the second offset, the distance between the second depth interval and the camera device; and a depth value of the first target being determined based on the first probability value, the second probability value, the first depth value, and the second depth value.

[0015] This application also provides an electronic device, which includes a processor and a storage device. The processor is used to implement the depth prediction method by executing a computer program stored in the storage device.

[0016] This application also provides a vehicle that includes the aforementioned electronic device.

[0017] As can be seen from the above technical solutions, the embodiments of this application, by outputting the probability values ​​of the first target being in different depth intervals, can transform the depth prediction problem into a probability distribution problem of discrete intervals, avoiding the problem of being unable to assess the confidence level of the prediction results due to directly predicting depth values. Furthermore, the probability values ​​can serve as the basis for determining the depth prediction results, reducing the error in the predicted depth values ​​caused by outputting a single numerical value. In addition, the offset corresponding to each depth interval can be used to correct the deviation between the center of the depth interval and the actual depth. Based on the offset and the intrinsic parameters of the camera device, perspective projection maps the two-dimensional image features to three-dimensional space, which can compensate for the prediction error of the depth prediction model. Moreover, by fusing the results of two depth intervals using probability values, the influence of the depth values ​​in the high-probability interval on the depth prediction results is increased, ensuring that the depth prediction model prioritizes more reliable estimates in high-confidence depth intervals, reducing the impact of outliers on the prediction results, thereby improving the accuracy of predicting the depth from the target to the vehicle. Attached Figure Description

[0018] Figure 1 This is an application scenario diagram of a depth prediction method provided in an embodiment of this application.

[0019] Figure 2 This is a flowchart of a depth prediction method provided in an embodiment of this application.

[0020] Figure 3 This is a flowchart of a method for training a depth prediction model according to an embodiment of this application.

[0021] Figure 4 This is a flowchart of a method for constructing labeled data provided in another embodiment of this application.

[0022] Figure 5 This is a flowchart of a method for determining the loss value of a first model according to an embodiment of this application.

[0023] Figure 6 This is a flowchart of a method for determining a second loss value provided in an embodiment of this application.

[0024] Figure 7 This is a flowchart of a method for determining a third loss value provided in an embodiment of this application.

[0025] Figure 8 This is a flowchart of a method for determining a first depth range and a second depth range provided in an embodiment of this application.

[0026] Figure 9 This is a flowchart of a method for determining a first depth value of a first target according to an embodiment of this application.

[0027] Figure 10 This is a flowchart of a method for determining a second depth value of a first target according to an embodiment of this application.

[0028] Figure 11 This is a flowchart of a method for determining the depth value of a first target according to an embodiment of this application.

[0029] Figure 12 This is a functional block diagram of a depth prediction device provided in an embodiment of this application.

[0030] Figure 13 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0031] To better understand the purpose, features, and advantages of this application, a detailed description of the application is provided below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of this application can be combined with each other. Numerous specific details are set forth in the following description to provide a thorough understanding of this application; the described embodiments are only a part of the embodiments of this application, and not all of them.

[0032] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0033] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0034] This application provides a depth prediction method that can be applied to one or more electronic devices. An electronic device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0035] Electronic devices can be any electronic product that allows human-computer interaction with a customer, such as personal computers, tablets, smartphones, personal digital assistants (PDAs), game consoles, interactive network television (IPTV), smart wearable devices, etc.

[0036] Electronic devices may also include network devices and / or client devices. The network devices include, but are not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers.

[0037] The networks in which electronic devices are located include, but are not limited to, the Internet, wide area networks, metropolitan area networks, local area networks, and virtual private networks (VPNs).

[0038] like Figure 1The diagram illustrates an application scenario of a depth prediction method provided in an embodiment of this application. This depth prediction method can be applied to an electronic device 100. The electronic device 100 can be an in-vehicle chip, or an electronic device with data processing capabilities within a vehicle network server connected to the vehicle. This application does not limit the specific form of the electronic device 100. Specifically, the electronic device is communicatively connected to a camera device 300 in the vehicle 200. The camera device 300 is used to capture images of a first target (e.g., ...) around the vehicle 200. Figure 1 Pedestrian 401 and obstacle 402 are shown.

[0039] To finely divide the depth range between the first target and the camera device 300, multiple depth ranges can be divided within the imaging range of the camera device 300 based on preset interval distances. Specifically, the starting point can be the location of the camera device 300, and the ending point can be the farthest imaging distance of the camera device 300, with each preset interval distance representing a depth range. For example, Figure 1 The depth intervals 501, 502, 503, 504 and 505 shown in this application do not limit the interval distance or the number of depth intervals.

[0040] In this application, after receiving a first image of a first target surrounding a vehicle 200 captured by a camera device 200, the electronic device 100 determines, based on a pre-trained depth prediction model, a first probability value and a first offset of the first target being within a first depth interval, and a second probability value and a second offset of the first target being within a second depth interval. The electronic device 100 further determines a first depth value of the first target based on the first offset, the distance between the first depth interval and the camera device 300, and a second depth value of the first target based on the second offset, the distance between the second depth interval and the camera device 300. The first and second depth intervals can be any depth interval within the imaging range of the camera device 300. The electronic device 100 also determines a depth value of the first target based on the first probability value, the second probability value, the first depth value, and the second depth value. The depth value indicates the distance between the first target and the camera device 300.

[0041] like Figure 2 The diagram shown is a flowchart of a depth prediction method according to an embodiment of this application. The order of the steps in the flowchart can be changed, and some steps can be omitted, depending on different requirements. The depth prediction method provided in this embodiment includes the following steps.

[0042] S20, based on the first image of the first target around the vehicle captured by the camera device, and based on a pre-trained depth prediction model, determine the first probability value and first offset of the first target being in a first depth range, and the second probability value and second offset of the first target being in a second depth range.

[0043] In one embodiment of this application, to identify the distance between a first target near a vehicle and the vehicle, a first image of the first target captured by an onboard camera device can be analyzed based on a pre-trained depth prediction model. The model determines a first probability value and a first offset of the first target being in a first depth interval, and a second probability value and a second offset of the first target being in a second depth interval, based on image information in the first image. The first target can be a pedestrian, obstacle, other vehicle, or plant / animal near the vehicle; the specific type of the first target is not limited in this application. The depth prediction model can be a convolutional neural network model, a transfer learning model, or a residual neural network model; this application does not limit the specific type. Specifically, the depth prediction model is used to determine the probability value of the first target in the first image being in multiple pre-set depth intervals, and the offset of the first target in each depth interval, based on image information in the first image. Thus, the distance between the first target and the onboard camera device is represented in the form of quantitative data based on the probability value and offset, improving the accuracy of depth prediction for objects near the vehicle.

[0044] For example, when the first probability value of the first target being in the first depth interval is 0.9, it indicates that the probability of the first target being within the first depth interval is 90%; when the corresponding first offset is 0.1, it indicates that the distance between the first target and the midpoint of the first depth interval can be 0.1 meters; when the second probability value of the first target being in the second depth interval is 0.8, it indicates that the probability of the first target being within the second depth interval is 80%; when the corresponding second offset is 0.3, it indicates that the distance between the first target and the midpoint of the second depth interval can be 0.3 meters.

[0045] In one embodiment of this application, to improve the accuracy of determining the depth value of the first target, a depth prediction model is trained based on pre-constructed labeled data, and the ability of the depth prediction model to resolve images captured by the camera device is improved. Specifically, for the method of training the depth prediction model, please refer to [link to relevant documentation]. Figure 3 , Figure 4 , Figure 5 , Figure 6 as well as Figure 7 The corresponding detailed explanation.

[0046] In one embodiment of this application, to improve the accuracy of characterizing the depth value of the first target based on the first depth interval and the second depth interval, multiple depth intervals can be pre-divided within the imaging range of the camera device at a preset interval distance. Based on the probability value of the first target in each depth interval, the first depth interval and the second depth interval are selected from the multiple depth intervals. Specifically, when sorting the probability values ​​in descending order, the first probability value can be the first-ranked probability value, and the second probability value can be the second-ranked probability value. For details on how to determine the first depth interval and the second depth interval, please refer to [link to relevant documentation]. Figure 8 The corresponding detailed explanation.

[0047] S21, determine a first depth value of the first target based on the first offset, the distance between the first depth interval and the camera device; and determine a second depth value of the first target based on the second offset, the distance between the second depth interval and the camera device.

[0048] In one embodiment of this application, to improve the accuracy of predicting the depth value of a first target, a first depth value corresponding to the first target being in a first depth range can be determined based on a first offset; and a second depth value corresponding to the first target being in a second depth range can be determined based on a second offset. This simulates the distance between the first target and the camera device when the first target is in a depth range with a high probability, thereby improving the accuracy of predicting the depth value of the first target.

[0049] In one embodiment of this application, the specific method for determining the first depth value of the first target based on the first offset, the first depth interval, and the distance between the camera device is described in the following article. Figure 9 The corresponding detailed explanation.

[0050] In one embodiment of this application, the specific method for determining the second depth value of the first target based on the second offset, the distance between the second depth range and the camera device is described in the following article. Figure 10 The corresponding detailed explanation.

[0051] S22, based on the first probability value, the second probability value, the first depth value, and the second depth value, determine the depth value of the first target.

[0052] In one embodiment of this application, a first depth value can be used to indicate the distance between the first target and the camera device when the first target is within a first depth range; a second depth value can be used to indicate the distance between the first target and the camera device when the first target is within a second depth range. To reduce the error in predicting the depth value of the first target based on the data output by the depth prediction model, the depth value of the first target can be determined jointly based on the first depth value and the second depth value. This can improve the accuracy of predicting the depth value of the first target.

[0053] In cases where the first probability value is greater than the second probability value and the difference between the first probability value and the second probability value is large, it indicates that the first target is more likely to be in the first depth range. Therefore, when determining the depth value of the first target, the similarity between the depth value of the first target and the first depth value is high.

[0054] In cases where the first probability value is less than the second probability value and the difference between the first and second probability values ​​is large, it indicates that the first target is more likely to be in the second depth range. Therefore, when determining the depth value of the first target, the similarity between the depth value of the first target and the second depth value is relatively high.

[0055] When the difference between the first probability value and the second probability value is small, it indicates that the probability of the first target being in the first depth interval is similar to the probability of it being in the second depth interval. Therefore, the weights of the first and second depth values ​​can be determined based on the first and second probability values, thereby adjusting their importance to improve the accuracy of determining the depth value of the first target. A higher weight for the first depth value indicates a higher similarity between the first target's depth value and the first depth value; similarly, a higher weight for the second depth value indicates a higher similarity between the first target's depth value and the second depth value.

[0056] Specifically, for the detailed method of determining the depth value of the first target based on the first probability value, the second probability value, the first depth value, and the second depth value, please refer to [link to relevant documentation]. Figure 11 The corresponding detailed explanation.

[0057] As can be seen from the above technical solutions, the embodiments of this application, by outputting the probability values ​​of the first target being in different depth intervals, can transform the depth prediction problem into a probability distribution problem of discrete intervals, avoiding the problem of being unable to assess the confidence level of the prediction results due to directly predicting depth values. Furthermore, the probability values ​​can serve as the basis for determining the depth prediction results, reducing the error in the predicted depth values ​​caused by outputting a single numerical value. In addition, the offset corresponding to each depth interval can be used to correct the deviation between the center of the depth interval and the actual depth. Based on the offset and the intrinsic parameters of the camera device, perspective projection maps the two-dimensional image features to three-dimensional space, which can compensate for the prediction error of the depth prediction model. Moreover, by fusing the results of two depth intervals using probability values, the influence of the depth values ​​in the high-probability interval on the depth prediction results is increased, ensuring that the depth prediction model prioritizes more reliable estimates in high-confidence depth intervals, reducing the impact of outliers on the prediction results, thereby improving the accuracy of predicting the depth from the target to the vehicle.

[0058] like Figure 3 The diagram shown is a flowchart of a method for training a depth prediction model according to an embodiment of this application. The order of steps in this flowchart can be changed, and some steps can be omitted, depending on different requirements. The method for training a depth prediction model provided in this embodiment includes the following steps.

[0059] S30, based on pre-acquired sample images, construct labeled data; wherein, the sample images include a second target.

[0060] In one embodiment of this application, to improve the performance of the depth prediction model and thus the accuracy of predicting the depth value of the first target, the depth prediction model can be trained based on pre-acquired sample images to reduce the errors in the probability values ​​and offsets output by the depth prediction model, thereby improving the accuracy of predicting the depth value of the first target. The sample images include a second target. For example, the second target can be pedestrians, obstacles, other vehicles, or plants and animals around the vehicle; the specific type of the second target is not limited in this application. Specifically, labeled data can be constructed based on the sample images to provide data support for subsequent training of the depth prediction model. The labeled data includes the labeled probability value of the second target being in any one of multiple depth intervals, and the true offset of the second target from the midpoint of any depth interval. For details on how to construct the labeled data, please refer to [link to relevant documentation]. Figure 4 The corresponding detailed explanation.

[0061] S31, Based on the sample image and a pre-built first model, determine the predicted probability value and predicted offset of the second target in multiple depth intervals.

[0062] In one embodiment of this application, the pre-constructed first model can be a convolutional neural network model, a transfer learning model, or a residual neural network model; this application does not limit this. Specifically, the first model is used to determine, based on image information in the sample image, the predicted probability value of the second target in the sample image being located in multiple pre-set depth intervals, and the predicted offset of the second target being located in each depth interval. Thus, based on the predicted probability value and the predicted offset, the distance between the second target predicted by the first model and the vehicle-mounted camera device is characterized in the form of quantified data.

[0063] For example, when the predicted probability value of the second target being in any depth interval is 0.9, it indicates that the probability of the second target being located within that depth interval predicted by the first model is 90%; when the corresponding prediction offset is 0.1, it indicates that the distance between the second target and the midpoint of that depth interval can be 0.1 meters; when the predicted probability value of the second target being in any depth interval is 0.8, it indicates that the probability of the second target being located within that depth interval is 80%; when the corresponding second offset is 0.3, it indicates that the distance between the second target and the midpoint of that depth interval can be 0.3 meters.

[0064] S32, Based on the labeled data, the predicted probability value, and the predicted offset, determine the loss value of the first model.

[0065] In one embodiment of this application, the loss value of a first model can be determined based on labeled data, predicted probability values, and predicted offsets. This allows for the determination of the differences between the predicted probability values ​​and predicted offsets and the actual data based on the labeled data. The accuracy of the prediction results output by the first model can be quantitatively characterized based on the loss value, thereby continuously updating the first model according to the loss value to improve its performance. Specifically, for the method of determining the loss value of the first model, please refer to [link to relevant documentation]. Figure 5 , Figure 6 and Figure 7 The corresponding detailed explanation.

[0066] S33, when the loss value meets the preset conditions, a deep prediction model trained to the convergence state is obtained.

[0067] In one embodiment of this application, if the loss value does not meet a preset condition, it indicates that the difference between the predicted result output by the first model and the true value is large, and the first model can be continuously updated based on the backpropagation algorithm; if the loss value meets the preset condition, it indicates that the difference between the predicted result output by the first model and the true value is small, and the updating of the first model can be stopped, thus obtaining a deep prediction model trained to a convergent state. For example, the preset condition may be that the loss value is less than a preset termination threshold, wherein the termination threshold may be 0.01, 0.02, etc., and this application does not limit the value of the termination threshold.

[0068] like Figure 4 The diagram shown is a flowchart of a method for constructing labeled data according to an embodiment of this application. The order of steps in this flowchart can be changed, and some steps can be omitted, depending on different requirements. The method for constructing labeled data provided in this embodiment includes the following steps.

[0069] S40, according to the preset interval distance, determine multiple depth intervals within the imaging range of the camera device.

[0070] In one embodiment of this application, in order to divide the depth range of the distance between the second target and the camera device with a smaller granularity, multiple depth ranges can be divided within the imaging range of the camera device based on a preset range distance.

[0071] In one embodiment of this application, the depth interval can be divided at preset intervals, with the starting point being the location of the camera device and the ending point being the farthest imaging distance of the camera device. For example, n depth intervals can be divided, and the calculation of n satisfies the following relationship: ; in, This represents the distance between intervals; for example, the distance between intervals could be 1 meter. This represents the furthest imaging distance of the camera device.

[0072] S41, based on the distance between the second target and the camera device, determine the actual offset of the second target relative to the midpoint of any one of the plurality of depth intervals.

[0073] In one embodiment of this application, the distance between the second target and the camera device can be normalized based on the intrinsic parameters of the camera device and preset normalization parameters. This eliminates the dimensional differences in the training data and improves the performance of the depth prediction model during training. Specifically, the depth value of the second target can be normalized according to the following relationship: ; in, Z represents the normalized depth value; Z represents the actual depth value of the second target, i.e., the distance between the second target and the camera device; f represents the focal length of the camera device. This represents the preset normalization parameter (for example, the value of scale can be 1000).

[0074] In one embodiment of this application, the offset between the normalized depth value of the second target and the midpoint of any depth interval can be determined. Specifically, the offset is calculated in a way that satisfies the following relationship: ; in, The normalized depth value representing the second target is offset from the midpoint of the depth interval with index i; i represents the index of the depth interval.

[0075] In one embodiment of this application, in order to improve the confidence level of representing the true depth value of the second target based on the offset and probability value, an attention mechanism can be simulated based on the following piecewise function: values ​​with larger offsets are set to zero, while smaller offsets are retained as the true offsets. ; in, This represents the true offset of the second target relative to the midpoint of the i-th depth interval; This represents the absolute value of the normalized depth value of the second target and the offset from the midpoint of the depth interval with index i; where i represents the index of the depth interval. If the absolute value is greater than 2, it indicates that the offset is too large, and the probability that the second target is within the depth interval corresponding to that offset is low; therefore, the offset can be set to 0.

[0076] S42, based on the true offset, determine the annotation probability value of the second target being in any of the depth intervals.

[0077] In one embodiment of this application, the annotation probability value of the second target being located in any depth interval can be determined based on the actual offset of the second target in any depth interval. Specifically, the normalized depth value of the second target... The smaller the offset from the midpoint of any depth interval, the greater the probability that the second target is located within that depth interval. Specifically, the method for determining the annotation probability value based on the actual offset satisfies the following relationship: ; in, This represents the labeled probability value that the second target is located in the i-th depth interval; This represents the absolute value of the true offset of the second target in the i-th depth interval; This represents the sum of the absolute values ​​of all true offsets, and .

[0078] In one embodiment of this application, in order to adjust the probability values ​​of the second target being in each depth interval to the same dimension, and to avoid the problem of reduced accuracy of predicted depth values ​​due to excessive differences in the dimensions of different probability values, the following can also be done: Normalization is performed, specifically, the normalization of the annotation depth values ​​follows this relationship: ; in, This represents the normalized labeled probability value corresponding to the second target being in the i-th depth interval; This represents the sum of all labeled probability values, and Among these, a higher annotation probability value indicates a greater likelihood that the second target is located within the depth range corresponding to that annotation probability value.

[0079] S43, Based on the actual offset and the annotation probability value, determine the annotation data.

[0080] In one embodiment of this application, the true offset and labeled probability value corresponding to all depth intervals can be determined as labeled data.

[0081] like Figure 5 The diagram shown is a flowchart of a method for determining the loss value of a first model according to an embodiment of this application. The order of steps in this flowchart can be changed, and some steps can be omitted, depending on different requirements. The method for determining the loss value of a first model provided in this embodiment includes the following steps.

[0082] S50, based on the cross-entropy between the predicted probability value and the labeled probability value, and based on the difference between the predicted offset and the true offset, determine the first loss value of the first model.

[0083] In one embodiment of this application, the cross-entropy between predicted probability values ​​and labeled probability values ​​can be calculated using the KL divergence formula to measure the similarity between the probability distributions of the predicted and labeled probability values. The higher the similarity between the probability distributions of the predicted and labeled probability values, the smaller the cross-entropy, and thus the smaller the first loss value of the first model. The method for determining the cross-entropy satisfies the following relationship: ; in, This represents the cross-entropy between the predicted probability value and the labeled probability value; This represents the predicted probability that the second target is located in the i-th depth interval; This represents the labeled probability value of the second target being in the i-th depth interval.

[0084] ; in, P represents the difference between the predicted offset and the true offset. i offset The predicted offset of the second target in the i-th depth interval; G i offset This represents the actual offset of the second target within the i-th depth interval.

[0085] S51, based on the order of the predicted probability values ​​from largest to smallest, determine the third depth interval and the fourth depth interval from the plurality of depth intervals; based on the third predicted offset of the second target in the third depth interval and the fourth predicted offset of the second target in the fourth depth interval, determine the second loss value of the first model.

[0086] In one embodiment of this application, when sorting the predicted probability values ​​in descending order, the third probability value can be the first-ranked predicted probability value, and the fourth probability value can be the second-ranked predicted probability value. To determine the performance of the first model based on the predicted offset corresponding to the depth interval with higher predicted probabilities, a second loss value for the first model can be determined based on the third predicted offset of the second target in the third depth interval and the fourth predicted offset of the second target in the fourth depth interval. Specifically, for the detailed method of determining the second loss value of the first model, please refer to [link to relevant documentation]. Figure 6 The corresponding detailed explanation.

[0087] S52, based on the third predicted offset, the fourth predicted offset, the intrinsic parameters of the camera device, and the morphological information of the second target obtained in advance, determine the third loss value of the first model.

[0088] In one embodiment of this application, to ensure that the depth prediction model learns the stereo information of the second target during training and has the ability to analyze the stereo information of the target in the image, the three-dimensional coordinates of the center point and multiple vertices of the second target can be determined based on the intrinsic parameters of the camera device and the pre-acquired morphological information of the second target. Then, a third loss value of the first model is determined based on the three-dimensional coordinates and their projection coordinates in the image. Specifically, for the detailed method of determining the third loss value of the first model, please refer to [link to relevant documentation]. Figure 7 The corresponding detailed explanation.

[0089] S53, based on the first loss value, the second loss value and the third loss value, determine the loss value of the first model.

[0090] In one embodiment of this application, a first loss value, a second loss value, and a third loss value can be determined as the loss value of a first model.

[0091] In one embodiment of this application, the sum of the first loss value, the second loss value, and the third loss value can also be determined as the loss value of the first model.

[0092] In one embodiment of this application, the first loss, the second loss value, and the third loss value can be weighted and summed based on preset weight parameters to obtain the loss value of the first model.

[0093] This application does not limit the specific method for determining the loss value of the first model based on the first loss value, the second loss value, and the third loss value.

[0094] like Figure 6 The diagram shown is a flowchart of a method for determining a second loss value according to an embodiment of this application. The order of the steps in this flowchart can be changed, and some steps can be omitted, depending on different requirements. The method for determining a second loss value provided in this embodiment of the application includes the following steps.

[0095] S60, based on the intrinsic parameters of the camera device and the preset normalization parameters, the distance between the second target and the camera device is normalized to obtain the normalized depth value of the second target.

[0096] In one embodiment of this application, the distance between the second target and the camera device can be normalized based on the intrinsic parameters of the camera device and preset normalization parameters. This eliminates the dimensional differences in the training data and improves the performance of the depth prediction model during training. Specifically, the depth value of the second target can be normalized according to the following relationship: ; in, Z represents the normalized depth value; Z represents the actual depth value of the second target, i.e., the distance between the second target and the camera device; f represents the focal length of the camera device. This represents the preset normalization parameter (for example, the value of scale can be 1000).

[0097] S61, determine the third predicted depth of the second target based on the third predicted offset, the distance between the third depth interval and the camera device; and determine the fourth predicted depth of the second target based on the fourth predicted offset, the distance between the fourth interval and the camera device.

[0098] In one embodiment of this application, the method for determining the third predicted depth satisfies the following relationship: ; in, represents the third prediction depth; ind3 represents the index of the third depth interval; offset3 represents the third prediction offset; where, It can be used to indicate the distance between the third depth zone and the camera device.

[0099] In one embodiment of this application, the method for determining the fourth prediction depth satisfies the following relationship: ; in, represents the fourth prediction depth; ind4 represents the index of the fourth depth interval; offset4 represents the fourth prediction offset; where, It can be used to indicate the distance between the fourth depth zone and the camera device.

[0100] S62, the second loss value is determined based on the difference between the normalized depth value and the third predicted depth, and the difference between the normalized depth value and the fourth predicted depth.

[0101] In one embodiment of this application, the method for determining the second loss value satisfies the following relationship: ; Where L2 represents the second loss value of the first model; The normalized depth value representing the second objective; Represents the third prediction depth; This represents the fourth prediction depth.

[0102] like Figure 7 The diagram shown is a flowchart of a method for determining a third loss value according to an embodiment of this application. The order of steps in this flowchart can be changed, and some steps can be omitted, depending on different requirements. The method for determining a third loss value provided in this embodiment of the application includes the following steps.

[0103] S70, based on the intrinsic parameters, update the third prediction depth and the fourth prediction depth respectively to obtain the third depth value corresponding to the third prediction depth and the fourth depth value corresponding to the fourth prediction depth.

[0104] In one embodiment of this application, the third and fourth predicted depths can be updated based on the intrinsic parameters of the camera device, thereby removing normalized information from the third and fourth predicted depths to improve the difference between them and simulate the stereoscopic information of the second target. Specifically, the method for determining the third and fourth depth values ​​satisfies the following relationship: ; Where x takes the value 3 or 4, and when x takes the value 3, This represents the third depth value, when x is 4. This represents the fourth depth value; when x is 3, This represents the third prediction depth, when x is 4. The fourth predicted depth is represented by f; f represents the focal length of the camera device. This represents the preset normalization parameter (for example, the value of scale can be 1000).

[0105] S71, based on the third depth value, the fourth depth value, the first two-dimensional coordinates of the center point of the second target in the sample image, and the intrinsic parameters, determine the first three-dimensional coordinates of the center point of the second target in the three-dimensional coordinate system corresponding to the camera device.

[0106] In one embodiment of this application, the method for determining the first three-dimensional coordinates of the center point of the second target in the three-dimensional coordinate system corresponding to the camera device satisfies the following relationship: ; in, The intrinsic parameter matrix of the camera device The inverse matrix; The first two-dimensional coordinates of the center point of the second target in the sample image; Represents the x-coordinate in the first three-dimensional coordinate system; Represents the ordinate in the first three-dimensional coordinate system; Represents the vertical coordinate in the first three-dimensional coordinate system.

[0107] S72, based on the morphological information and the first three-dimensional coordinates, determine the second three-dimensional coordinates of the vertex of the second target in the three-dimensional coordinate system.

[0108] In one embodiment of this application, the morphological information of the second target includes information such as the size and orientation angle of the second target, which can be based on the first three-dimensional coordinates. Determine the second three-dimensional coordinates of multiple vertices of the second target in the three-dimensional coordinate system of the camera device. Specifically, the second three-dimensional coordinates of the multiple vertices can be... Where j represents the index of the vertex of the second target.

[0109] S73, based on the intrinsic parameters and the second three-dimensional coordinates, determine the projection coordinates of the vertex of the second target in the sample image.

[0110] In one embodiment of this application, the projected coordinates of multiple vertices of the second target in the sample image can be determined based on the intrinsic parameter matrix K of the camera device. .

[0111] S74, Based on the projected coordinates and the second two-dimensional coordinates of the second target in the sample image, determine the third loss value.

[0112] In one embodiment of this application, a third loss value can be determined based on the projected coordinates and the second two-dimensional coordinates of the second target in the sample image. The second two-dimensional coordinates indicate the position of the pixels occupied by the vertices of the second target in the sample image. For example, the second two-dimensional coordinates of multiple vertices can be denoted as... .

[0113] In one embodiment of this application, the method for determining the third loss value satisfies the following relationship: .

[0114] like Figure 8 The diagram shown is a flowchart of a method for determining a first depth interval and a second depth interval according to an embodiment of this application. The order of steps in this flowchart can be changed, and some steps can be omitted, depending on different requirements. The method for determining a first depth interval and a second depth interval provided in this embodiment of the application includes the following steps.

[0115] S80, based on the first image and a pre-trained depth prediction model, determine the probability value of the first target being located in a preset plurality of depth intervals.

[0116] In one embodiment of this application, the probability value of a first target located in multiple depth intervals, determined based on a depth prediction model, can be denoted as: ,in Where i is the index of the depth interval.

[0117] For example, when the probability value of the second target being within the depth interval with index 1 is 0.9, it indicates that the probability of the first target being within that depth interval predicted by the depth prediction model is 90%. When the probability of the second target being within the depth range at index 2 is 0.8, it indicates that the probability of the second target being within that depth range is 80%. .

[0118] S81, based on the probability values ​​in descending order, determine the first depth interval and the second depth interval from the plurality of depth intervals.

[0119] In one embodiment of this application, the depth interval with the highest probability value can be determined as the first depth interval based on the order of probability values ​​from largest to smallest, and the depth interval with the second highest probability value can be determined as the second depth interval.

[0120] like Figure 9The diagram shown is a flowchart of a method for determining a first depth value of a first target according to an embodiment of this application. The order of steps in this flowchart can be changed, and some steps can be omitted, depending on different requirements. The method for determining the first depth value of the first target provided in this embodiment includes the following steps.

[0121] S90, determine the first predicted depth of the first target based on the offset of the first target in the first depth range and the distance between the first depth range and the camera device.

[0122] In one embodiment of this application, the method for determining the first predicted depth of the first target satisfies the following relationship: ; in, This represents the first predicted depth value; represents the distance between the first depth range and the camera device; offset1 represents the offset of the first target in the first depth range.

[0123] S91, update the first predicted depth based on the intrinsic parameters of the camera device to obtain the first depth value.

[0124] In one embodiment of this application, the first predicted depth can be updated based on the intrinsic parameters of the camera device, thereby removing normalized information from the first predicted depth to simulate the three-dimensional information of the first target. Specifically, the method for determining the first depth value satisfies the following relationship: ; in, Represents the first depth value. f represents the first predicted depth; f represents the focal length of the camera device. This represents the preset normalization parameter (for example, the value of scale can be 1000).

[0125] like Figure 10 The diagram shown is a flowchart of a method for determining a second depth value of a first target according to an embodiment of this application. The order of steps in this flowchart can be changed, and some steps can be omitted, depending on different requirements. The method for determining the second depth value of the first target according to an embodiment of this application includes the following steps.

[0126] S100, determine the second predicted depth of the first target based on the offset of the first target in the second depth range and the distance between the second depth range and the camera device.

[0127] In one embodiment of this application, the method for determining the second predicted depth of the first target satisfies the following relationship: ; in, This represents the second predicted depth value; represents the distance between the second depth range and the camera device; offset2 represents the offset of the first target in the second depth range.

[0128] S101, update the second predicted depth based on the intrinsic parameters of the camera device to obtain the second depth value.

[0129] In one embodiment of this application, the second predicted depth can be updated based on the intrinsic parameters of the camera device, thereby removing normalized information from the second predicted depth to simulate the stereoscopic information of the first target. Specifically, the method for determining the second depth value satisfies the following relationship: ; in, Represents the second depth value. f represents the second predicted depth; f represents the focal length of the camera device. This represents the preset normalization parameter (for example, the value of scale can be 1000).

[0130] like Figure 11 The diagram shows a flowchart of a method for determining the depth value of the first target according to an embodiment of this application. The order of steps in this flowchart can be changed, and some steps can be omitted, depending on different requirements. The method for determining the depth value of the first target according to an embodiment of this application includes the following steps.

[0131] S110, determine the absolute value of the difference between the first probability value and the second probability value.

[0132] In one embodiment of this application, the depth value of the first target can be determined based on the difference between the first probability value and the second probability value.

[0133] S111, if the absolute value is less than a preset threshold, determine a first weight of the first depth value and a second weight of the second depth value based on the first probability value and the second probability value; and calculate the weighted sum of the first depth value and the second depth value based on the first weight and the second weight to obtain the depth value of the first target.

[0134] In one embodiment of this application, when the difference between the first probability value and the second probability value is small, it indicates that the probability of the first target being in the first depth interval is similar to the probability of it being in the second depth interval. Therefore, the weights of the first depth value and the second depth value can be determined based on the first probability value and the second probability value, thereby adjusting the importance of the first depth value and the second depth value to improve the accuracy of determining the depth value of the first target. Specifically, a higher weight corresponding to the first depth value indicates a higher similarity between the depth value of the first target and the first depth value; similarly, a higher weight corresponding to the second depth value indicates a higher similarity between the depth value of the first target and the second depth value. Specifically, when the absolute value is less than a preset threshold, the method for determining the depth value of the first target satisfies the following relationship: ; in, The depth value representing the first target; Represents the first depth value; cls1 represents the second depth value; cls2 represents the first probability value; cls2 represents the second probability value.

[0135] S112, if the absolute value is greater than the preset threshold and the first probability value is greater than the second probability value, the first depth value is determined to be the depth value of the first target.

[0136] In one embodiment of this application, when the first probability value is greater than the second probability value, and the difference between the first probability value and the second probability value is large, it indicates that the probability of the first target being in the first depth range is high. Therefore, when determining the depth value of the first target, the similarity between the depth value of the first target and the first depth value is high. Thus, the first depth value can be determined as the depth value of the first target.

[0137] S113, if the absolute value is greater than the preset threshold and the second probability value is greater than the first probability value, the second depth value is determined to be the depth value of the first target.

[0138] In one embodiment of this application, when the first probability value is less than the second probability value, and the difference between the first probability value and the second probability value is large, it indicates that the probability of the first target being in the second depth range is high. Therefore, when determining the depth value of the first target, the similarity between the depth value of the first target and the second depth value is high. Thus, the second depth value can be determined as the depth value of the first target.

[0139] Please see Figure 12 , Figure 12This is a schematic diagram of a depth prediction system 1200 provided in an embodiment of this application. The depth prediction system 1200 includes an electronic device 100 and a camera device 1210. The electronic device 100 is communicatively connected to the camera device 1210, which is used to capture images of a first target 1230 surrounding a vehicle 1220. Based on the first image of the first target 1230 surrounding the vehicle 1220 captured by the camera device 1210, the electronic device 100 determines, based on a pre-trained depth prediction model, a first probability value and a first offset of the first target 1230 being in a first depth range, and a second probability value and a second offset of the first target being in a second depth range. The electronic device 100 also determines a first depth value of the first target 1230 based on the first offset, the distance between the first depth range and the camera device 1210, and a second depth value of the first target 1230 based on the second offset, the distance between the second depth range and the camera device 1210. The first depth value indicates the distance between the first target 1230 and the camera device 1210 when the first target 1230 is within a first depth range; the second depth value indicates the distance between the first target 1230 and the camera device 1210 when the first target 1230 is within a second depth range. The electronic device 100 further determines the depth value of the first target 1230 based on the first probability value, the second probability value, the first depth value, and the second depth value.

[0140] Please see Figure 13 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 100 includes a memory 12 and a processor 13. The memory 12 is used to store computer-readable instructions, and the processor 13 executes the computer-readable instructions stored in the memory to implement a depth prediction method as described in any of the above embodiments.

[0141] In one embodiment of this application, the electronic device 100 further includes a bus and a computer program stored in the memory 12 and executable on the processor 13, such as a depth prediction program.

[0142] Figure 13 Only an electronic device 100 with memory 12 and processor 13 is shown; those skilled in the art will understand that... Figure 13 The structure shown does not constitute a limitation on the electronic device 100, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0143] Combination Figure 2The memory 12 in the electronic device 100 stores a plurality of computer-readable instructions to implement the depth prediction method, and the processor 13 can execute the plurality of instructions to: determine, based on a first image of a first target around the vehicle captured by the camera device, a first probability value and a first offset of the first target being in a first depth interval, and a second probability value and a second offset of the first target being in a second depth interval, based on a pre-trained depth prediction model; determine a first depth value of the first target based on the first offset, the distance between the first depth interval and the camera device; and determine a second depth value of the first target based on the second offset, the distance between the second depth interval and the camera device; and determine a depth value of the first target based on the first probability value, the second probability value, the first depth value and the second depth value.

[0144] Specifically, the processor 13's implementation method for the above instructions can be found in [reference needed]. Figure 2 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0145] Those skilled in the art will understand that the schematic diagram is merely an example of the electronic device 100 and does not constitute a limitation on the electronic device 100. The electronic device 100 may be a bus-type structure or a star-type structure. The electronic device 100 may also include more or fewer other hardware or software than shown in the diagram, or different component arrangements. For example, the electronic device 100 may also include input / output devices, network access devices, etc.

[0146] It should be noted that electronic device 100 is only an example. Other existing or future electronic products that are suitable for this application should also be included within the scope of protection of this application and are incorporated herein by reference.

[0147] The memory 12 includes at least one type of readable storage medium, which can be non-volatile or volatile. The readable storage medium includes flash memory, portable hard drives, multimedia cards, card-type memory (e.g., SD or DX memory), magnetic storage, magnetic disks, optical disks, etc. In some embodiments, the memory 12 can be an internal storage unit of the electronic device 100, such as the portable hard drive of the electronic device 100. In other embodiments, the memory 12 can also be an external storage device of the electronic device 100, such as a plug-in portable hard drive, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 100. The memory 12 can be used not only to store application software and various types of data installed on the electronic device 100, such as the code of a depth prediction program, but also to temporarily store data that has been output or will be output.

[0148] In some embodiments, the processor 13 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits packaged with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 13 is the control unit of the electronic device 100, connecting various components of the electronic device 100 via various interfaces and lines. It executes programs or modules stored in the memory 12 (e.g., executing depth prediction programs) and calls data stored in the memory 12 to perform various functions of the electronic device 100 and process data.

[0149] The processor 13 executes the operating system of the electronic device 100 and various installed applications. The processor 13 executes these applications to implement the steps in the various depth prediction method embodiments described above, for example... Figure 2 The steps are shown.

[0150] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to complete this application. The one or more modules / units may be a series of computer-readable instruction segments capable of performing a specific function, which describe the execution process of the computer program in the electronic device 100.

[0151] The integrated unit implemented as a software functional module described above can be stored in a computer-readable storage medium. This software functional module, stored in a storage medium, includes several instructions to cause a computer device (which may be a personal computer, computer equipment, or network device, etc.) or processor to execute portions of the depth prediction method described in the various embodiments of this application.

[0152] If the modules / units integrated in the electronic device 100 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware devices. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above.

[0153] The computer program includes computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory, and other memory.

[0154] Furthermore, the computer-readable storage medium may primarily include a stored program area and a stored data area, wherein the stored program area may store the operating system, an application program required for at least one function, etc.; and the stored data area may store data created based on the use of blockchain nodes, etc.

[0155] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, in... Figure 13 The symbol is represented by only one arrow, but this does not indicate that there is only one bus or one type of bus. The bus is configured to enable communication between the memory 12 and at least one processor 13, etc.

[0156] This application also provides a vehicle (not shown), which includes the aforementioned electronic device.

[0157] This application also provides a computer-readable storage medium (not shown) storing computer-readable instructions, which are executed by a processor in an electronic device to implement the depth prediction method described in any of the above embodiments.

[0158] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0159] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0160] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0161] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices described in the specification may also be implemented by a single unit or device through software or hardware. Terms such as "first," "second," etc., are used to indicate names and do not indicate any specific order.

[0162] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this application without departing from the spirit and scope of the technical solutions of this application.

Claims

1. A depth prediction method applied to an electronic device, the electronic device being communicatively connected to a camera device in a vehicle, characterized in that, The method includes: Based on the first image of the first target around the vehicle captured by the camera device, and based on a pre-trained depth prediction model, a first probability value and a first offset of the first target being in a first depth range, and a second probability value and a second offset of the first target being in a second depth range are determined. A first depth value of the first target is determined based on the first offset, the distance between the first depth range and the camera device; and a second depth value of the first target is determined based on the second offset, the distance between the second depth range and the camera device. The depth value of the first target is determined based on the first probability value, the second probability value, the first depth value, and the second depth value.

2. The depth prediction method as described in claim 1, characterized in that, Determining the depth value of the first target based on the first probability value, the second probability value, the first depth value, and the second depth value includes: Determine the absolute value of the difference between the first probability value and the second probability value; If the absolute value is less than a preset threshold, a first weight for the first depth value and a second weight for the second depth value are determined based on the first probability value and the second probability value; based on the first weight and the second weight, the first depth value and the second depth value are weighted and summed to obtain the depth value of the first target; If the absolute value is greater than the preset threshold and the first probability value is greater than the second probability value, the first depth value is determined to be the depth value of the first target. If the absolute value is greater than the preset threshold and the second probability value is greater than the first probability value, the second depth value is determined to be the depth value of the first target.

3. The depth prediction method as described in claim 1, characterized in that, The method further includes determining the first depth interval and the second depth interval, wherein determining the first depth interval and the second depth interval includes: Based on the first image, and using a pre-trained depth prediction model, the probability values ​​of the first target being located in multiple preset depth intervals are determined. Based on the probability values ​​in descending order, a first depth interval and a second depth interval are determined from the plurality of depth intervals.

4. The depth prediction method as described in claim 1, characterized in that, Determining the first depth value of the first target based on the first offset, the first depth range, and the distance between the camera device includes: The first predicted depth of the first target is determined based on the offset of the first target in the first depth range and the distance between the first depth range and the camera device. The first predicted depth is updated based on the intrinsic parameters of the camera device to obtain the first depth value.

5. The depth prediction method as described in claim 1, characterized in that, Determining the second depth value of the first target based on the second offset, the distance between the second depth range and the camera device includes: The second predicted depth of the first target is determined based on the offset of the first target in the second depth range and the distance between the second depth range and the camera device; The second predicted depth is updated based on the intrinsic parameters of the camera device to obtain the second depth value.

6. The depth prediction method as described in claim 1, characterized in that, The method further includes training the deep prediction model, wherein training the deep prediction model includes: Annotated data is constructed based on pre-acquired sample images; wherein, the sample images include a second target; Based on the sample images and a pre-built first model, the predicted probability value and predicted offset of the second target in multiple depth ranges are determined. Based on the labeled data, the predicted probability value, and the predicted offset, the loss value of the first model is determined; When the loss value meets the preset conditions, a deep prediction model trained to a convergent state is obtained.

7. The depth prediction method as described in claim 6, characterized in that, The construction of labeled data based on pre-acquired sample images includes: Based on a preset interval distance, multiple depth intervals are determined within the imaging range of the camera device; Based on the distance between the second target and the camera device, determine the actual offset of the second target relative to the midpoint of any one of the plurality of depth intervals; Based on the actual offset, determine the annotation probability value of the second target being in any of the depth intervals; The labeled data is determined based on the actual offset and the labeled probability value.

8. The depth prediction method as described in claim 7, characterized in that, The step of determining the loss value of the first model based on the labeled data, the predicted probability value, and the predicted offset includes: The first loss value of the first model is determined based on the cross-entropy between the predicted probability value and the labeled probability value, and based on the difference between the predicted offset and the true offset. Based on the order of the predicted probability values ​​from largest to smallest, a third depth interval and a fourth depth interval are determined from the plurality of depth intervals; based on the third predicted offset of the second target in the third depth interval and the fourth predicted offset of the second target in the fourth depth interval, a second loss value of the first model is determined; Based on the third predicted offset, the fourth predicted offset, the intrinsic parameters of the camera device, and the morphological information of the second target obtained in advance, the third loss value of the first model is determined. The loss value of the first model is determined based on the first loss value, the second loss value, and the third loss value.

9. The depth prediction method as described in claim 8, characterized in that, The step of determining the second loss value of the first model based on the third predicted offset of the second target in the third depth interval and the fourth predicted offset of the second target in the fourth depth interval includes: Based on the intrinsic parameters of the camera device and the preset normalization parameters, the distance between the second target and the camera device is normalized to obtain the normalized depth value of the second target. The third predicted depth of the second target is determined based on the third predicted offset, the distance between the third depth interval and the camera device; and the fourth predicted depth of the second target is determined based on the fourth predicted offset, the distance between the fourth interval and the camera device. The second loss value is determined based on the difference between the normalized depth value and the third predicted depth, and the difference between the normalized depth value and the fourth predicted depth.

10. The depth prediction method as described in claim 9, characterized in that, The step of determining the third loss value of the first model based on the third predicted offset, the fourth predicted offset, the intrinsic parameters of the camera device, and the pre-acquired morphological information of the second target includes: Based on the intrinsic parameters, the third prediction depth and the fourth prediction depth are updated respectively to obtain the third depth value corresponding to the third prediction depth and the fourth depth value corresponding to the fourth prediction depth; Based on the third depth value, the fourth depth value, the first two-dimensional coordinates of the center point of the second target in the sample image, and the intrinsic parameters, the first three-dimensional coordinates of the center point of the second target in the three-dimensional coordinate system corresponding to the camera device are determined. Based on the morphological information and the first three-dimensional coordinates, the second three-dimensional coordinates of the vertex of the second target in the three-dimensional coordinate system are determined; Based on the intrinsic parameters and the second three-dimensional coordinates, the projection coordinates of the vertex of the second target in the sample image are determined; The third loss value is determined based on the projected coordinates and the second two-dimensional coordinates of the second target in the sample image.

11. A depth prediction system, characterized in that, The system includes an electronic device and a camera device. The electronic device is communicatively connected to the camera device, which is used to capture images of a first target around the vehicle. Based on the first image of the first target captured by the camera device, the electronic device determines, using a pre-trained depth prediction model, a first probability value and a first offset for the first target to be in a first depth range, and a second probability value and a second offset for the first target to be in a second depth range. Based on the first offset, the distance between the first depth range and the camera device, the electronic device determines a first depth value for the first target. And, based on the second offset, the distance between the second depth range and the camera device, the second depth value of the first target is determined; The depth value of the first target is determined based on the first probability value, the second probability value, the first depth value, and the second depth value.

12. An electronic device, characterized in that, The electronic device includes a processor and a storage device, wherein the processor is configured to implement the depth prediction method as described in any one of claims 1 to 10 when executing a computer program stored in the storage device.

13. A vehicle, characterized in that, The vehicle includes the electronic equipment as described in claim 9.

Images