Method and device for predicting target depth information, electronic device and storage medium

Abstract

The present application discloses a method and device for predicting the depth information of a target, an electronic device and a storage medium, the method comprising: obtaining an image to be predicted of the target; processing the image to be predicted using a monocular depth information estimation model to obtain a depth map corresponding to the image to be predicted; processing the image to be predicted using a monocular target detection model to obtain the center point position information of the target and the corresponding first depth information; determining the second depth information corresponding to the target in the depth map according to the center point position information of the target and the depth map corresponding to the image to be predicted; fusing the first depth information and the second depth information using a depth information fusion model to obtain the final depth information of the target. When predicting the depth information of the target, the present application takes into account the influence of the depth information of the non-center point area and the background area of ​​the target on the prediction of the depth information of the target, and by combining these depth information for fusion processing, more accurate target depth information can be obtained.

Description

Technical Field

[0001] The present application relates to the technical field of target detection, and in particular to a method and device for predicting depth information of a target, an electronic device, and a storage medium. Background Art

[0002] The difficulty of monocular 3D object detection has always been the prediction of target depth. For example, the depth information is lost when the image is mapped from 3D space to 2D plane. Therefore, depth processing has always been the focus of monocular 3D object detection research.

[0003] The existing technical solution predicts the target depth value mainly by training the prediction model through the training set, and then determines the final depth value based on the prediction value directly output by the prediction model using the mean and variance statistically obtained from the label information of the training set.

[0004] However, the existing technical solutions only use the depth label values ​​of the center points of all targets when training the depth value prediction model. These label values ​​do not involve the depth information of other parts of the target and the background area. Therefore, the accuracy of depth information prediction needs to be improved. Summary of the invention

[0005] Embodiments of the present application provide a method and device for predicting depth information of a target, an electronic device, and a storage medium to improve the accuracy of monocular 3D target depth information prediction.

[0006] The present application embodiment adopts the following technical solutions:

[0007] In a first aspect, an embodiment of the present application provides a method for predicting depth information of a target, wherein the method comprises:

[0008] Obtain the target image to be predicted;

[0009] Processing the image to be predicted using a monocular depth information estimation model to obtain a depth map corresponding to the image to be predicted;

[0010] Processing the image to be predicted using a monocular target detection model to obtain center point position information of the target and corresponding first depth information;

[0011] Determine second depth information corresponding to the target in the depth map according to the center point position information of the target and the depth map corresponding to the image to be predicted;

[0012] The first depth information and the second depth information are fused using a depth information fusion model to obtain final depth information of the target.

[0013] Optionally, the monocular depth information estimation model is trained in the following manner:

[0014] Obtaining an original training image and a label depth map corresponding to the original training image;

[0015] According to a preset reduction ratio, the sizes of the original training image and the corresponding label depth map are adjusted to obtain an adjusted training image and a corresponding label depth map;

[0016] The monocular depth information estimation model is trained using the adjusted training image and the corresponding labeled depth map to obtain a trained monocular depth information estimation model.

[0017] Optionally, the depth information fusion model is trained in the following manner:

[0018] Processing the adjusted training image using the trained monocular depth information estimation model to obtain a depth map corresponding to the adjusted training image;

[0019] Processing the original training image using the monocular target detection model to obtain center point position information of the target in the original training image and corresponding third depth information;

[0020] Determine fourth depth information of the center point position information of the target in the original training image in the depth map corresponding to the adjusted training image according to the center point position information of the target in the original training image and the depth map corresponding to the adjusted training image;

[0021] The depth information fusion model is trained according to the third depth information and the fourth depth information, and the label depth information of the target to obtain a trained depth information fusion model.

[0022] Optionally, the using the monocular target detection model to process the original training image to obtain the center point position information of the target in the original training image and the corresponding third depth information includes:

[0023] Using the monocular target detection model to process the original training image to obtain a prediction box of the target;

[0024] Determine the intersection-over-union ratio of the predicted frame of the target and the label frame of the target;

[0025] When the intersection-over-union ratio is greater than a preset threshold, the center point position information and the third depth information of the target corresponding to the prediction frame of the target are obtained.

[0026] Optionally, the using a monocular depth information estimation model to process the image to be predicted to obtain a depth map corresponding to the image to be predicted includes:

[0027] Adjusting the size of the image to be predicted according to a preset reduction ratio to obtain an adjusted image to be predicted;

[0028] The adjusted image to be predicted is processed using a monocular depth information estimation model to obtain a depth map corresponding to the adjusted image to be predicted.

[0029] Optionally, the center point position information of the target includes the center point position coordinates of the target, and determining, according to the center point position information of the target and the depth map corresponding to the image to be predicted, the second depth information corresponding to the target in the depth map includes:

[0030] Determining the adjusted center point position coordinates according to the preset reduction ratio and the center point position coordinates of the target;

[0031] The second depth information corresponding to the adjusted center point position coordinates is determined in the depth map corresponding to the image to be predicted.

[0032] Optionally, the depth information fusion model is composed of two fully connected layer networks, and the using the depth information fusion model to fuse the first depth information and the second depth information to obtain the final depth information of the target includes:

[0033] The final depth information of the target is determined according to the parameters corresponding to the channels of the fully connected layers in the depth information fusion model, as well as the first depth information and the second depth information.

[0034] In a second aspect, an embodiment of the present application further provides a device for predicting depth information of a target, wherein the device comprises:

[0035] An acquisition unit, used for acquiring a target image to be predicted;

[0036] A depth information estimation unit, configured to process the image to be predicted using a monocular depth information estimation model to obtain a depth map corresponding to the image to be predicted;

[0037] A target detection unit, used to process the image to be predicted using a monocular target detection model to obtain center point position information of the target and corresponding first depth information;

[0038] A determining unit, configured to determine second depth information corresponding to the target in the depth map according to the center point position information of the target and the depth map corresponding to the image to be predicted;

[0039] A depth information fusion unit is used to fuse the first depth information and the second depth information using a depth information fusion model to obtain final depth information of the target.

[0040] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0041] Processor; and

[0042] A memory arranged to store computer executable instructions which, when executed, cause the processor to perform any of the methods described above.

[0043] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores one or more programs. When the one or more programs are executed by an electronic device including multiple application programs, the electronic device executes any of the aforementioned methods.

[0044] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects: the method for predicting the depth information of the target in the embodiments of the present application first obtains the image to be predicted of the target; then uses the monocular depth information estimation model to process the image to be predicted to obtain the depth map corresponding to the image to be predicted; then uses the monocular target detection model to process the image to be predicted to obtain the center point position information of the target and the corresponding first depth information; then determines the second depth information corresponding to the target in the depth map based on the center point position information of the target and the depth map corresponding to the image to be predicted; finally uses the depth information fusion model to fuse the first depth information and the second depth information to obtain the final depth information of the target. The method for predicting the depth information of the target in the embodiments of the present application takes into account the influence of the depth information of the non-center point area and the background area of ​​the target on the prediction of the target depth information when predicting the target depth information. By combining these depth information for fusion processing, more accurate target depth information can be obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0046] Figure 1 A schematic diagram of a flow chart of a method for predicting depth information of a target in an embodiment of the present application;

[0047] Figure 2 A schematic diagram of the structure of a depth information prediction device for a target in an embodiment of the present application;

[0048] Figure 3 This is a schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0050] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0051] In the monocular 3D target detection task scenario, by inputting a single RGB image, the categories of the targets in the image and their 3D position information are predicted. The 3D position information includes the h, w, l of the target, i.e., height, width, length, the position of the target center point (x, y, z), and the yaw angle θ. It is usually assumed that the pitch angle and roll angle of the target are 0. Among the many variables that need to be predicted, correctly predicting the depth value z in the center point position coordinates is a major difficulty and is also the key to accurately predicting the target's 3D position information.

[0052] Taking the SMOKE monocular 3D target detection model in the prior art as an example, in order to predict the target variables (h, w, l, x, y, z, θ), it is first necessary to predict the depth value z of the target center point position coordinates:

[0053] (1)

[0054] Then restore x and y by projecting the center point in the camera intrinsic parameters and image coordinate system:

[0055] (2)

[0056] Use the average height, width, and length regression to predict the height, width, and length of the target:

[0057] (3)

[0058] Predict the values ​​of two trigonometric functions about the angle α:

[0059] (4)

[0060] The heading angle θ can be obtained by the following formula:

[0061] (5)

[0062] From the above solution process, it can be seen that the prediction of many variables depends on the depth value z, so predicting the accurate depth value z is the key to the entire position information prediction.

[0063] In the existing technical solution, the depth value z is mainly predicted by the mean value statistically obtained from the label information of the training set. and variance ,and is the predicted value of the model, so the z value can be calculated using the above formula (1).

[0064] However, from the current experimental situation, the model's x c ,y c ,δ xc ,δ yc ,δ h ,δ w ,δ l The predictions of variables such as α and α are relatively accurate, but the prediction accuracy of the depth value z is always poor.

[0065] Based on this, the embodiment of the present application provides a method for predicting depth information of a target, such as Figure 1 As shown, a flowchart of a method for predicting depth information of a target in an embodiment of the present application is provided, and the method at least includes the following steps S110 to S150:

[0066] Step S110, obtaining the target image to be predicted.

[0067] When predicting the depth information of a target, it is necessary to first obtain the image of the target to be predicted. Taking the autonomous driving scenario as an example, the target here can be a car, truck, bus, pedestrian, bicycle, etc. on the road. Of course, it can also be other targets in other application scenarios, which are not specifically limited here.

[0068] Step S120: Process the image to be predicted using a monocular depth information estimation model to obtain a depth map corresponding to the image to be predicted.

[0069] After obtaining the image to be predicted, the pre-trained monocular depth information estimation model is used to perform depth prediction on the image to be predicted, so as to obtain a depth map corresponding to the image to be predicted. The depth map contains the depth information corresponding to each pixel in the entire image to be predicted, that is, it not only contains the depth information of the pixel points in the target area, but also contains the depth information of the pixel points in the non-target area and the background area. The accuracy of depth information prediction is improved through position comparison and mutual promotion.

[0070] Step S130: Process the image to be predicted using a monocular target detection model to obtain center point position information of the target and corresponding first depth information.

[0071] The above-mentioned monocular depth information estimation model is mainly used to predict the depth information of each pixel in the entire image, and does not involve detection and recognition of targets. The core of the embodiment of the present application is to predict the depth information of the target in the image. Therefore, it is also necessary to use a pre-trained monocular target detection model to perform target detection on the predicted image, so as to obtain the center point position information of the target in the image and its corresponding first depth information.

[0072] The monocular target detection model can be obtained by training based on the existing technology. For example, the SMOKE monocular 3D target detection model in the existing technology can be used for training. The specific training process will not be described in detail.

[0073] Step S140: determining second depth information corresponding to the target in the depth map according to the center point position information of the target and the depth map corresponding to the image to be predicted.

[0074] Since the aforementioned monocular depth information estimation model outputs the predicted results of the depth information of each pixel in the entire image, the depth value of each pixel position is predicted through the interaction with the depth values ​​of other pixels, so the obtained depth value is more accurate and comprehensive.

[0075] Based on this, after the embodiment of the present application obtains the depth map corresponding to the image to be predicted through the above-mentioned step S120 and obtains the center point position information of the target in the image to be predicted and its corresponding first depth information through step S130, the second depth information corresponding to the position can be queried in the depth map corresponding to the above-mentioned image to be predicted according to the center point position information of the target, so as to use the second depth information to improve the accuracy of the first depth information directly output by the monocular target detection model.

[0076] Step S150: Using a depth information fusion model, the first depth information and the second depth information are fused to obtain final depth information of the target.

[0077] Specifically, after obtaining the first depth information and the second depth information mentioned above, the first depth information and the second depth information can be fused using a pre-trained depth information fusion model to obtain the final depth information of the target. The final depth information is more accurate because it simultaneously considers the influence of the depth information of the non-center point area and the background area of ​​the target on the prediction of the target depth information.

[0078] The target depth information prediction method of the embodiment of the present application takes into account the influence of the depth information of the non-center point area and the background area of ​​the target on the prediction of the target depth information when predicting the target depth information. By combining these depth information for fusion processing, more accurate target depth information can be obtained.

[0079] In one embodiment of the present application, the monocular depth information estimation model is trained in the following manner: obtaining an original training image and a labeled depth map corresponding to the original training image; adjusting the size of the original training image and the corresponding labeled depth map according to a preset reduction ratio to obtain an adjusted training image and a corresponding labeled depth map; and training the monocular depth information estimation model using the adjusted training image and the corresponding labeled depth map to obtain a trained monocular depth information estimation model.

[0080] When training a monocular depth information estimation model, the embodiment of the present application can first obtain the original training image and the corresponding label depth map. Since the depth information of the target in the image is actually required in the subsequent steps, the actual depth value difference between the target center point pixel and the surrounding neighboring pixels is not large. Therefore, in order to further improve the computational efficiency of the monocular depth information estimation model, the original training image of the input model and the corresponding label depth map can be reduced to a certain proportion of the original size and then used for training according to a certain reduction ratio, thereby improving the training efficiency. The reduction ratio here can be flexibly adjusted according to the actual training needs and training effects, for example, it can be set to 1 / 4, 1 / 2, etc.

[0081] The above monocular depth information estimation model can be implemented using the FastDepth model. The overall structure of the model is relatively simple, and the Encoder-Decoder architecture is adopted, which can greatly improve the computational efficiency of the model while ensuring accuracy. In addition to using FastDepth, those skilled in the art can also flexibly select other types of depth information estimation models according to actual needs, which are not specifically limited here.

[0082] In one embodiment of the present application, the depth information fusion model is trained in the following manner: using the trained monocular depth information estimation model to process the adjusted training image to obtain a depth map corresponding to the adjusted training image; using the monocular target detection model to process the original training image to obtain the center point position information of the target in the original training image and the corresponding third depth information; based on the center point position information of the target in the original training image and the depth map corresponding to the adjusted training image, determining the fourth depth information of the center point position information of the target in the original training image in the depth map corresponding to the adjusted training image; training the depth information fusion model based on the third depth information and the fourth depth information, as well as the label depth information of the target, to obtain the trained depth information fusion model.

[0083] When training the depth information fusion model, the embodiment of the present application can first use the monocular depth information estimation model such as FastDepth obtained by the above embodiment to predict each training image in the training set, where the training image refers to the resized training image, so as to obtain the depth map depth_map corresponding to each resized image. Then, the monocular 3D target detection model such as SMOKE obtained in the above embodiment is used to predict each original training image in the training set, and the center point coordinates (x_{\mathcal {E}}) of the target in the original training image in the image coordinate system are obtained. c ,y c ) and the corresponding predicted depth value z obj .

[0084] Since the image size input in the monocular depth information estimation model is different from the image size input in the monocular 3D target detection model, the center point coordinates (x c ,y c ), the center point coordinates (x c ,y c ), for example, if the image size in the input monocular depth information estimation model is 1 / 4 of the original image size, then the center point coordinates of the target are adjusted to (x c / 4,y c / 4), then use (x c / 4,y c / 4) The corresponding depth value z can be queried in the above depth_map d .

[0085] Based on this, we can obtain a dataset about the depth value {z obj ,z d ,z}N , where N represents the number of samples and z represents the true value of the depth label. The deep information fusion model is trained using this dataset to obtain the trained deep information fusion model.

[0086] In one embodiment of the present application, the use of the monocular target detection model to process the original training image to obtain the center point position information of the target in the original training image and the corresponding third depth information includes: using the monocular target detection model to process the original training image to obtain a prediction box of the target; determining the intersection-and-union ratio of the prediction box of the target and the label box of the target; when the intersection-and-union ratio is greater than a preset threshold, obtaining the center point position information and the third depth information of the target corresponding to the prediction box of the target.

[0087] When the embodiment of the present application uses a monocular target detection model to process the original training image, the prediction frame of the target output by the monocular target detection model can be first obtained. The degree of closeness between the prediction frame of the target and the pre-marked label frame can be determined by calculating the intersection of Union (IoU) between the prediction frame of the target and the pre-marked label frame. That is, the larger the IoU, the larger the overlapping area between the prediction frame of the target and the label frame, and the better the detection effect. Conversely, the worse the detection effect.

[0088] Therefore, based on the size of the intersection over union ratio, some targets with very small overlap areas between the prediction box and the label box can be directly filtered out, thereby improving the training efficiency of the subsequent model. Here, a threshold for judgment can be set in advance. When the calculated IoU is greater than the threshold, the prediction result of the target can be retained for subsequent training. The size of the threshold can be flexibly set according to actual needs, but it should not be too large. For example, it can be set to 0.01 to avoid insufficient number of samples available for training due to the limitation of the threshold, which affects the training effect.

[0089] In one embodiment of the present application, the use of a monocular depth information estimation model to process the image to be predicted to obtain a depth map corresponding to the image to be predicted includes: adjusting the size of the image to be predicted according to a preset reduction ratio to obtain an adjusted image to be predicted; and processing the adjusted image to be predicted using a monocular depth information estimation model to obtain a depth map corresponding to the adjusted image to be predicted.

[0090] As mentioned above, in the training stage of the monocular depth information estimation model, the embodiment of the present application reduces the size of the image input to the model. Therefore, in the model prediction stage, the size of the image to be predicted can also be adjusted according to the same reduction ratio to obtain the adjusted image to be predicted, and then the trained monocular depth information estimation model is used to perform depth estimation on the adjusted image to be predicted to obtain the corresponding depth map.

[0091] In one embodiment of the present application, the center point position information of the target includes the center point position coordinates of the target, and determining the second depth information corresponding to the target in the depth map based on the center point position information of the target and the depth map corresponding to the image to be predicted includes: determining the adjusted center point position coordinates based on the preset reduction ratio and the center point position coordinates of the target; and determining the second depth information corresponding to the adjusted center point position coordinates in the depth map corresponding to the image to be predicted.

[0092] As mentioned above, the image size input into the monocular depth information estimation model is different from the image size input into the monocular 3D target detection model. Therefore, when determining the second depth information based on the center point position information of the target and the depth map corresponding to the image to be predicted, the center point position coordinates of the target can also be adjusted according to a preset reduction ratio, and then the second depth information corresponding to the point coordinates can be found in the depth map corresponding to the image to be predicted based on the adjusted center point position coordinates.

[0093] In one embodiment of the present application, the depth information fusion model is composed of two layers of fully connected layer networks, and the first depth information and the second depth information are fused using the depth information fusion model to obtain the final depth information of the target, including: determining the final depth information of the target according to the parameters corresponding to the channels of the fully connected layers in the depth information fusion model, as well as the first depth information and the second depth information.

[0094] The depth information fusion model of the embodiment of the present application is mainly composed of two layers of fully connected layer networks, the number of channels in the first layer is 2, and the number of channels in the second layer is 1. Therefore, according to the parameters corresponding to each channel of each fully connected layer, as well as the first depth information and the second depth information, the final depth information of the target can be calculated by weighted fusion, which can be specifically in the following form:

[0095] z = w 2,1,1 * (w 1,1,1 * z obj +w 1,1,2 * z d ) + w 2,1,2 *(w 1,2,1 * zobj +w 1,2,2 * z d )(6)

[0096] Among them, w in the above formula 1,1,1 It can be regarded as the first parameter of the first channel of the first layer, w 1,1,2 It can be regarded as the second parameter of the first channel of the first layer, and so on for other parameters.

[0097] All w in the above formula constitute the parameters of the deep information fusion model, which can be initialized to random floating point values ​​between [0, 1] before training.

[0098] Of course, in addition to adopting the deep information fusion model of the above structure, technicians in this field can also flexibly adjust the number of fully connected layers and channels according to actual needs such as training cost and training efficiency, which is not specifically limited here.

[0099] The present application also provides a device 200 for predicting depth information of a target. Figure 2 As shown, a schematic diagram of the structure of a depth information prediction device for a target in an embodiment of the present application is provided, wherein the device 200 includes: an acquisition unit 210, a depth information estimation unit 220, a target detection unit 230, a determination unit 240, and a depth information fusion unit 250, wherein:

[0100] An acquisition unit 210 is used to acquire a target image to be predicted;

[0101] The depth information estimation unit 220 is used to process the image to be predicted using a monocular depth information estimation model to obtain a depth map corresponding to the image to be predicted;

[0102] The target detection unit 230 is used to process the image to be predicted using a monocular target detection model to obtain the center point position information of the target and the corresponding first depth information;

[0103] A determination unit 240 is configured to determine second depth information corresponding to the target in the depth map according to the center point position information of the target and the depth map corresponding to the image to be predicted;

[0104] The depth information fusion unit 250 is used to fuse the first depth information and the second depth information using a depth information fusion model to obtain final depth information of the target.

[0105] In one embodiment of the present application, the monocular depth information estimation model is trained in the following manner: obtaining an original training image and a labeled depth map corresponding to the original training image; adjusting the size of the original training image and the corresponding labeled depth map according to a preset reduction ratio to obtain an adjusted training image and a corresponding labeled depth map; and training the monocular depth information estimation model using the adjusted training image and the corresponding labeled depth map to obtain a trained monocular depth information estimation model.

[0106] In one embodiment of the present application, the depth information fusion model is trained in the following manner: using the trained monocular depth information estimation model to process the adjusted training image to obtain a depth map corresponding to the adjusted training image; using the monocular target detection model to process the original training image to obtain the center point position information of the target in the original training image and the corresponding third depth information; based on the center point position information of the target in the original training image and the depth map corresponding to the adjusted training image, determining the fourth depth information of the center point position information of the target in the original training image in the depth map corresponding to the adjusted training image; training the depth information fusion model based on the third depth information and the fourth depth information, as well as the label depth information of the target, to obtain the trained depth information fusion model.

[0107] In one embodiment of the present application, the depth information fusion model is trained in the following manner: using the monocular target detection model to process the original training image to obtain a prediction box of the target; determining the intersection-and-union ratio of the prediction box of the target and the label box of the target; when the intersection-and-union ratio is greater than a preset threshold, obtaining the center point position information and third depth information of the target corresponding to the prediction box of the target.

[0108] In one embodiment of the present application, the depth information estimation unit 220 is specifically used to: adjust the size of the image to be predicted according to a preset reduction ratio to obtain an adjusted image to be predicted; use a monocular depth information estimation model to process the adjusted image to be predicted to obtain a depth map corresponding to the adjusted image to be predicted.

[0109] In one embodiment of the present application, the center point position information of the target includes the center point position coordinates of the target, and the determination unit 240 is specifically used to: determine the adjusted center point position coordinates according to the preset reduction ratio and the center point position coordinates of the target; and determine the second depth information corresponding to the adjusted center point position coordinates in the depth map corresponding to the image to be predicted.

[0110] In one embodiment of the present application, the depth information fusion model is composed of two layers of fully connected layer networks, and the depth information fusion unit 250 is specifically used to determine the final depth information of the target according to the parameters corresponding to the channels of the fully connected layers in the depth information fusion model, as well as the first depth information and the second depth information.

[0111] Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 3 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. Of course, the electronic device may also include hardware required for other services.

[0112] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0113] The memory is used to store the program. Specifically, the program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides instructions and data to the processor.

[0114] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a target depth information prediction device at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:

[0115] Obtain the target image to be predicted;

[0116] Processing the image to be predicted using a monocular depth information estimation model to obtain a depth map corresponding to the image to be predicted;

[0117] Processing the image to be predicted using a monocular target detection model to obtain center point position information of the target and corresponding first depth information;

[0118] Determine second depth information corresponding to the target in the depth map according to the center point position information of the target and the depth map corresponding to the image to be predicted;

[0119] The first depth information and the second depth information are fused using a depth information fusion model to obtain final depth information of the target.

[0120] The above application Figure 1 The method performed by the target depth information prediction device disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in software form. The above processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in a decoding processor. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0121] The electronic device may also perform Figure 1 A method for predicting the depth information of a target in the present invention is implemented by a device for predicting the depth information of a target in the present invention. Figure 1 The functions of the illustrated embodiment will not be described in detail in the embodiments of the present application.

[0122] The present application also provides a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by an electronic device including multiple application programs, enable the electronic device to execute Figure 1 The method performed by the target depth information prediction device in the illustrated embodiment is specifically used to perform:

[0123] Obtain the target image to be predicted;

[0124] Processing the image to be predicted using a monocular depth information estimation model to obtain a depth map corresponding to the image to be predicted;

[0125] Processing the image to be predicted using a monocular target detection model to obtain center point position information of the target and corresponding first depth information;

[0126] Determine second depth information corresponding to the target in the depth map according to the center point position information of the target and the depth map corresponding to the image to be predicted;

[0127] The first depth information and the second depth information are fused using a depth information fusion model to obtain final depth information of the target.

[0128] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0129] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0130] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0131] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0132] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0133] Memory may include non-permanent storage in a computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0134] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0135] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0136] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0137] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A method for predicting depth information of a target, wherein: The method comprises: Obtain the target image to be predicted; Processing the image to be predicted using a monocular depth information estimation model to obtain a depth map corresponding to the image to be predicted; Processing the image to be predicted using a monocular target detection model to obtain center point position information of the target and corresponding first depth information; Determine second depth information corresponding to the target in the depth map according to the center point position information of the target and the depth map corresponding to the image to be predicted; Using a depth information fusion model to fuse the first depth information and the second depth information to obtain final depth information of the target; The center point position information of the target includes the center point position coordinates of the target, and determining, according to the center point position information of the target and the depth map corresponding to the image to be predicted, second depth information corresponding to the target in the depth map includes: Determining the adjusted center point position coordinates according to the preset reduction ratio and the center point position coordinates of the target; The second depth information corresponding to the adjusted center point position coordinates is determined in the depth map corresponding to the image to be predicted.

2. The method of claim 1, wherein: The monocular depth information estimation model is trained in the following way: Obtaining an original training image and a label depth map corresponding to the original training image; According to a preset reduction ratio, the sizes of the original training image and the corresponding label depth map are adjusted to obtain an adjusted training image and a corresponding label depth map; The monocular depth information estimation model is trained using the adjusted training image and the corresponding labeled depth map to obtain a trained monocular depth information estimation model.

3. The method of claim 2, wherein: The depth information fusion model is trained in the following way: Processing the adjusted training image using the trained monocular depth information estimation model to obtain a depth map corresponding to the adjusted training image; Processing the original training image using the monocular target detection model to obtain center point position information of the target in the original training image and corresponding third depth information; Determine fourth depth information of the center point position information of the target in the original training image in the depth map corresponding to the adjusted training image according to the center point position information of the target in the original training image and the depth map corresponding to the adjusted training image; The depth information fusion model is trained according to the third depth information and the fourth depth information, and the label depth information of the target to obtain a trained depth information fusion model.

4. The method of claim 3, wherein: The process of processing the original training image by using the monocular target detection model to obtain the center point position information of the target in the original training image and the corresponding third depth information includes: Using the monocular target detection model to process the original training image to obtain a prediction box of the target; Determine the intersection-over-union ratio of the target prediction box and the target label box; When the intersection-over-union ratio is greater than a preset threshold, the center point position information and the third depth information of the target corresponding to the prediction frame of the target are obtained.

5. The method of claim 1, wherein: The using a monocular depth information estimation model to process the image to be predicted to obtain a depth map corresponding to the image to be predicted includes: Adjusting the size of the image to be predicted according to a preset reduction ratio to obtain an adjusted image to be predicted; The adjusted image to be predicted is processed using a monocular depth information estimation model to obtain a depth map corresponding to the adjusted image to be predicted.

6. The method of claim 1, wherein: The depth information fusion model is composed of two fully connected layer networks, and the first depth information and the second depth information are fused by the depth information fusion model to obtain the final depth information of the target, including: The final depth information of the target is determined according to the parameters corresponding to the channels of the fully connected layers in the depth information fusion model, as well as the first depth information and the second depth information.

7. A device for predicting depth information of an object, wherein: The device comprises: An acquisition unit, used for acquiring a target image to be predicted; A depth information estimation unit, configured to process the image to be predicted using a monocular depth information estimation model to obtain a depth map corresponding to the image to be predicted; A target detection unit, used to process the image to be predicted using a monocular target detection model to obtain center point position information of the target and corresponding first depth information; A determining unit, configured to determine second depth information corresponding to the target in the depth map according to the center point position information of the target and the depth map corresponding to the image to be predicted; A depth information fusion unit, configured to fuse the first depth information and the second depth information using a depth information fusion model to obtain final depth information of the target; The center point position information of the target includes the center point position coordinates of the target, and the determining unit is specifically used for: Determining the adjusted center point position coordinates according to the preset reduction ratio and the center point position coordinates of the target; The second depth information corresponding to the adjusted center point position coordinates is determined in the depth map corresponding to the image to be predicted.

8. An electronic device, comprising: processor; as well as A memory arranged to store computer executable instructions, which when executed cause the processor to perform the method of any one of claims 1 to 6.

9. A computer-readable storage medium storing one or more programs, which, when executed by an electronic device including a plurality of application programs, causes the electronic device to execute any one of the methods of claims 1 to 6.

Images