Monocular depth estimation method and device for irregular roads, storage medium, and terminal

Abstract

The present invention discloses a monocular depth estimation method and device, storage medium, and terminal for irregular roads. These methods relate to the field of image processing technology and aim to address the difficulty of depth estimation on irregular roads. The method comprises acquiring an initial image from an image acquisition device and performing effective target extraction on the initial image to obtain a target image uniquely containing a target vehicle; extracting depth information from the target image using a trained monocular depth estimation network model to obtain a depth image of the target image; and fitting the depth information of the target vehicle in the depth image to obtain the distance between the target vehicle and the image acquisition device. The method is primarily used for extracting distance information of vehicle targets on irregular roads.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a monocular depth estimation method and device for irregular roads, a storage medium, and a terminal. Background Art

[0002] Monocular depth estimation is an image analysis method that estimates the distance between an object and the image capture device based on an image captured by a single device. Existing monocular depth estimation methods determine the distance to the target vehicle based on the combined features of the road surface, vegetation, and vehicle in the image. These methods are primarily suitable for highways with relatively regular road conditions. However, when unmanned driving systems encounter irregular roads, such as mining roads, these roads are prone to complex surface conditions due to their irregular structure, close proximity to cliffs, mounds, and other road features, making monocular depth estimation impossible. Summary of the Invention

[0003] In view of this, the present invention provides a monocular depth estimation method and device, storage medium, and terminal for irregular roads, the main purpose of which is to solve the problem of low accuracy of monocular depth estimation in mining environments in existing monocular depth estimation methods for irregular roads.

[0004] According to one aspect of the present invention, a monocular depth estimation method for irregular roads is provided, comprising:

[0005] Acquire an initial image from an image acquisition device, and obtain a target image that uniquely includes a target vehicle by performing effective target extraction on the initial image;

[0006] Extracting depth information from the target image using a trained monocular depth estimation network model to obtain a depth image of the target image, wherein the trained monocular depth estimation network model is trained based on high-dimensional information extracted by a high-dimensional convolutional network;

[0007] The distance between the target vehicle and the image acquisition device is obtained by fitting the depth information of the target vehicle in the depth image.

[0008] Furthermore, the step of extracting effective targets from the initial image to obtain a target image that uniquely includes the target vehicle comprises:

[0009] performing target detection on the initial image, and if a target is detected, determining at least one target vehicle in the initial image;

[0010] Determining a non-target vehicle image region according to the target vehicle image region, and obtaining a target image by performing mask processing on the non-target vehicle image region in the initial image;

[0011] If the target is not detected, the initial image is acquired again.

[0012] Furthermore, before extracting depth information from the target image using the trained monocular depth estimation network model to obtain the depth image of the target vehicle, the method further includes:

[0013] Build an initial monocular depth estimation network model containing at least five layers of convolutional networks;

[0014] Performing mask processing on each sample image in the training sample image set to obtain a masked training sample image set;

[0015] The initial monocular depth estimation network model is trained using the masked training sample image set to obtain a trained monocular depth estimation network model.

[0016] Furthermore, the initial monocular depth estimation network model is trained using the masked training sample image set to obtain a trained monocular depth estimation network model, including:

[0017] Using the initial monocular depth estimation network model to extract high-dimensional information from the masked training sample image to obtain high-dimensional information, and reconstructing the image using the high-dimensional information to obtain a reconstructed image of the training sample image;

[0018] Determining a loss function based on the similarity between the reconstructed image and the training sample image;

[0019] The initial monocular depth estimation network model is trained using the loss function to obtain a trained monocular depth estimation network model.

[0020] Furthermore, the extracting high-dimensional information from the training sample image using the initial monocular depth estimation network model to obtain high-dimensional information includes:

[0021] Extracting depth information from the target image using the initial monocular depth estimation network model, and extracting high-dimensional abstract features output by the high-dimensional convolutional network in the initial monocular depth estimation network model;

[0022] High-dimensional information is obtained by upsampling and fusing the high-dimensional abstract features.

[0023] Furthermore, before training the initial monocular depth estimation network model using the masked training sample image set, the method further includes:

[0024] Acquire continuous frame image samples, and perform frame extraction processing on the continuous frame image samples at equal time intervals to obtain training sample images corresponding to multiple image frames;

[0025] A training sample image set is constructed according to the training sample images.

[0026] Furthermore, the obtaining of the distance between the target vehicle and the image acquisition device by fitting the depth information of the target vehicle in the depth image includes:

[0027] Performing discrete point pixel filtering on the depth information of the target vehicle in the depth image to obtain filtered depth information of the target vehicle;

[0028] The depth information of the target vehicle after the filtering process is fitted using a distance histogram distribution algorithm to obtain the distance between the target vehicle and the image acquisition device.

[0029] According to another aspect of the present invention, a monocular depth estimation device for irregular roads is provided, comprising:

[0030] An acquisition module is used to acquire an initial image from an image acquisition device, and obtain a target image that uniquely contains a target vehicle by performing effective target extraction on the initial image;

[0031] An extraction module is used to extract depth information from the target image using a trained monocular depth estimation network model to obtain a depth image of the target image, wherein the trained monocular depth estimation network model is trained based on high-dimensional information extracted by a high-dimensional convolutional network;

[0032] The determination module is used to obtain the distance between the target vehicle and the image acquisition device by fitting the depth information of the target vehicle in the depth image.

[0033] Furthermore, the acquisition module includes:

[0034] a target detection unit, configured to perform target detection on the initial image, and if a target is detected, determine at least one target vehicle in the initial image;

[0035] a mask processing unit, configured to determine a non-target vehicle image region in the initial image based on the image region of the target vehicle, and obtain a target image by performing mask processing on the non-target vehicle image region;

[0036] The acquisition unit is configured to reacquire the initial image if no target is detected.

[0037] Furthermore, the device further comprises:

[0038] A construction module for building an initial monocular depth estimation network model containing at least five layers of convolutional networks;

[0039] A mask processing module is used to perform mask processing on each sample image in the training sample image set to obtain a masked training sample image set;

[0040] A training module is used to train the initial monocular depth estimation network model using the training sample image set after the mask processing to obtain a trained monocular depth estimation network model.

[0041] Furthermore, the training module includes:

[0042] A reconstruction unit is used to extract high-dimensional information from the masked training sample image using the initial monocular depth estimation network model to obtain high-dimensional information, and reconstruct the image using the high-dimensional information to obtain a reconstructed image of the training sample image;

[0043] a determining unit, configured to determine a loss function according to a similarity between the reconstructed image and the training sample image;

[0044] A training unit is used to train the initial monocular depth estimation network model using the loss function to obtain a trained monocular depth estimation network model.

[0045] Furthermore, in a specific application scenario, the training unit is used to extract depth information from the target image using the initial monocular depth estimation network model, and extract high-dimensional abstract features output by the high-dimensional convolutional network in the initial monocular depth estimation network model;

[0046] High-dimensional information is obtained by upsampling and fusing the high-dimensional abstract features.

[0047] Furthermore, the device further comprises:

[0048] The acquisition module is further configured to acquire continuous frame image samples and perform frame extraction processing on the continuous frame image samples at equal time intervals to obtain training sample images corresponding to a plurality of image frames;

[0049] The construction module is further configured to construct a training sample image set based on the training sample images.

[0050] Furthermore, the determining module includes:

[0051] a filtering unit, configured to perform discrete point pixel filtering on the depth information of the target vehicle in the depth image to obtain filtered depth information of the target vehicle;

[0052] The fitting unit is used to perform fitting processing on the depth information of the target vehicle after the filtering processing using a distance histogram distribution algorithm to obtain the distance between the target vehicle and the image acquisition device.

[0053] According to another aspect of the present invention, a storage medium is provided, wherein the storage medium stores at least one executable instruction, and the executable instruction enables a processor to perform operations corresponding to the above-mentioned monocular depth estimation method for irregular roads.

[0054] According to another aspect of the present invention, there is provided a terminal, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus;

[0055] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the above-mentioned monocular depth estimation method for irregular roads.

[0056] By means of the above technical solution, the technical solution provided by the embodiment of the present invention has at least the following advantages:

[0057] The present invention provides a monocular depth estimation method and device for irregular roads, a storage medium, and a terminal. In an embodiment of the present invention, an initial image is acquired from an image acquisition device, and an effective target extraction is performed on the initial image to obtain a target image that uniquely contains a target vehicle; depth information is extracted from the target image using a trained monocular depth estimation network model to obtain a depth image of the target image, where the trained monocular depth estimation network model is trained based on high-dimensional information extracted by a high-dimensional convolutional network; the distance between the target vehicle and the image acquisition device is obtained by fitting the depth information of the target vehicle in the depth image, thereby greatly reducing the interference of the mining environment on image feature extraction, enabling the model to converge stably, reducing the difficulty of model training, and at the same time ensuring the accuracy of depth information extraction, thereby greatly improving the applicability of monocular depth estimation to mining environments.

[0058] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present invention. The same reference symbols are used throughout the drawings to represent the same components. In the drawings:

[0060] Figure 1 A flow chart of a monocular depth estimation method for irregular roads provided by an embodiment of the present invention is shown;

[0061] Figure 2 A flow chart of another monocular depth estimation method for irregular roads provided by an embodiment of the present invention is shown;

[0062] Figure 3 A flow chart of another monocular depth estimation method for irregular roads provided by an embodiment of the present invention is shown;

[0063] Figure 4 The following is a block diagram showing the composition of a monocular depth estimation device for irregular roads provided by an embodiment of the present invention;

[0064] Figure 5 A schematic structural diagram of a terminal provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0065] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0066] When unmanned driving equipment faces irregular roads, the road structure is irregular, the road conditions are complex, and cliffs, mounds, etc. are very close to road features, which makes it difficult to estimate the distance, with low accuracy, or even impossible to obtain an estimation result. The embodiment of the present invention provides a monocular depth estimation method for irregular roads, such as Figure 1 As shown, the method includes:

[0067] 101. Acquire an initial image from an image acquisition device, and obtain a target image that uniquely includes a target vehicle by performing effective target extraction on the initial image.

[0068] In an embodiment of the present invention, the image acquisition device is a vehicle-mounted image acquisition device installed on an unmanned mining vehicle, which may be a driving recorder, a vehicle-mounted camera, etc. The embodiment of the present invention does not make any specific limitations. The initial image is a single road image taken by the image acquisition device, and the road image includes at least one target vehicle. The target vehicle is a vehicle within the field of view of the image acquisition device that needs to be identified. In order to eliminate the interference of non-target vehicles in the image, after obtaining the initial image, an image containing only the rectangular area where the target vehicle is located is extracted from the initial image, that is, the target image. For example, target detection is performed on the initial image to determine the target vehicle in the initial image, and the pixel values ​​of the image in the area outside the target vehicle area are set to fixed values, so that the feature extraction network model cannot identify the non-target vehicle area.

[0069] It should be noted that compared to public roads, roads are primarily composed of gravel, soil, vegetation, potholes, and other components. The pavement structure is complex, the image features are dense, and the similarity between features is high. Due to the irregularity of mining roads, the number of effective key points in the feature extraction process of the initial image is relatively small, which is not conducive to the extraction of effective features and the correct convergence of the convolutional network model. Therefore, it is more difficult to extract depth information from the entire image area in the initial image. By detecting the target vehicle in the initial image and extracting the only target image containing the target vehicle, the difficulty of extracting the target vehicle's features in the initial image can be effectively reduced, thereby greatly improving the accuracy of subsequent target image feature extraction.

[0070] 102. Use the trained monocular depth estimation network model to extract depth information from the target image to obtain a depth image of the target image.

[0071] In this embodiment of the present invention, the monocular depth estimation network model is a pre-trained multi-layer convolutional network model. Each convolutional network layer extracts different depth information for each pixel in the target image. Because the pixels in the image area where non-target vehicles are located in the target image are assigned fixed values, the convolutional network cannot extract depth information for pixels in this area. Therefore, the convolutional network only extracts depth information for the pixels in the image area where the target vehicle is located, obtaining a depth image of the target vehicle in the target image.

[0072] It should be noted that the trained monocular depth estimation network model is obtained by training based on high-dimensional information extracted by a high-dimensional convolutional network. Since the feature information extracted by the shallow convolutional network is surface and appearance-related, this feature information does not have significant effectiveness in the irregular road environment of the mining area. Therefore, training is performed based on the high-dimensional abstract features extracted by the high-dimensional convolutional network in the multi-layer convolutional network. For example, the monocular depth estimation network model is a 5-layer convolutional network. After each layer of the convolutional network completes feature extraction, only the abstract feature information output by the 3rd and 4th layers of the convolutional network is extracted. Image reconstruction is performed based on the high-dimensional distance information obtained based on the abstract feature information, and the loss function is determined based on the reconstructed image to obtain the trained monocular depth estimation network model. By extracting only abstract feature information for image reconstruction, the interference of shallow feature information such as brightness, illumination, and target vehicle structure can be eliminated, and the image reconstruction process can be prevented from being affected by the environment and noise, thereby guiding the convolutional network model to converge correctly and quickly.

[0073] 103. Obtain the distance between the target vehicle and the image acquisition device by performing fitting processing on the depth information of the target vehicle in the depth image.

[0074] In an embodiment of the present invention, the depth image is a three-dimensional matrix stored according to the image pixel position, including the distance information between each pixel point of the target vehicle in the target image and the image acquisition device. Since the distances of each pixel point are irregularly distributed, it is necessary to fit the distances corresponding to each pixel point to determine a unique distance as the distance between the target vehicle and the image acquisition device. For example, multiple peak distances are extracted from all the distances, and the multiple peak distances are weighted to obtain unique distance information. By fitting all the distances in the depth information, the interference of a small amount of extreme distances in the depth information can be eliminated, and the most representative distance information can be extracted, thereby effectively improving the accuracy of monocular depth estimation.

[0075] In one embodiment of the present invention, for further explanation and limitation, as Figure 2 As shown, step 101 of extracting effective targets from the initial image to obtain a target image that uniquely includes the target vehicle includes:

[0076] 201. Perform target detection on the initial image. If a target is detected, determine at least one target vehicle in the initial image.

[0077] 202. Determine a non-target vehicle image region in the initial image based on the target vehicle image region, and obtain a target image by performing masking on the non-target vehicle image region;

[0078] 203. If the target is not detected, the initial image is acquired again.

[0079] In the embodiment of the present invention, due to the complex mining environment, the road and the surrounding environment, such as cliffs and mounds, are extremely similar. If a monocular depth estimation network model is used to extract depth information from the entire image area, the model will find it difficult to achieve stable convergence, resulting in monocular depth estimation errors. Therefore, a target detection method is used to identify the circumscribed rectangular image area of ​​the target vehicle that needs the most attention in the image, and the pixel values ​​of the image area outside the circumscribed rectangle of the target vehicle are set to a fixed value to achieve the effect of masking the non-target vehicle image area. The number of target vehicles can be one or more. The target detection algorithm can be a Yolo (You Only Look Once) algorithm or other object recognition and positioning algorithm, which is not specifically limited in the embodiment of the present invention. By performing target detection and masking on the image, only the target vehicle whose distance needs to be estimated is retained in the image, and invalid features in the mining road image are removed, so that the monocular depth estimation network model can converge stably, thereby improving the applicability of the monocular depth estimation network model to the mining environment.

[0080] In one embodiment of the present invention, for further explanation and limitation, as Figure 2 As shown, before extracting depth information from the target image using the trained monocular depth estimation network model in step 102 to obtain a depth image of the target image, the method further includes:

[0081] 301. Build an initial monocular depth estimation network model containing at least five layers of convolutional networks.

[0082] 302. Perform mask processing on each sample image in the training sample image set to obtain a masked training sample image set.

[0083] 303. Train the initial monocular depth estimation network model using the masked training sample image set to obtain a trained monocular depth estimation network model.

[0084] In an embodiment of the present invention, an initial monocular depth estimation network model is pre-constructed in order to accurately extract features from the target image. In order to extract all features of the image, at least five layers of convolutional networks are constructed in the initial monocular depth estimation network model to achieve complete extraction of shallow structural features and high-dimensional abstract features. The masking process for the training sample images is the same as the masking process for the initial images and will not be repeated here. Training the initial monocular depth estimation network model using the masked training sample image set can enable the initial model to converge stably and accurately estimate the depth information of the target vehicle.

[0085] In one embodiment of the present invention, for further explanation and limitation, the step 303 of training the initial monocular depth estimation network model using the masked training sample image set to obtain a trained monocular depth estimation network model includes:

[0086] The initial monocular depth estimation network model is used to extract high-dimensional information from the masked training sample image to obtain high-dimensional information, and the image is reconstructed using the high-dimensional information to obtain a reconstructed image of the training sample image.

[0087] A loss function is determined according to the similarity between the reconstructed image and the training sample image.

[0088] The initial monocular depth estimation network model is trained using the loss function to obtain a trained monocular depth estimation network model.

[0089] In an embodiment of the present invention, in order to determine the training effect of the initial monocular depth estimation network model, image reconstruction is performed based on high-dimensional information. Specifically, key points are extracted from the previous and next frame images of the current training sample image, and the high-dimensional information is matched with the extracted key points to generate a reconstructed image of the current training sample image. Furthermore, the model weights are continuously corrected by calculating the similarity between the current training sample image and the reconstructed image until the model converges, that is, the similarity between the current training sample image and the reconstructed image is maximized and remains unchanged within a preset time, thereby obtaining a monocular depth estimation network model that has completed training. The preset time can be customized according to the actual needs of model training, and the key point extraction algorithm can be Canny, Sobel, SIFT, SURF, etc. The embodiment of the present invention does not specifically limit the preset time and key point extraction algorithm.

[0090] In one embodiment of the present invention, for further explanation and limitation, the extracting high-dimensional information from the training sample image using the initial monocular depth estimation network model to obtain high-dimensional information includes:

[0091] The initial monocular depth estimation network model is used to extract depth information of the target image, and high-dimensional abstract features output by the high-dimensional convolutional network in the initial monocular depth estimation network model are extracted.

[0092] High-dimensional information is obtained by upsampling and fusing the high-dimensional abstract features.

[0093] In an embodiment of the present invention, each layer of the convolutional network of the initial monocular depth estimation network model extracts features of different dimensions in the target image respectively. For example, in a five-layer convolutional network model, the 1st and 2nd layer convolutional networks extract shallow structural features, and the 3rd and 4th layer convolutional networks extract high-dimensional abstract features. In order to further filter out insignificant features and ensure the stable convergence of the model, only the high-dimensional abstract features extracted by the high-dimensional convolutional network are extracted to determine the depth information training results. Since the image size will continue to decrease as the dimension of the convolutional network increases during the process of extracting image features by the convolutional network, in order to obtain accurate high-dimensional information, upsampling is required to restore the image size when using the image features. Furthermore, the features output by each layer of the high-dimensional convolutional network are averagely weighted to obtain high-dimensional information. Among them, upsampling can use the adjacent pixel value calculation method or other upsampling algorithms, which are not specifically limited in the embodiment of the present invention.

[0094] It's important to note that in the complex environment of mining areas, high-dimensional features are significantly more effective than shallower features. Extracting high-dimensional information from target vehicle images as training results for the model can further filter out invalid features in the image, providing more accurate positive guidance for the model, enabling rapid and stable model convergence. This effectively improves the accuracy of feature extraction while shortening model training time.

[0095] In one embodiment of the present invention, for further explanation and limitation, before training the initial monocular depth estimation network model using the masked training sample image set in step 301, the method further includes:

[0096] Continuous frame image samples are acquired, and frame extraction processing is performed on the continuous frame image samples at equal time intervals to obtain training sample images corresponding to multiple image frames.

[0097] A training sample image set is constructed according to the training sample images.

[0098] In an embodiment of the present invention, in order to ensure the training effect of the model, continuous frame image samples are extracted at equal time intervals to obtain multiple training sample images. The continuous frame image samples are the entire video data collected by vehicle-mounted image acquisition devices such as driving recorders and vehicle-mounted cameras. The time interval for extracting frames can be customized according to the needs of the actual application scenario, and the embodiment of the present invention does not make any specific limitations. By extracting frames from the video data at equal time intervals and extracting training sample images, the training sample images can have both similar feature information and certain differences, thereby ensuring the training difficulty of the model and improving the generalization ability of the model. In addition, the continuous frame image samples also facilitate image reconstruction, thereby improving the accuracy of the loss function.

[0099] In one embodiment of the present invention, for further explanation and limitation, step 103 of obtaining the distance between the target vehicle and the image acquisition device by fitting the depth information of the target vehicle in the depth image includes:

[0100] Perform discrete point pixel filtering processing on the target vehicle depth information in the depth to obtain filtered target vehicle depth information.

[0101] The depth information of the target vehicle after the filtering process is fitted using a distance histogram distribution algorithm to obtain the distance between the target vehicle and the image acquisition device.

[0102] In an embodiment of the present invention, the depth image includes the distance of each pixel in the circumscribed rectangular image area of ​​the target vehicle. In order to determine a unique distance value as the distance between the target vehicle and the image acquisition device, the distance data in the depth information needs to be fitted. Specifically, 1) the relatively discrete distance values ​​in the distance values ​​of each pixel are removed, 2) the pixel points are segmented according to the distance accuracy of equal time intervals, and the number of pixels in each distance segment of the target vehicle pixel points is counted to form a distance distribution histogram, 3) the distance segment in which the number of pixels is greater than the preset proportion is counted as the distance peak, 4) the weighted summation calculation is performed on each distance peak to obtain a unique distance value, that is, the distance between the target vehicle and the image acquisition device. Among them, the preset proportion can be 60%, and can also be customized according to actual application requirements, and is not specifically limited in the embodiment of the present invention.

[0103] The present invention provides a monocular depth estimation method for irregular roads. An embodiment of the present invention obtains an initial image from an image acquisition device, and obtains a target image that uniquely contains a target vehicle by performing effective target extraction on the initial image; uses a trained monocular depth estimation network model to extract depth information from the target image to obtain a depth image of the target image, wherein the trained monocular depth estimation network model is trained based on high-dimensional information extracted by a high-dimensional convolutional network; and obtains the distance between the target vehicle and the image acquisition device by fitting the depth information of the target vehicle in the depth image, thereby greatly reducing the interference of the mining environment on image feature extraction, enabling the model to converge stably, reducing the difficulty of model training, and at the same time ensuring the accuracy of feature extraction, thereby greatly improving the applicability to the mining environment and the accuracy of monocular depth estimation.

[0104] Furthermore, as a response to the above Figure 1 The embodiment of the present invention provides a monocular depth estimation device for irregular roads, such as Figure 4 As shown, the device includes:

[0105] An acquisition module 41 is configured to acquire an initial image from an image acquisition device and obtain a target image that uniquely includes a target vehicle by performing effective target extraction on the initial image;

[0106] An extraction module 42 is configured to extract depth information from the target image using the trained monocular depth estimation network model to obtain a depth image of the target image;

[0107] The determination module 43 is configured to obtain the distance between the target vehicle and the image acquisition device by performing fitting processing on the depth information of the target vehicle in the depth image.

[0108] Furthermore, the acquisition module includes:

[0109] a target detection unit, configured to perform target detection on the initial image, and if a target is detected, determine at least one target vehicle in the initial image;

[0110] a mask processing unit, configured to determine a non-target vehicle image region in the initial image based on the image region of the target vehicle, and obtain a target image by performing mask processing on the non-target vehicle image region;

[0111] The acquisition unit is configured to reacquire the initial image if no target is detected.

[0112] Furthermore, the device further comprises:

[0113] A construction module for building an initial monocular depth estimation network model containing at least five layers of convolutional networks;

[0114] A mask processing module is used to perform mask processing on each sample image in the training sample image set to obtain a masked training sample image set;

[0115] A training module is used to train the initial monocular depth estimation network model using the training sample image set after the mask processing to obtain a trained monocular depth estimation network model.

[0116] Furthermore, the training module includes:

[0117] A reconstruction unit is used to extract high-dimensional information from the masked training sample image using the initial monocular depth estimation network model to obtain high-dimensional information, and reconstruct the image using the high-dimensional information to obtain a reconstructed image of the training sample image;

[0118] a determining unit, configured to determine a loss function according to a similarity between the reconstructed image and the training sample image;

[0119] A training unit is used to train the initial monocular depth estimation network model using the loss function to obtain a trained monocular depth estimation network model.

[0120] Furthermore, in a specific application scenario, the training unit is used to extract depth information from the target image using the initial monocular depth estimation network model, and extract high-dimensional abstract features output by the high-dimensional convolutional network in the initial monocular depth estimation network model;

[0121] High-dimensional information is obtained by upsampling and fusing the high-dimensional abstract features.

[0122] Furthermore, the device further comprises:

[0123] The acquisition module is further configured to acquire continuous frame image samples and perform frame extraction processing on the continuous frame image samples at equal time intervals to obtain training sample images corresponding to a plurality of image frames;

[0124] The construction module is further configured to construct a training sample image set based on the training sample images.

[0125] Furthermore, the determining module includes:

[0126] a filtering unit, configured to perform discrete point pixel filtering on the depth information of the target vehicle in the depth image to obtain filtered depth information of the target vehicle;

[0127] The fitting unit is used to perform fitting processing on the filtered target vehicle depth information using a distance histogram distribution algorithm to obtain the distance between the target vehicle and the image acquisition device.

[0128] The present invention provides a monocular depth estimation device. An embodiment of the present invention obtains an initial image from an image acquisition device, and obtains a target image that uniquely contains a target vehicle by performing effective target extraction on the initial image; uses a trained monocular depth estimation network model to extract depth information from the target image to obtain a depth image of the target image, and the trained monocular depth estimation network model is trained based on high-dimensional information extracted by a high-dimensional convolutional network; and obtains the distance between the target vehicle and the image acquisition device by fitting the depth information of the target vehicle in the depth image, thereby greatly reducing the interference of the mining environment on image feature extraction, enabling the model to converge stably, reducing the difficulty of model training, and at the same time ensuring the accuracy of feature extraction, thereby greatly improving the accuracy of monocular depth estimation and its applicability to mining environments.

[0129] According to one embodiment of the present invention, a storage medium is provided, wherein the storage medium stores at least one executable instruction. The computer-executable instruction can execute the monocular depth estimation method for irregular roads in any of the above method embodiments.

[0130] Figure 5 A schematic structural diagram of a terminal provided according to an embodiment of the present invention is shown. The specific embodiment of the present invention does not limit the specific implementation of the terminal.

[0131] like Figure 5 As shown, the terminal may include: a processor (processor) 502 , a communications interface (Communications Interface) 504 , a memory (memory) 506 , and a communication bus 508 .

[0132] The processor 502 , the communication interface 504 , and the memory 506 communicate with each other via a communication bus 508 .

[0133] The communication interface 504 is used to communicate with other devices such as clients or other servers.

[0134] The processor 502 is configured to execute the program 510 , and specifically to execute the relevant steps in the embodiment of the monocular depth estimation method for irregular roads.

[0135] Specifically, the program 510 may include program codes, which include computer operation instructions.

[0136] Processor 502 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention. The one or more processors included in the terminal may be processors of the same type, such as one or more CPUs, or processors of different types, such as one or more CPUs and one or more ASICs.

[0137] The memory 506 is used to store the program 510. The memory 506 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0138] The program 510 can be specifically used to cause the processor 502 to perform the following operations:

[0139] Acquire an initial image from an image acquisition device, and obtain a target image that uniquely includes a target vehicle by performing effective target extraction on the initial image;

[0140] Extracting depth information from the target image using a trained monocular depth estimation network model to obtain a depth image of the target image, wherein the trained monocular depth estimation network model is trained based on high-dimensional information extracted by a high-dimensional convolutional network;

[0141] The distance between the target vehicle and the image acquisition device is obtained by fitting the depth information of the target vehicle in the depth image.

[0142] Obviously, those skilled in the art will appreciate that the various modules or steps of the present invention described above can be implemented using a general-purpose computing device, centralized on a single computing device, or distributed across a network of multiple computing devices. Alternatively, they can be implemented using program code executable by a computing device, which can then be stored in a storage device and executed by the computing device. In some cases, the steps shown or described can be performed in a different order than that shown, or can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0143] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A monocular depth estimation method for irregular roads, characterized in that: include: Acquire an initial image from an image acquisition device, and obtain a target image that uniquely includes a target vehicle by performing effective target extraction on the initial image; Extracting depth information from the target image using a trained monocular depth estimation network model to obtain a depth image of the target image, wherein the trained monocular depth estimation network model is trained based on high-dimensional information extracted by a high-dimensional convolutional network, wherein the high-dimensional information is obtained by averaging weighted features output by each layer of the high-dimensional convolutional network; The distance between the target vehicle and the image acquisition device is obtained by fitting the depth information of the target vehicle in the depth image, including: performing discrete point pixel filtering on the depth information of the target vehicle in the depth image to obtain the filtered depth information of the target vehicle; and using a distance histogram distribution algorithm to fit the depth information of the filtered target vehicle to obtain the distance between the target vehicle and the image acquisition device.

2. The method according to claim 1, characterized in that The step of extracting effective targets from the initial image to obtain a target image that uniquely includes the target vehicle comprises: Performing target detection on the initial image, and if a target is detected, determining at least one target vehicle in the initial image; Determining a non-target vehicle image region in the initial image according to the image region of the target vehicle, and obtaining a target image by performing mask processing on the non-target vehicle image region; If the target is not detected, the initial image is acquired again.

3. The method according to claim 1, characterized in that Before extracting depth information from the target image using the trained monocular depth estimation network model to obtain a depth image of the target image, the method further includes: Build an initial monocular depth estimation network model containing at least five layers of convolutional networks; Performing mask processing on each sample image in the training sample image set to obtain a masked training sample image set; The initial monocular depth estimation network model is trained using the masked training sample image set to obtain a trained monocular depth estimation network model.

4. The method according to claim 3, characterized in that The method of training the initial monocular depth estimation network model using the masked training sample image set to obtain a trained monocular depth estimation network model includes: Using the initial monocular depth estimation network model to extract high-dimensional information from the masked training sample image to obtain high-dimensional information, and reconstructing the image using the high-dimensional information to obtain a reconstructed image of the training sample image; Determining a loss function based on the similarity between the reconstructed image and the training sample image; The initial monocular depth estimation network model is trained using the loss function to obtain a trained monocular depth estimation network model.

5. The method according to claim 4, characterized in that The extracting high-dimensional information from the training sample image using the initial monocular depth estimation network model to obtain high-dimensional information includes: Extracting depth information from the target image using the initial monocular depth estimation network model, and extracting high-dimensional abstract features output by the high-dimensional convolutional network in the initial monocular depth estimation network model; High-dimensional information is obtained by upsampling and fusing the high-dimensional abstract features.

6. The method according to claim 3, characterized in that Before training the initial monocular depth estimation network model using the masked training sample image set, the method further includes: Acquire continuous frame image samples, and perform frame extraction processing on the continuous frame image samples at equal time intervals to obtain training sample images corresponding to multiple image frames; A training sample image set is constructed according to the training sample images.

7. A monocular depth estimation device for irregular roads, characterized in that: include: An acquisition module is used to acquire an initial image from an image acquisition device, and obtain a target image that uniquely contains a target vehicle by performing effective target extraction on the initial image; An extraction module is used to extract depth information from the target image using a trained monocular depth estimation network model to obtain a depth image of the target image, wherein the trained monocular depth estimation network model is trained based on high-dimensional information extracted by a high-dimensional convolutional network, wherein the high-dimensional information is obtained by averaging weighted features output by each layer of the high-dimensional convolutional network; A determination module is used to obtain the distance between the target vehicle and the image acquisition device by fitting the depth information of the target vehicle in the depth image, including: performing discrete point pixel filtering on the depth information of the target vehicle in the depth image to obtain the filtered depth information of the target vehicle; and using a distance histogram distribution algorithm to fit the depth information of the filtered target vehicle to obtain the distance between the target vehicle and the image acquisition device.

8. A storage medium storing at least one executable instruction, wherein the executable instruction causes a processor to perform operations corresponding to the monocular depth estimation method for an irregular road according to any one of claims 1 to 6.

9. A terminal comprising: A processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the monocular depth estimation method for irregular roads according to any one of claims 1 to 6.

Images