A distance detection method, device and storage medium based on a monocular camera on an intelligent connected car

Abstract

The present invention provides a distance detection method, device and storage medium based on a monocular camera on an intelligent networked vehicle, and relates to the field of image processing technology. In the present invention, a monocular camera on a vehicle is used to shoot the outside of the vehicle to obtain an environmental image; target detection is performed on the environmental image to obtain an image of the detection target; perspective transformation is performed on the image of the detection target to obtain a first depth image; the environmental image is input into a depth estimation model to obtain a second depth image; the images of the detection target in the first depth image and the second depth image are fused to obtain the depth value of the detection target. The present invention obtains a more accurate distance detection method through perspective transformation and a depth estimation model.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a distance detection method, device and storage medium based on a monocular camera on an intelligent networked vehicle. Background Art

[0002] In the development of intelligent connected vehicles, environmental perception technology is one of the key technologies to achieve autonomous driving and improve road safety. Traditional environmental perception systems usually rely on multi-sensor fusion technologies, such as LiDAR, millimeter-wave radar, and multi-camera systems to capture detailed information about the surrounding environment. Although such systems have excellent perception accuracy, they are limited by their high cost, complex system architecture, and large demand for computing resources. In addition, these systems must accurately interpret the surrounding environment, such as vehicles, pedestrians, and traffic signs, in real time to ensure driving safety and comfort. In this context, edge computing technology has become an increasingly important research direction because it can process large amounts of data on the vehicle side, significantly reduce latency, and improve system response speed, especially when the network connection is unstable.

[0003] As a more economical perception device, monocular cameras have been widely used in intelligent connected vehicles due to their easy installation, high cost-effectiveness and easy maintenance. The images collected by monocular cameras, combined with cutting-edge image processing and deep learning technologies, can effectively realize the perception of the surrounding environment. However, since monocular cameras can only provide two-dimensional images, it is difficult to directly obtain the depth information of objects, which limits their application in the field of environmental perception. In order to overcome this limitation, depth estimation technology has gradually received attention in recent years. By inferring depth information from monocular images, it is expected to make up for the shortcomings of monocular cameras in obtaining depth information.

[0004] In the actual driving process of intelligent connected vehicles, it is necessary not only to accurately perceive the surrounding environment, but also to convert the perceived information into specific position information in the vehicle coordinate system to support the vehicle's autonomous decision-making and control. This process usually requires complex coordinate transformation and spatial positioning algorithms, which places higher requirements on calculation accuracy and real-time performance. Under existing technical conditions, coordinate transformation often relies on the depth information provided by a multi-sensor system, which makes the system that relies only on a monocular camera face a huge challenge in how to quickly and accurately complete the coordinate transformation.

[0005] Based on the above problems, the present invention is proposed. Summary of the invention

[0006] In view of the above problems, the present invention proposes a distance detection method, device and storage medium based on a monocular camera on an intelligent connected car. Through perspective transformation and depth estimation model, a more accurate distance detection method is obtained to solve the problem that the single distance measurement method in the prior art is not accurate enough in complex scenes, and the traditional monocular distance measurement method often leads to inaccurate distance estimation due to environmental changes (such as lighting changes, obstructions, etc.), and it is difficult to strike a balance between real-time performance and accuracy.

[0007] In a first aspect, the present invention provides a distance detection method based on a monocular camera on an intelligent connected vehicle, comprising:

[0008] The monocular camera on the vehicle is used to shoot the outside of the vehicle to obtain an environmental image;

[0009] Performing target detection on the environment image to obtain an image of the detected target;

[0010] Performing a perspective transformation on the image of the detection target to obtain a first depth image;

[0011] Inputting the environment image into a depth estimation model to obtain a second depth image;

[0012] The images of the detection target in the first depth image and the second depth image are fused to obtain a depth value of the detection target; the depth of the detection target is the distance between the detection target and the monocular camera;

[0013] Among them, the depth estimation model includes: a convolutional encoder module, a sparse depth map generation module and a Transformer encoder module connected in parallel to the output of the convolutional encoder module, and a multimodal fusion decoder module connected to the outputs of the sparse depth map generation module and the Transformer encoder module.

[0014] In a second aspect, the present invention provides an electronic device, comprising:

[0015] at least one processor, and a memory communicatively coupled to at least one of the processors;

[0016] The memory stores instructions that can be executed by at least one of the processors, and the instructions are executed by at least one of the processors so that at least one of the processors can execute the distance detection method based on a monocular camera on an intelligent connected vehicle described in any embodiment.

[0017] In a third aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, and the computer instructions can be executed by one or more processors to implement the distance detection method based on a monocular camera on an intelligent connected vehicle as described in any embodiment.

[0018] Compared with the prior art, the present invention has the following technical effects:

[0019] 1. Improved accuracy: Through perspective transformation and depth estimation model, the present invention can estimate the distance of an object more accurately. This method combines the advantages of two distance measurement methods and effectively improves the accuracy of distance measurement under various environmental conditions.

[0020] 2. Enhanced robustness: The present invention combines two different ranging methods, perspective transformation and depth estimation, so that it can still maintain a high ranging accuracy when facing complex scenes (such as occlusion and lighting changes), thereby improving the overall robustness of the system.

[0021] 3. Balance between real-time performance and accuracy: The present invention can ensure high-precision detection while maintaining real-time requirements, and is suitable for scenarios with high real-time requirements such as intelligent driving systems.

[0022] 4. Wide scope of application: The present invention is not only suitable for distance measurement in intelligent connected vehicles, but can also be widely used in other scenarios requiring monocular vision ranging, such as drones and robot navigation, and has broad application prospects.

[0023] In summary, the present invention solves the distance measurement problem of monocular vision systems in complex scenes by integrating multiple distance measurement methods, and provides technical support for intelligent driving and other visual application fields.

[0024] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above description and other purposes, features and advantages of the present invention more obvious and easy to understand, the preferred embodiments are specifically cited and described in detail as follows. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0026] Figure 1 A flow chart of a distance detection method based on a monocular camera on an intelligent connected vehicle provided by an embodiment of the present invention;

[0027] Figure 2 A schematic diagram of calibration points provided by an embodiment of the present invention;

[0028] Figure 3 is a structural diagram of a depth estimation model provided by an embodiment of the present invention;

[0029] Figure 4 A structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0031] In the description of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be connected, detachably connected, or integrated; it can be mechanically connected or electrically connected; it can be directly connected or indirectly connected through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0032] Example 1

[0033] The present invention proposes a distance detection method based on a monocular camera on an intelligent networked vehicle, which is applicable to the case where a monocular camera installed on the vehicle is used to measure the distance of an external target. The method can be executed by an electronic device.

[0034] See also Figure 1 , the method specifically comprises the following steps:

[0035] S110: photographing the exterior of the vehicle through a monocular camera on the vehicle to obtain an environmental image.

[0036] The monocular camera is placed at the front of the vehicle to maximize the capture of environmental information in front of the vehicle. This setup enables the system to obtain high-quality visual data in real time, providing reliable input for target detection.

[0037] The environment image captured by the monocular camera is the image in front of the vehicle. It is a two-dimensional image and does not have depth information.

[0038] S120: Perform target detection on the environment image to obtain an image of the detected target.

[0039] Optionally, an advanced target detection model YOLOv8 is used to identify and select obstacles in the environment image, that is, to detect targets. The YOLOv8 model is an efficient target detection algorithm that uses deep learning technology to achieve fast and accurate recognition of different types of targets.

[0040] These obstacles may include pedestrians, other vehicles, or any object that may affect the vehicle's movement. Each obstacle is given a rectangular box in the image, and the image coordinates of this rectangular box can accurately represent the position of the obstacle in the field of view.

[0041] Next, determine the midpoint of the bottom edge of each obstacle box. This point is selected as the coordinate point representing the position of the obstacle in the space in front of the vehicle. The reason for selecting the midpoint of the bottom edge is that it is usually closest to the point where the obstacle touches the ground, thus providing the most accurate reference for coordinate transformation. In addition, the object detection algorithm can also provide information about the width of the obstacle box.

[0042] Optionally, the object detection model needs to be trained. Optionally, to improve the training accuracy, environmental image samples on real roads are collected to train the object detection model, ensuring that the model can adapt to various driving conditions. In this way, the model can detect multiple targets such as pedestrians, bicycles, various motor vehicles and non-motor vehicles in real time, greatly enhancing the environmental perception ability of the autonomous vehicle. This is crucial to improving the safety of autonomous vehicles, because various obstacles and pedestrians on the road can be identified in time, allowing the vehicle to respond quickly and avoid potential collisions.

[0043] S130: Perform perspective transformation on the image of the detection target to obtain a first depth image.

[0044] Before performing perspective transformation, it is necessary to calibrate the known environmental points. This includes the following two steps:

[0045] S131. Determine the perspective transformation matrix from the image coordinate system of the monocular camera to the world coordinate system by calibrating known environmental points.

[0046] First, call the calibration program to open the monocular camera, mark two points on the left and right sides of the near-vehicle screen in the monocular camera (the two points on the near-vehicle side), and then synchronously translate these two points 50 meters to the distance to obtain two new points (the two points on the far-vehicle side). The 50 meters here can also be adjusted. Then record the image coordinates of these four points on the image taken by the monocular camera, as well as the length and width of the rectangular area formed by the four points in the world coordinate system and the distance between the two points on the near-vehicle side and the front of the vehicle. Note that the rectangular area here should be as large as possible, that is, the larger the rectangular area, the better the coordinate conversion effect. For example Figure 2 As shown, the calibration program will display the image coordinates of the 4 marked points. Figure 2 Four points on the environment image are shown.

[0047] Then run the coordinate conversion program, in which the image coordinates of the four points after conversion have been preset. By inputting the image coordinates of the four points and the length and width of the rectangular area in the world coordinate system, the conversion matrix is ​​obtained. The present invention can simulate the way a lens captures real-world scenes through perspective transformation, and realize image transformation by changing the viewing angle. The basic principle is to project a picture or coordinate point to a new visual plane. This process provides great flexibility and allows any quadrilateral area to be mapped to another quadrilateral area. Whether it is stretching, compression or distortion transformation, it can be achieved through perspective transformation.

[0048] The math behind perspective transformations is based on matrix multiplication, using a 3x3 transformation matrix. This transformation matrix is ​​able to represent mappings from one plane to another, including rotations, scaling, translations, and more complex perspective distortions. Perspective transformations use homogeneous coordinates to represent 2D vectors. This is because in a homogeneous coordinate system, affine and perspective transformations can be represented using linear algebra, not just simple linear transformations. Homogeneous coordinates are implemented by adding an extra dimension, which converts a 2D coordinate (x, y) into a 3D coordinate (x, y, 1).

[0049] In perspective transformation, by applying a 3x3 transformation matrix, the new position of the transformed point can be calculated. This process can be expressed as the following matrix multiplication:

[0050] ; (1)

[0051] in, is the perspective transformation matrix, a , b , c , d , e , f , g , h , t are the elements in the perspective transformation matrix. x' and y' are the transformed coordinates, and w' is the third dimension in the homogeneous coordinate system. To convert the result back to two-dimensional space, we divide x' and y' by w', which gives the final coordinates ( ):

[0052] ; (2)

[0053] This approach allows perspective transformation to handle more complex perspective transformation problems, such as simulating changes in camera angles, adjusting viewing angles, etc., rather than just simple in-plane transformations. In actual applications, a set of known environmental points in a rectangular area are preset within the software (i.e., the four points near the vehicle end and the far vehicle end mentioned above). These points represent the ideal layout of the road in the real world. At the same time, the image coordinates of the rectangular area (also four points) are extracted from the image captured by the monocular camera. With these two sets of coordinate points, the system can automatically calculate the perspective transformation matrix required to map the coordinate points in the image coordinate system to the coordinate points in the world coordinate system.

[0054] S132, inputting the coordinates of each point in the image of the detection target into the perspective transformation matrix to obtain the three-dimensional coordinates of each point in the environment.

[0055] Each point in the image of the detected target is passed through a coordinate conversion program to convert the image coordinates into coordinate points in the world coordinate system. This conversion process relies on the perspective transformation matrix previously obtained through image calibration technology. The converted three-dimensional coordinates provide the vehicle with: the lateral and longitudinal offset distances of the obstacle relative to the front of the vehicle, as well as the size of the obstacle, which is crucial for assessing the specific location, size and potential collision risk of the obstacle.

[0056] S140: Input the environment image into a depth estimation model to obtain a second depth image.

[0057] Figure 3 : is a structural diagram of a depth estimation model provided by an embodiment of the present invention. The depth estimation model includes: a convolutional encoder module, a sparse depth map generation module and a Transformer encoder module connected in parallel to the output of the convolutional encoder module, and a multimodal fusion decoder module connected to the outputs of the sparse depth map generation module and the Transformer encoder module.

[0058] Specifically, the convolutional encoder module uses a deep residual network (such as ResNet-50) as the backbone network, and connects a channel attention mechanism layer (such as SENet network) after each convolution layer in the backbone network to enhance the feature expression capability. The convolutional encoder module outputs a first feature map;

[0059] The sparse depth map generation module includes a shallow convolution layer; the first feature map is input into the shallow convolution layer to generate a sparse depth map. This sparse depth map provides preliminary depth information. The Transformer encoder module includes a mapping layer (Patch Embedding layer) and a multi-layer Transformer encoder. Specifically, the first feature map is input into the PatchEmbedding layer and converted into a fixed-length feature vector. Then the multi-layer Transformer encoder is connected, and finally the encoded global dense depth feature map is output. Among them, each layer of the Transformer encoder includes a multi-head self-attention mechanism (Multi-Head Self-Attention, MHSA) and a feed-forward network (Feed-Forward Network, FFN). The self-attention mechanism can effectively capture long-distance dependencies in the image and improve the ability to capture depth information in complex scenes.

[0060] The sparse depth map and the global dense depth feature map are input into the multimodal fusion decoder module, and the second depth image is output. The second depth image is actually a matrix of the same dimension as the environment image, and the matrix elements are the depth values ​​of each pixel. Among them, the multimodal fusion decoder module includes a cross-attention mechanism layer, multiple upsampling layers, a convolution layer, and an activation layer. The Cross-Attention mechanism layer can effectively associate features at different levels, making the model perform better. The output of each upsampling layer is connected to the convolution encoder module by skip connections according to the size of the output feature. Specifically, multiple upsampling layers gradually upsample the fusion features output by the Cross-Attention mechanism layer to increase the dimension to restore the specific details in the image. The deep residual network in the convolution encoder module includes multiple convolution layers, which actually downsamples the input image step by step to reduce the dimension. Therefore, the feature map output by any layer in the encoder module can be spliced ​​with the feature map of the same scale input by an upsampling layer, and then input into the upsampling layer. Each upsampled input feature map is concatenated with the feature map output by a layer of the convolutional encoder module, thereby forming a U-shaped network structure as a whole. This embodiment uses upsampling layers combined with skip connections to gradually restore high-resolution feature maps, while fusing low-level features from the convolutional encoder module, and generating the final second depth image through two convolutional layers and an activation layer.

[0061] The depth estimation model is trained as a whole. Environmental images taken on real roads can be collected in advance as training samples. The depth map corresponding to the environmental image can be obtained through vehicle-mounted radar or actual measurement as a label. The loss function is constructed based on the difference between the depth image output by the depth estimation model and the depth of the label at each pixel / point. The loss functions used include multi-scale depth loss (Multi-scale Depth Loss) and edge-aware loss (Edge-awareLoss) to ensure that the edge details of the image are retained while generating high-quality depth maps. By minimizing the loss function, the parameters in the depth estimation model are optimized.

[0062] The depth estimation model proposed in this embodiment is a model that combines a convolutional neural network (CNN) and a Transformer architecture. The model uses a convolutional encoder to extract local features, a Transformer to capture global dependencies, a sparse depth map generation module to generate a sparse depth map, and a multimodal fusion decoder module to integrate dense depth features and sparse depth features to generate a high-precision depth image.

[0063] S150: Fuse the images of the detection target in the first depth image and the second depth image to obtain a depth value of the detection target.

[0064] Perspective transformation itself depends on the geometric parameters of the camera, and is easily affected by various factors (such as object shape, light, and viewing angle) in complex scenes, resulting in reduced ranging accuracy. Therefore, the present invention can help make up for these deficiencies by combining depth estimation.

[0065] Comprehensively utilize the ranging results of perspective transformation and depth estimation to improve the overall ranging accuracy and reliability and reduce the limitations and errors of a single method. The depth values ​​of the same position points in the images of the detection target in the first depth image and the second depth image are weighted and summed to obtain the depth value of each position point of the detection target. The calculation of weights includes the following two fusion strategies. The first fusion strategy uses a deep learning model and has certain accuracy advantages, but the calculation process is slower than the second method; the second fusion strategy uses prior knowledge to calculate confidence, and the calculation process is relatively fast, but the accuracy is worse than the first method. You can choose according to actual needs.

[0066] The first fusion strategy: determining the scene type of the environment image according to the scene classification model; determining the weight of the first depth image and the weight of the image of the detection target in the second depth image according to the scene type.

[0067] Specifically, see the following formula:

[0068] ; (3)

[0069] in, is the depth value of the fused detection target, that is, the distance between the detection target and the monocular camera. is the depth value in the first depth image, is the depth value in the image of the detection target in the second depth image. is the weight of the first depth image, is the weight of the image of the detection target in the second depth image. Since the second depth image is derived from the environment image, it includes multiple detection targets and invalid background. It is necessary to extract the image of the detection target from the second depth image according to the detection frame output by the target detection model for calculation. Formula (3) performs weighted summation for the same position points, and finally obtains the depth values ​​of multiple position points.

[0070] Determined by the scene type of the environment image and For example, in long-distance scenes, the perspective transformation may be more accurate and be given a greater weight. In complex or unknown environments, depth estimation may be more reliable and be given a greater weight. The advantages of this method are: simple implementation, high computational efficiency, and the ability to dynamically adjust weights according to actual conditions to meet the needs of different scenarios.

[0071] The judgment of scene type is mainly completed through a lightweight scene classification model. First of all, data collection is required. Use the vehicle's camera to collect a large amount of driving picture data, including various typical scenes, such as long-distance open roads, complex urban environments, close-range parking lots, etc. Then annotate these data manually and divide each frame of the image into different scene categories, which are divided into three categories here: long-distance scenes, complex scenes, and close-range scenes. Then use the scene classification model to train on the dataset. The model structure consists of three consecutive groups of convolutional pooling layers and a fully connected layer. The convolution kernel size is 3x3, the pooling size is 2x2, and the ReLU activation function is used. Finally, the output layer outputs the category number to which the scene belongs. The training process uses cross entropy loss for classification tasks. When the model training is completed, the scene classification model is used for scene classification, and then different weight strategies are selected. Specifically, in long-distance scenes ,For example, =80%, =20%. In complex scenes, ,For example, =30%, =70%. In close-range scenes, and It can fluctuate between 40%-60%.

[0072] The second fusion strategy: obtain the calibration quality of the first depth image through a reprojection algorithm; obtain the reliability of the image of the detection target in the second depth image through the confidence and uncertainty of the depth estimation model; determine the weight of the first depth image and the weight of the image of the detection target in the second depth image based on the calibration quality and reliability.

[0073] When obtaining the calibration quality, the internal and external parameters of the monocular camera are solved by minimizing the reprojection error during the calibration process for known environmental points. The reprojection error is the difference between the real point and the reprojected point during the calibration process. The commonly used calculation method is:

[0074] ; (4)

[0075] in is a point on the environment image, ( , ) is the reprojection of points in the environment to points on the image through the calibration parameters, and Q is the total number of calibration points. Reprojection error The smaller it is, the higher the calibration quality. Normalized calibration quality :

[0076] ; (5)

[0077] Reprojection Error The smaller, The closer it is to 1, the higher the calibration quality. Here we can take the normalized calibration quality during the calibration process. As the calibration quality of the first depth image.

[0078] When obtaining reliability, first obtain the confidence of the depth estimation model. Since the depth estimation model outputs the depth value of each point in the image, it also gives the confidence of each point. Therefore, the confidence of the depth estimation model is actually a matrix, called the confidence matrix. The confidence matrix is ​​given after the depth estimation model outputs the second depth image. Then obtain the uncertainty of the depth estimation model. Specifically, it is calculated in the following way: by performing multiple forward propagations on the input environment image, multiple depth images are obtained, which contain multiple groups of depth values, and the variance or standard deviation of the depth values ​​at the same point between the multiple groups of depth values ​​is calculated, which is used as a measure of uncertainty.

[0079] ; (6)

[0080] Among them, take any point on the detection target, the coordinates on the environment image are (x, y), is the depth value of the point obtained by the i-th forward propagation, is the average of the P estimated depths of the point. Uncertainty The smaller it is, the more confident the depth estimation model is in the depth estimation of the pixel.

[0081] Considering the model output confidence matrix and uncertainty estimate, reliability can be defined :

[0082] ; (7)

[0083] in: It is the confidence of the coordinate point (x, y) output by the depth estimation model, and its value range is [0, 1]. is the uncertainty estimate of the coordinate point (x, y), the smaller it is, the higher the confidence level is.

[0084] This embodiment combines the confidence of the direct output of the depth estimation model with the uncertainty estimation, which can more accurately reflect the reliability of the depth estimation model.

[0085] Further, referring to formula (8), the calibration quality and reliability are calculated based on and .

[0086] ; (8)

[0087] ; (9)

[0088] The advantages of this strategy are: 1. The fusion process is more intelligent, can be adjusted adaptively, and has strong adaptability. 2. In a changing and uncertain environment, it can effectively improve the reliability and accuracy of ranging.

[0089] Optionally, in order to effectively deploy the target detection model and depth estimation model on the vehicle, the present invention also uses TensorRT technology. TensorRT is a high-performance deep learning inference engine designed for production deployment. Using TensorRT can optimize deep learning models, increase processing speed, and reduce latency, which is critical for real-time target detection and depth estimation. By combining the model with TensorRT, the model can be deployed on the vehicle-side edge computing unit, thereby achieving efficient target detection and depth estimation functions.

[0090] Compared with the prior art, the present invention has the following technical effects:

[0091] 1. Improved accuracy: Through perspective transformation and depth estimation model, the present invention can estimate the distance of an object more accurately. This method combines the advantages of two distance measurement methods and effectively improves the accuracy of distance measurement under various environmental conditions.

[0092] 2. Enhanced robustness: The present invention combines two different ranging methods, perspective transformation and depth estimation, so that it can still maintain a high ranging accuracy when facing complex scenes (such as occlusion and lighting changes), thereby improving the overall robustness of the system.

[0093] 3. Balance between real-time performance and accuracy: The present invention can ensure high-precision detection while maintaining real-time requirements, and is suitable for scenarios with high real-time requirements such as intelligent driving systems.

[0094] 4. Wide scope of application: The present invention is not only suitable for distance measurement in intelligent connected vehicles, but can also be widely used in other scenarios requiring monocular vision ranging, such as drones and robot navigation, and has broad application prospects.

[0095] In summary, the present invention solves the distance measurement problem of monocular vision systems in complex scenes by integrating multiple distance measurement methods, and provides technical support for intelligent driving and other visual application fields.

[0096] Example 2

[0097] like Figure 4 As shown, an embodiment of the present application provides an electronic device, including:

[0098] at least one processor; and

[0099] a memory communicatively connected to at least one of the processors; wherein,

[0100] The memory stores instructions executable by at least one of the processors, and the instructions are executed by at least one of the processors to enable at least one of the processors to perform the above method. At least one processor in the electronic device can perform the above method, and thus has at least the same advantages as the above method.

[0101] Optionally, the electronic device also includes interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on a memory to display graphical information of a GUI (Graphical User Interface) on an external input / output device (such as a display device coupled to an interface). In other embodiments, if necessary, multiple processors can be used together with multiple memories, and / or multiple buses can be used together with multiple memories. Similarly, multiple electronic devices can be connected (for example, as a server array, a group of blade servers, or a multi-processor system), and each device provides some necessary operations. Figure 4 A processor 301 is taken as an example.

[0102] The memory 302 is a computer-readable storage medium that can be used to store software programs, computer executable programs and modules, such as the program instructions / modules corresponding to the distance detection method based on a monocular camera on the intelligent networked car in the embodiment of the present application. The processor 301 executes various functional applications and data processing of the electronic device by running the software programs, instructions and modules stored in the memory 302, that is, realizing the above-mentioned distance detection method based on a monocular camera on the intelligent networked car.

[0103] The memory 302 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 302 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some examples, the memory 302 may further include a memory remotely arranged relative to the processor 301, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0104] The electronic device may further include: an input device 303 and an output device 304. The processor 301, the memory 302, the input device 303 and the output device 304 may be connected via a bus or other means, and the figure takes the connection via a bus as an example.

[0105] The input device 303 can receive input digital or character information, and the output device 304 can include a display device, an auxiliary lighting device (e.g., LED), and a tactile feedback device (e.g., a vibration motor), etc. The display device can include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display, and a plasma display. In some embodiments, the display device can be a touch screen.

[0106] Example 3

[0107] This embodiment provides a computer-readable storage medium, on which computer instructions are stored, and the computer instructions are used to enable the computer to execute the above method. The computer instructions on the computer-readable storage medium are used to enable the computer to execute the above method, and thus have at least the same advantages as the above method.

[0108] The computer-readable storage medium in this application may adopt any combination of one or more computer-readable media. The medium may be a computer-readable signal medium or a computer-readable storage medium. The medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples of the medium (a non-exhaustive list) include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, the medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device, or device.

[0109] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0110] The program code contained on the computer-readable medium can be transmitted using any appropriate medium, including but not limited to wireless, wire, optical cable, RF (Radio Frequency), etc., or any suitable combination of the above.

[0111] Computer program code for performing the operation of the present application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect through the Internet).

[0112] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A distance detection method based on a monocular camera on an intelligent connected car, characterized in that: include: The monocular camera on the vehicle is used to shoot the outside of the vehicle to obtain an environmental image; Performing target detection on the environment image to obtain an image of the detected target; Performing a perspective transformation on the image of the detection target to obtain a first depth image; Inputting the environment image into a depth estimation model to obtain a second depth image; Determining the scene type of the environment image according to the scene classification model; the scene types include long-distance scenes, complex scenes and close-range scenes; Determining a weight of the first depth image and a weight of an image of a detection target in the second depth image according to the scene type; In a long-distance scene, the weight of the first depth image is greater than the weight of the second depth image; In complex scenes, the weight of the first depth image is less than that of the second depth image. In close-range scenes, the weight of the first depth image and the weight of the second depth image float between 40% and 60%. Performing weighted summation on depth values ​​of the same position point in the image of the detection target in the first depth image and the second depth image to obtain a depth value of each position point of the detection target; The depth of the detection target is the distance between the detection target and the monocular camera; The depth estimation model includes: a convolutional encoder module, a sparse depth map generation module and a Transformer encoder module connected to the output of the convolutional encoder module and connected in parallel, and a multimodal fusion decoder module connected to the outputs of the sparse depth map generation module and the Transformer encoder module; The convolutional encoder module adopts a deep residual network as a backbone network, and a channel attention mechanism layer is connected after each convolutional layer in the backbone network; the convolutional encoder module outputs a first feature map; The sparse depth map generation module includes a shallow convolution layer; the first feature map is input into the shallow convolution layer to generate a sparse depth map; The Transformer encoder module includes a mapping layer and a multi-layer Transformer encoder; the first feature map is input into the mapping layer and converted into a feature vector of a fixed length, and then connected to the multi-layer Transformer encoder, and finally the encoded global dense depth feature map is output; The multimodal fusion decoder module includes a cross-attention mechanism layer, multiple upsampling layers, a convolution layer and an activation layer; the sparse depth map and the global dense depth feature map are input into the multimodal fusion decoder module to output a second depth image; the multiple upsampling layers gradually upsample the fusion features output from the cross-attention mechanism layer; the deep residual network in the convolution encoder module includes multiple convolution layers to gradually downsample the input image; the feature map output by any layer in the encoder module is spliced ​​with the feature map of the same scale input by an upsampling layer, and then input into the upsampling layer; The output of each upsampling layer is cross-layer connected with the convolution encoder module according to the size of the output features.

2. The method according to claim 1, characterized in that Performing a perspective transformation on the image of the detection target to obtain a first depth image includes: By calibrating known environmental points, the perspective transformation matrix from the image coordinate system of the monocular camera to the world coordinate system is determined; The coordinates of each point in the image of the detection target are input into the perspective transformation matrix to obtain the three-dimensional coordinates of each point in the environment.

3. The method according to claim 1, characterized in that Before weighted summing of the depth values ​​of the same position points in the images of the detection targets in the first depth image and the second depth image is performed, the method further includes: Obtaining a calibration quality of the first depth image by a reprojection algorithm; Obtaining reliability of an image of the detection target in the second depth image through confidence and uncertainty of the depth estimation model; A weight of the first depth image and a weight of an image of the detection target in the second depth image are determined according to the calibration quality and reliability.

4. An electronic device, characterized in that: include: at least one processor, and a memory communicatively coupled to at least one of the processors; The memory stores instructions that can be executed by at least one of the processors, and the instructions are executed by at least one of the processors so that at least one of the processors can execute the distance detection method based on a monocular camera on an intelligent connected vehicle according to any one of claims 1-3.

5. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, which can be executed by one or more processors to implement the distance detection method based on a monocular camera on an intelligent connected vehicle as described in any one of claims 1-3.

Images