Obstacle detection method, device, computing equipment and system

Abstract

The invention provides an obstacle detection method and device, computing equipment and a system, and relates to the field of image processing. The method comprises the steps that after an image is acquired, a depth map of the image is constructed based on structural constraint information, after the depth map is constructed, the depth map is processed to obtain a region identification map comprising a plurality of regions, and any one of the regions is a driving region or a non-driving region. Wherein the structure constraint information comprises the semantic type of each pixel point in the image, and the non-driving area is regarded as an obstacle. Therefore, the construction of the depth map is guided according to the semantic types of the pixel points, so that the depth distribution of the pixel points in the depth map better conforms to the depth distribution rule corresponding to the semantic types to which the pixel points belong, the accuracy of the depth map is improved, driving areas such as the ground and non-driving areas such as obstacles can be accurately identified, and the accuracy of the depth map is improved. And thus, the obstacle recognition accuracy is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to an obstacle detection method, device, computing equipment and system. Background Art

[0002] After identifying the road surface and obstacles, the intelligent driving system can control vehicles, robots and other equipment to achieve automatic driving (or assisted driving) and automatic parking (or assisted parking). Existing intelligent driving systems are usually equipped with sensors such as visual cameras, ultrasonic radars and / or lidars to identify the road surface and obstacles based on the data obtained by the sensors.

[0003] Universal obstacles mainly refer to target types that are difficult to uniformly define in autonomous driving scenarios, such as stones, fallen trees, waste tires, and all other abnormal protrusions on the ground. However, due to blind spots in sensors or insufficient generalization of recognition algorithms, the obstacle detection methods of existing intelligent driving systems cannot accurately identify universal obstacles. Therefore, the obstacle recognition accuracy of existing intelligent driving systems is poor, resulting in vehicles, robots, and other equipment being unable to detect obstacles during driving and causing collisions. Summary of the invention

[0004] The embodiments of the present application provide an obstacle detection method, apparatus, computing device and system, which can solve the problem of poor obstacle recognition accuracy of intelligent driving systems.

[0005] In a first aspect, an obstacle detection method is provided. The obstacle detection method comprises: acquiring an image, and then constructing a first depth map of the image based on a structural constraint, wherein the structural constraint information includes the semantic type of each pixel in the image; after completing the construction of the first depth map, the first depth map is processed to obtain a region identification map. The region identification map comprises a plurality of regions, wherein any region among the plurality of regions is a driving region or a non-driving region, wherein the non-driving region is regarded as an obstacle.

[0006] The semantic type includes at least one of ground information, wall information and sky information. The driving area refers to the area on the ground where the vehicle can drive safely, and the non-driving area refers to the area on the wall, obstacles and other areas where the vehicle cannot drive safely.

[0007] Based on the above obstacle detection method, structural constraint information is introduced when constructing a depth map of an image. In the process of constructing the depth map, the depth construction of pixels of each semantic type is constrained according to the semantic types of the pixels, such as ground information, wall information or sky information. In this way, using the semantic type of the pixel to constrain the construction of the depth map can make the depth distribution of the pixel in the depth map more consistent with the depth distribution law corresponding to the semantic type to which it belongs, thereby improving the accuracy of the depth map, so that the driving area such as the ground and the non-driving area such as obstacles containing general obstacles can be accurately identified in the area identification map, thereby improving the recognition accuracy of obstacles such as general obstacles, thereby ensuring the safety of automatic driving (or assisted driving) and automatic parking (or assisted parking).

[0008] The present application does not limit the execution subject of the obstacle detection method, and the obstacle detection method can be executed by any computing device that can realize image processing functions such as image recognition. For example, the execution subject of the obstacle detection method can be a computing device such as a processor (central processing unit, CPU) or a vehicle computer, computer, robot, etc. containing a processor. The following is an example of the execution subject of the obstacle detection method being a computing device.

[0009] As a possible implementation, the computing device inputs the image into a dense depth estimation model to obtain a first depth map, where the dense depth estimation model is trained based on structural constraint information. For example, the supervisory signal of the dense depth estimation model during training includes a structural constraint supervisory signal. The first depth map includes a depth value of each pixel in the image, that is, the distance between each pixel and a camera used to capture the image.

[0010] In this way, dense depth estimation is used to perform pixel-by-pixel depth matching on the image, which ensures the integrity of the depth value in the first depth map, and is conducive to improving the segmentation accuracy of the driving area and the non-driving area, thereby improving the obstacle recognition accuracy.

[0011] As a possible implementation, the dense depth estimation model includes a monocular depth estimation network, and the training process of the dense depth estimation model includes: the computing device constructs at least one supervisory signal, and the at least one supervisory signal includes a structural constraint supervisory signal. Then, the computing device inputs the reference frame image into the monocular depth estimation network to obtain a second depth map output by the monocular depth estimation network. The computing device then calculates the loss of each supervisory signal in the at least one supervisory signal based on the second depth map, and determines the total loss function based on the loss of each supervisory signal. Finally, the computing device performs back propagation based on the total loss function to update the parameters of the dense depth estimation model. Among them, the reference frame image and the image used to construct the first depth map are taken by the same camera, and the computing device can directly obtain the image and / or the reference frame image from the camera, or obtain the image and / or the reference frame image from a storage device for storing images taken by the camera.

[0012] In this way, during the training process of the dense depth estimation model, the structural constraint supervision signal is used as the supervision of the model training, and the loss of the structural constraint supervision signal is introduced to guide the update of the model parameters, so that the depth map output by the trained dense depth estimation model meets the conditions of the structural constraint.

[0013] Optionally, the computing device repeats the above training process of the dense depth estimation model until the model converges to obtain a trained dense depth estimation model.

[0014] Optionally, the step of constructing a structural constraint supervision signal includes: the computing device projects the point cloud corresponding to the reference frame image onto the reference frame image to obtain a third depth map; the computing device respectively fits the depth values ​​of pixel points belonging to the same semantic type in the third depth map to obtain a structural constraint supervision signal, that is, the plane depth of at least one plane contained in the third depth map.

[0015] In this way, the computing device treats the pixels of the same semantic type in the image as the pixels of the same object or the same plane to fit the depth value, thereby improving the fitting accuracy of the depth value and the accuracy of the structural constraint supervision signal. Furthermore, the abnormal (such as missing depth) depth value in the predicted depth value of each plane can be identified by using the plane depth of the plane as the structural constraint supervision signal, thereby improving the above abnormal problem during the model training process and ensuring the accuracy of the dense depth estimation model obtained by training based on the structural constraint supervision signal.

[0016] Optionally, the semantic type of each pixel in the above image is extracted by a semantic segmentation model.

[0017] As a possible implementation manner, the loss of the structure constraint supervision signal may include at least one of a plane normal vector constraint loss, a plane depth constraint loss, and the like.

[0018] Optionally, the step of calculating the normal vector constraint loss of the plane includes: the computing device determines the pixel points of the first plane contained in the second depth map according to the plane depth of at least one plane, and the first plane is one of the at least one plane contained in the second depth map; then the computing device determines the average normal vector of the pixel points of m sampling points in the first plane; the computing device then determines the average normal vector difference between the normal vector of each pixel point contained in the first plane and the average normal vector as the normal vector constraint loss of the plane. Wherein, m is an integer greater than or equal to 3, and each sampling point contains multiple pixel points.

[0019] In this way, planes are divided according to the semantic type of pixels, and the depth values ​​of pixels on each plane predicted by the monocular depth estimation network are calculated using normal vector loss, so that the normal vector constraint loss of each plane can be obtained, thereby improving the accuracy of the normal vector constraint loss of the plane. Furthermore, the subsequent dense depth estimation model is trained based on the normal vector constraint loss of the plane, which is conducive to improving the training effect, thereby improving the accuracy of the dense depth estimation model after training.

[0020] Optionally, the step of calculating the depth constraint loss of the plane includes: the computing device determines the pixel points of the second plane contained in the second depth map according to the plane depth of at least one plane, the second plane is one of the at least one plane contained in the second depth map, and the plane corresponding to the second plane in the third depth map is the third plane; then, the computing device fills the depth values ​​of the pixel points with missing depth values ​​in the third plane based on the plane depth of at least one plane to obtain a fourth depth map; the computing device then calculates the average depth difference between the depth values ​​of the second plane of the second depth map and all the pixel points in the same position in the fourth depth map as the depth constraint loss of the plane.

[0021] In this way, the third depth map fitted according to the semantic type of the pixels is used to complete the depth values ​​of the missing pixels in the point cloud projection result, thereby improving the completeness and accuracy of the structural constraint supervision signal. The subsequent dense depth estimation model is trained based on the depth constraint loss of the plane, which is conducive to improving the training effect, thereby improving the accuracy of the dense depth estimation model after training.

[0022] As a possible implementation method, the computing device converts the depth map from the pixel coordinate system to the vehicle coordinate system to obtain a fifth depth map, then adds multiple grids to the fifth depth map, and uses a ground segmentation algorithm to determine whether the area contained in each grid in the fifth depth map is a driving area or a non-driving area, so as to complete obstacle detection. Among them, the non-driving area can be regarded as an obstacle.

[0023] Optionally, the computing device corrects the depth value of each pixel in the fifth depth map according to the semantic type of each pixel in the fifth depth map to obtain a corrected depth map, and then uses a ground segmentation algorithm to determine whether the area contained in each grid in the corrected depth map is a driving area or a non-driving area.

[0024] In this way, the semantic type of pixel points is introduced to guide ground segmentation, and the semantic information is used to further correct geometric information such as depth values, thereby improving the accuracy of obstacle detection, reducing collision risks, and further improving driving safety.

[0025] As a possible implementation, the computing device may also notify the user (eg, driver, etc.) of the obstacle information after completing the obstacle detection. The computing device may notify the user by voice broadcast, image display, etc.

[0026] Optionally, the user triggers a notification instruction, and the computing device presents obstacle information of the obstacle to the user in response to the notification instruction.

[0027] Optionally, the obstacle information includes distance information and category information. The distance information is determined by the depth value of the obstacle. The category information includes a first category and a second category, the first category is used to indicate obstacles that are expected to collide with the driving path and the driving speed, and the second category is used to indicate obstacles that are not expected to collide with the driving path and the driving speed.

[0028] In this way, obstacle information, such as obstacle category information and distance information, is notified to the user according to user needs, so that the user can promptly know the distance of the obstacle and which obstacles pose a collision risk during driving, which helps the user determine how to avoid the obstacle and improves driving safety.

[0029] In a second aspect, an obstacle detection device is provided, which includes various modules for executing the obstacle detection method in the first aspect or any possible implementation of the first aspect. The obstacle detection device in the present application may include a transceiver module, a depth estimation module and a segmentation module. The transceiver module is used to acquire an image. The depth estimation module is used to construct a first depth map of the image based on structural constraint information, and the structural constraint information includes the semantic type of each pixel in the image. The segmentation module is used to process the first depth map to obtain a region identification map, and the region identification map includes multiple regions, any of the multiple regions is a driving area or a non-driving area, and the non-driving area is an obstacle.

[0030] In a third aspect, a computing device is provided, comprising a memory and a processor, wherein the memory is used to store a set of computer instructions, and when the processor executes the set of computer instructions, it is used to execute the operating steps of the obstacle detection method in the first aspect or any possible implementation of the first aspect.

[0031] In a fourth aspect, an obstacle detection system is provided. The obstacle detection system includes at least one camera and the above-mentioned computing device, wherein the at least one camera is used to acquire an image, and the computing device is used to perform the operation steps of the obstacle detection method in the first aspect or any possible implementation of the first aspect according to the image acquired by the camera.

[0032] In addition, the technical effects of the obstacle detection device described in the second aspect, the technical effects of the computing device described in the third aspect, and the technical effects of the obstacle detection system described in the fourth aspect can refer to the technical effects of the obstacle detection method described in the first aspect, and will not be repeated here.

[0033] In a fifth aspect, a readable storage medium is provided. The readable storage medium includes: a computer program or instructions; when the computer program or instructions are executed on a computer, the computer executes the obstacle detection method described in any possible implementation manner of the first aspect.

[0034] In a sixth aspect, a computer program product is provided, which includes a computer program or instructions, and when the computer program or instructions are executed on a computer, the computer executes the obstacle detection method described in any possible implementation manner of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A schematic diagram of the structure of a neural network;

[0036] Figure 2 A schematic diagram of the architecture of an obstacle detection system provided in this application;

[0037] Figure 3a A flowchart of an obstacle detection method provided in this application;

[0038] Figure 3b A schematic diagram of an obstacle display method provided for this application;

[0039] Figure 4a A schematic diagram of the structure of a dense depth estimation model provided in this application;

[0040] Figure 4b A schematic diagram of the structure of a monocular depth estimation network provided in this application;

[0041] Figure 5a A flowchart of a training process of a dense depth estimation model provided in this application;

[0042] Figure 5b A schematic diagram of a structural constraint supervision signal provided for this application;

[0043] Figure 6 A schematic diagram of a ground segmentation process provided for this application;

[0044] Figure 7 A schematic diagram of a multi-frame fusion process provided by this application;

[0045] Figure 8 A schematic diagram of the structure of an obstacle detection device provided in this application;

[0046] Fig. 9 A schematic diagram of the structure of a computing device provided in this application. DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings in the embodiments of the present application. In the present application, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b and c can be single or multiple.

[0048] It should be noted that, in this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0049] The obstacle detection method provided in the embodiments of the present application can be applied to scenarios in which vehicles, robots, etc. perform automatic driving (or assisted driving) and automatic parking (or assisted parking). The following is a brief introduction to the technologies that may be involved in the present application.

[0050] (1) Neural Network

[0051] A neural network can be composed of neurons, and a neuron can be a s The output of this operation unit satisfies the following formula (1).

[0052]

[0053] Among them, h W,b is the output of the operation unit, x is the input of the operation unit, s = 1, 2, ... n, n is a natural number greater than 1, W s For x s , b is the bias of the operation unit. f is the activation function of the neuron, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neuron into the output signal. The output signal of the activation function can be used as the input of the next layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neurons mentioned above, that is, the output of one neuron can be the input of another neuron. The input of each neuron can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neurons. The weight represents the strength of the connection between different neurons. The weight determines the influence of the input on the output. A weight close to 0 means that changing the input does not change the output. A negative weight means that increasing the input reduces the output.

[0054] like Figure 1 , which is a schematic diagram of a neural network structure. The neural network 100 includes N processing layers, where N is an integer greater than or equal to 3. The first layer of the neural network 100 is the input layer 110, which is responsible for receiving the input signal x i The last layer of the neural network 100 is the output layer 130, which is responsible for outputting the processing result h of the neural network. i The other layers except the first layer and the last layer are intermediate layers 140, which together constitute hidden layers 120. Each intermediate layer 140 in the hidden layer 120 can both receive input signals and output signals. The hidden layer 120 is responsible for the processing of input signals. Each layer represents a logical level of signal processing. Through multiple layers, data signals can be processed by multiple levels of logic.

[0055] In some feasible embodiments, the input signal of the neural network can be a signal in various forms such as a video signal and an image signal. The image signal can be image data input by a monocular camera, a binocular camera, a laser radar, an ultrasonic radar, etc. The input signal of the neural network also includes various other computer-processable engineering signals, which are not listed here one by one. If the neural network is used to perform deep learning on the image signal, the quality of the image processed by the neural network can be improved.

[0056] (2) Loss Function

[0057] In the process of training a neural network, because we hope that the output of the neural network is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the neural network according to the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the neural network). For example, if the predicted value of the network is too high, adjust the weight vector to make it predict lower, and keep adjusting until the neural network can predict the target value we really want or a value very close to the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the neural network becomes a process of minimizing this loss as much as possible.

[0058] The original meaning of gradient is a vector (vector), which means that the directional derivative of a function at this point reaches the maximum value along this direction, that is, the function changes fastest and at the largest rate along this direction (the direction of this gradient) at this point. When looking for the optimal parameters of each network layer during the training process of the neural network, it is necessary to determine the parameters that minimize the value of the loss function. In order to find the place where the value of the loss function is as small as possible, it is necessary to calculate the gradient of the loss function relative to the parameters, that is, when the gradient vector is closer to 0, it means that the loss function reaches a minimum point and the model accuracy reaches a maximum point.

[0059] (3) Monocular Depth Estimation

[0060] Depth estimation refers to the process of estimating the depth of a scene in an image captured by a camera using multiple two-dimensional images from one or more perspectives, that is, the vertical distance from each pixel in the scene to the camera imaging plane. Depth estimation is an important step in scene reconstruction, such as three-dimensional reconstruction, in the field of computer vision.

[0061] Monocular depth estimation is to predict or estimate the depth map using a two-dimensional image from a monocular camera. A monocular camera is a camera consisting of a single camera, including a monocular pinhole camera, a monocular fisheye camera, etc. A depth map is an image or image channel that contains information about the distance from the surface of a scene object to the viewpoint.

[0062] Nowadays, monocular depth estimation can be performed based on deep learning (such as neural networks). The basis of monocular depth estimation based on deep learning is that the relationship between pixel values ​​can reflect the depth relationship. Its estimation method can be summarized as fitting a function to map a two-dimensional image into a depth map.

[0063] Traditional monocular depth estimation usually includes the following two methods: (a) detecting the location of obstacles by fusing radar information such as ultrasonic radar and lidar through raster maps; (b) identifying the location of obstacles in images obtained by sensors such as cameras through visual recognition; (c) fusing radar information and visual recognition to determine the location of obstacles in the image.

[0064] The above method (a) detects obstacles by emitting and receiving ultrasonic, laser and other information. The emission power and effective field of view (FOV) of ultrasonic and laser are limited by the hardware conditions or installation position of the sensor itself. In addition, ultrasonic radar, lidar and other sensors have no height measurement capability, resulting in low obstacles such as curbs and ground locks being unable to be identified due to insufficient echo information.

[0065] The above method (b) is to identify obstacles in the image through visual recognition algorithms, but the obstacle types and accuracy that can be identified by visual recognition methods such as neural network models are related to the obstacle categories and data volume of the training set during the training process. Usually, the training of neural network models can only ensure that the neural network model can identify some common types of obstacles (such as car crash barriers used to limit the parking area of ​​parking lots, etc.), and cannot accurately identify common obstacles such as stone piers, columns, ground locks, fire hydrants, etc.

[0066] The above method (c) combines the above methods (a) and (b) to identify obstacles, but still has the shortcomings of the above methods (a) and (b), and cannot accurately identify general obstacles.

[0067] It can be seen that the traditional monocular depth estimation method is still unable to accurately identify general obstacles, and cannot effectively solve the problem of low obstacle recognition accuracy in autonomous driving (or assisted driving) and automatic parking (or assisted parking) scenarios. It will cause vehicles, robots, etc. to have dangerous accidents such as scratches and collisions during autonomous driving (or assisted driving) and automatic parking (or assisted parking) such as automatic parking and low-speed parking, and cannot guarantee driving safety.

[0068] The present application provides an obstacle detection method, specifically, an obstacle detection method of "introducing structural constraint information based on semantic information for depth estimation to identify obstacles". A computing device acquires an image, constructs a first depth map of the image based on the structural constraint information, and then processes the first depth map to obtain a region identification map, wherein any of the multiple regions in the region identification map is a driving area and a non-driving area, thereby treating the non-driving area as an obstacle. The structural constraint information includes the semantic type of each pixel in the image, and the semantic type includes at least one of ground information, wall information, and sky information.

[0069] Based on the above obstacle detection method, structural constraint information is introduced, that is, the depth value of each pixel is constructed by combining the semantic type of the pixel points of the image. In the process of constructing the depth map, the depth construction of the pixel points of each semantic type is constrained according to the semantic type of the pixel points belonging to ground information, wall information or sky information. In this way, the applicability of the depth estimation to the pixel points belonging to each semantic category such as ground information, wall information or sky information is guaranteed, and the depth estimation accuracy of the depth map is improved. In this way, the accuracy of the subsequent segmentation of the driving area and the non-driving area based on the depth map is improved, so that the computing device can more accurately segment the non-driving areas such as stone piers, pillars, ground locks, fire hydrants, etc., improve the recognition accuracy of general obstacles, improve the overall accuracy of obstacle recognition, and ensure the safety of automatic driving (or assisted driving) and automatic parking (or assisted parking).

[0070] The implementation of the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0071] Figure 2 This is a schematic diagram of the architecture of an obstacle detection system provided in this application. Figure 2 As shown, the obstacle detection system 200 includes an execution device 210 , a training device 220 , a database 230 , a terminal device 240 , a data storage system 250 and a data acquisition device 260 .

[0072] The execution device 210 may be a terminal, such as a mobile phone terminal, a tablet computer, a laptop computer, a virtual reality (VR) device, an augmented reality (AR) device, a mixed reality (MR) device, an extended reality (ER) device, a camera or a vehicle-mounted computer, etc. It may also be an edge device (for example, a box carrying a chip with processing capabilities), etc. Exemplarily, the execution device 210 in the present application may be a vehicle-mounted computer with automatic driving (or assisted driving) and automatic parking (or assisted parking) functions.

[0073] The training device 220 may be a terminal, or other computing device that supports integer computing, such as a server or a cloud device.

[0074] As a possible embodiment, the execution device 210 and the training device 220 are deployed on different physical devices (such as a server or a server in a cluster), or the execution device 210 and the training device 220 are different physical devices. Exemplarily, the execution device 210 and the training device 220 are processors deployed on different physical devices. For example, the execution device 210 can be a graphics processing unit (GPU), a central processing unit, other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. The training device 220 can be a graphics processing unit (GPU), a neural network processing unit (NPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the present application.

[0075] In another possible embodiment, the execution device 210 and the training device 220 are deployed on the same physical device, or the execution device 210 and the training device 220 are the same physical device.

[0076] The data acquisition device 260 is used to collect training data and store the training data in the database 230. The data acquisition device 260 and the execution device 210 and the training device 220 may be the same or different devices. In the embodiment of the present application, the data acquisition device 260 may be a sensor such as a camera, a laser radar, an ultrasonic radar, etc. Among them, the camera may be a vehicle-mounted surround view camera, a pinhole camera, a fisheye camera, etc. The training data may be image data acquired by the camera, and point clouds acquired by the laser radar, etc.

[0077] The training device 220 is used to train the neural network using the training data until the loss function in the neural network converges, and the neural network training is completed when the loss function value is less than a specific threshold, so that the neural network reaches a certain accuracy. For example, the training device 220 uses the training data as input, the supervisory signal such as the structural constraint supervisory signal as supervision, and the obstacles in the image as output to train the neural network. During the training process, back propagation is performed according to the loss function of the supervisory signal, and the parameters of the neural network are updated with the gradient until the loss function value is less than a specific threshold. Alternatively, all the training data in the database 230 are used for training, and the neural network training is completed, so that the trained neural network has the target functions such as image recognition and image classification. Furthermore, the training device 220 configures the trained neural network 201 to the execution device 210. The execution device 210 is used to realize the function of identifying obstacles in the image according to the trained neural network 201.

[0078] Optionally, the neural network 201 is used to construct a depth map of the image based on structural constraints. The structural constraint information includes the semantic type of each pixel in the image. Therefore, the neural network 201 can be monodepth2 suitable for monocular dense depth estimation, and the encoder of monodepth2 can have a semantic segmentation function to extract the semantic type of each pixel in the image.

[0079] The semantic type includes at least one of ground information, wall information and sky information.

[0080] In some embodiments, the execution device 210 and the training device 220 are the same computing device, and the computing device can configure the trained neural network 201 to itself and use the trained neural network 201 to achieve target functions such as dense depth estimation.

[0081] In other embodiments, the training device 220 may configure the trained neural network 201 to multiple execution devices 210. Each execution device 210 uses the trained neural network 201 to implement target functions such as dense depth estimation.

[0082] In combination with the obstacle detection system 200, the obstacle detection method provided in this embodiment can be applied to automatic driving (or assisted driving) and automatic parking (or assisted parking) scenarios of vehicles, robots and other equipment, such as automatic parking assist (APA), remote parking assist (RPA), home-zone parking assist (HPA), autonomous valet parking (AVP), etc.

[0083] It should be noted that, in actual applications, the training data maintained in the database 230 may not all come from the data acquisition device 260, but may also be received from other devices. In addition, the training device 220 may not train the neural network entirely based on the training data maintained by the database 230, but may also obtain training data from the cloud or other places to train the neural network. The above description should not be used as a limitation on the embodiments of the present application.

[0084] Furthermore, according to the functions performed by the execution device 210, the execution device 210 can be further divided into: Figure 2 In the illustrated architecture, as shown in the figure, the execution device 210 is configured with a computing module 211 , an I / O interface 212 and a pre-processing module 213 .

[0085] Taking the execution device 210 as an on-board computer as an example, the computing module 211 can be a GPU, CPU, other general-purpose processors, DSP, ASIC, FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. on the on-board computer.

[0086] For example, the computing module 211 is used to run the neural network 201 to implement functions such as image recognition and image processing of the neural network 201 to obtain a first depth map of the image.

[0087] For another example, the calculation module 211 is also used to process the first depth map output by the neural network 201 and convert the first depth map into a region identification map. The region identification map includes multiple regions, any region is a driving region and a non-driving region, and the non-driving region is regarded as an obstacle, thereby completing obstacle recognition.

[0088] The I / O interface 212 is used to exchange data with external devices. The user can input data to the I / O interface 212 through the terminal device 240, such as an instruction for instructing the execution device 210 to start the obstacle detection function to execute the obstacle detection method, or a notification instruction for instructing the obstacle information of the obstacle to be presented. In addition, the input data can also come from the database 230. Taking the execution device 210 as an on-board computer as an example, the I / O interface 212 can comply with the common vehicle bus specifications such as the local interconnect network (LIN) or the controller area network (CAN), and the terminal device 240 can be a display device of the on-board computer, etc.

[0089] The preprocessing module 213 is used to preprocess the input data received by the I / O interface 212. In the embodiment of the present application, the preprocessing module 213 can be used to generate training data, such as a training set, a validation set, and a test set, based on the input data received from the I / O interface 212. Optionally, the preprocessing module 213 can also perform a preprocessing operation such as denoising on the input data, such as image data, to eliminate irrelevant information in the image and restore useful real information.

[0090] When the execution device 210 preprocesses the input data, or when the computing module 211 of the execution device 210 performs calculations and other related processing, the execution device 210 can call the data, code, etc. in the data storage system 250 for corresponding processing, and can also store the data and instructions obtained from the corresponding processing in the data storage system 250.

[0091] Finally, the I / O interface 212 returns the processing result to the terminal device 240, thereby providing it to the user so that the user can view the processing result. In the embodiment of the present application, the processing result can be an image that identifies the driving area and the non-driving area, or an image that identifies obstacles.

[0092] The terminal device 240 can be used as a data collection terminal to collect the input data of the input I / O interface 212 and the processing results of the output I / O interface 212 as new sample data, and store them in the database 230. Of course, the sample data can also be collected without passing through the terminal device 240, but the I / O interface 212 can store the input data of the input I / O interface 212 and the processing results of the output I / O interface 212 as new sample data in the database 230.

[0093] In a possible embodiment of the present application, if the obstacle detection system 200 includes a notification device ( Figure 2(not shown), the I / O interface 212 can send the processing result such as obstacle information to the notification device, thereby providing it to the user. The notification device can be a display device, an audio playback device, etc.

[0094] Figure 2 This is only a schematic diagram of a system architecture provided by an embodiment of the present application. Figure 2 The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. Figure 2 In the embodiment, the data storage system 250 is an external memory relative to the execution device 210. In other cases, the data storage system 250 can also be placed in the execution device 210.

[0095] Next, the obstacle detection method provided by this embodiment is described in detail with reference to the accompanying drawings.

[0096] In the scenarios of autonomous driving (or assisted driving) and automatic parking (or assisted parking), the above Figure 2 The execution device 210 in the obstacle detection system 200 in the embodiment may be a vehicle-mounted computer. In the process of automatic driving (or assisted driving) and automatic parking (or assisted parking), the obstacle detection method is usually executed by the execution device 210, i.e., the vehicle-mounted computer. Therefore, the obstacle detection method is described in detail below by taking the execution device 210 as an example. Figure 3a As shown, the obstacle detection method may include the following steps 310 to 330.

[0097] Step 310: The execution device 210 acquires an image.

[0098] As a possible example, the execution device 210 obtains a two-dimensional image captured by the data acquisition device 260, i.e., a camera. The two-dimensional image is an image that does not contain depth information, such as a red-green-blue (RGB) color mode image, a YUV color mode image, etc.

[0099] Taking the data acquisition device 260 as a monocular camera installed on a vehicle as an example, the vehicle-mounted computer acquires images from the database 230 or the monocular camera. For example, after acquiring an image, the monocular camera sends the image to the vehicle-mounted computer, and the vehicle-mounted computer acquires the image sent by the monocular camera. For another example, after acquiring an image, the monocular camera stores the image in the database 230, and the vehicle-mounted computer acquires the image from the database 230.

[0100] Step 320: The execution device 210 constructs a first depth map of the image based on the structure constraint information.

[0101] As a possible example, the execution device 210 uses the structural constraint information as a constraint condition for calculating the depth of each pixel of the image, and calculates the depth value of each pixel in the image to construct a first depth map of the image. The depth value of each pixel is an absolute depth, that is, the distance between the camera that captures the image and each pixel in the image.

[0102] The structural constraint information includes the semantic type of each pixel in the image, and the semantic type includes at least one of planar structural constraints such as ground information, wall information, and sky information. In other possible embodiments, the semantic type may also include at least one of non-planar structural constraints such as spherical information and curved surface information.

[0103] As a possible implementation manner, the execution device 210 uses the structural constraint information as a constraint condition for calculating the depth of each pixel of the image, performs dense depth estimation on the image, and obtains a first depth map.

[0104] Optionally, the execution device 210 may also use a dense depth estimation model based on deep learning to perform dense depth estimation on the image.

[0105] Taking the dense depth estimation model as an example, the execution device 210 inputs the image into the dense depth estimation model to obtain a first depth map output by the dense depth estimation model. The supervision signal of the dense depth estimation model in the self-supervised training process includes a structural constraint supervision signal, so the dense depth estimation model has the function of using the structural constraint information as a constraint condition for calculating the depth of each pixel of the image to perform dense depth estimation on the image.

[0106] The above-mentioned dense depth estimation model can be obtained by training the depth estimation model using the structural constraint supervision signal as the supervision signal in the training process. For the specific training process of the dense depth estimation model, please refer to Figure 5a Steps 510 to 550 are shown and will not be described in detail here.

[0107] This application does not limit the specific type of the depth estimation model. For example, the depth estimation model can be a model such as monodepth2, U-Net, or Zoedepth.

[0108] Optionally, the semantic type of each pixel point may be extracted by an encoder of a depth estimation model, or may be extracted by a separate semantic segmentation network within or outside the depth estimation model.

[0109] Step 330: The execution device 210 processes the first depth map to obtain a region identification map.

[0110] As a possible example, the execution device 210 processes the first depth map using a ground segmentation algorithm to obtain a region identification map. The region identification map includes multiple regions, each of which is determined to be a driving region or a non-driving region. For example, the ground in the first depth map is a driving region, and the non-ground in the first depth map is a non-driving region, i.e., an obstacle.

[0111] Among them, ground segmentation algorithms include plane grid method, point cloud normal vector method, model fitting method, surface element grid method, etc.

[0112] As a first possible implementation manner, the execution device 210 converts the first depth map from a pixel coordinate system to a vehicle coordinate system, and then uses a ground segmentation algorithm to process the depth map in the vehicle coordinate system into a region identification map.

[0113] As a second possible implementation, the execution device 210 may convert the first depth map from the pixel coordinate system to the vehicle coordinate system, and then correct the depth value of each pixel according to the semantic type of each pixel, and then use the ground segmentation algorithm to process the corrected depth map into a region identification map. In this way, based on the different depth characteristics of pixels of different semantic types, the semantic types of the pixels contained in each surface in the depth map are used to assist the execution device 210 in segmenting the ground, wall and other planes, thereby improving the accuracy of ground segmentation.

[0114] As a third possible implementation, considering that the ground segmentation result is obtained based on a single frame image, there may be unstable problems such as missing pixels, so after the execution device 210 obtains the ground segmentation result based on the single frame image, it can also fuse the ground segmentation result of the single frame image and the ground segmentation result of the historical frame image to obtain the fused ground segmentation result. In this way, the accuracy of ground segmentation is further improved.

[0115] Please refer to the following for the specific steps of the conversion of the pixel coordinate system of the first depth map from the vehicle coordinate system, ground segmentation and multi-frame fusion. Figure 7 Steps 710 to 750 are not described in detail here.

[0116] In a possible embodiment of the present application, after completing the segmentation of the driving area and the non-driving area, the execution device 210 may also notify the user of obstacle information.

[0117] As a possible implementation, when the execution device 210 receives a notification instruction triggered by a user (such as a driver), the execution device 210 responds to the notification instruction and notifies the user of the obstacle information. The obstacle information may include category information and distance information of the obstacle.

[0118] Optionally, the execution device 210 may use a notification device such as a speaker or a display screen to notify the user of the obstacle information. For example, the execution device 210 is a vehicle-mounted computer, and the vehicle-mounted computer plays the obstacle information through the speaker of the terminal device 240, or displays the obstacle information through the display screen of the terminal device 240. The terminal device 240 may be a display device, a speaker, or other device connected to the vehicle-mounted computer, or may be a mobile phone, a laptop, a vehicle central control screen, or the like with an audio playback function or an audio playback function connected to the vehicle-mounted computer.

[0119] Optionally, the category information of the obstacle includes a first category and a second category, the first category is used to indicate obstacles that are expected to collide with based on the driving path and driving speed, and the second category is used to indicate obstacles that are expected not to collide with based on the driving path and driving speed.

[0120] The step of the execution device 210 determining whether the vehicle will collide with an obstacle may be as follows: the execution device 210 determines the position that the vehicle will reach at multiple times in the future according to the vehicle's driving path and driving speed. If the position that the vehicle reaches at a certain time in the future overlaps with the position of the obstacle, the obstacle that overlaps with the position that the vehicle reaches at a certain time in the future is marked as the first category. If the position that the vehicle reaches at any time in the future does not overlap with the position of the obstacle, the obstacle is marked as the second category.

[0121] For example, the execution device 210 notifies the user of obstacle information through the vehicle central control screen. The vehicle central control screen displays an obstacle information button. Before the user triggers the obstacle information button, the vehicle central control screen displays conventional reversing images, radar sectors, and other information. After the user triggers the obstacle button, Figure 3b As shown, in addition to displaying panoramic images, reversing images, radar sectors and other information in a conventional interface, the vehicle's central control screen can also display the category information and distance information of obstacles in an overlay interface. For example, the category information of obstacle 1 is the first category, i.e., a dangerous obstacle, and the distance information is 1.2 meters. The category information of obstacle 2 is the second category, i.e., a non-dangerous obstacle, and the distance information is 1.1 meters.

[0122] Based on the above steps 310 to 330, structural constraint information is introduced, that is, the depth of the image is estimated in combination with the semantic type of the pixel points of the image to obtain a depth map, so that the depth value of the pixel points of each plane is calculated with reference to the depth characteristics of the semantic type to which each pixel point belongs, ensuring the applicability of the depth estimation to the pixel points belonging to various semantic categories such as ground information, wall information or sky information, and improving the depth estimation accuracy of the depth map. In this way, the accuracy of the subsequent segmentation of driving areas and non-driving areas based on the depth map is improved, so that the computing device can more accurately segment non-driving areas such as stone piers, pillars, ground locks, fire hydrants, etc., improve the recognition accuracy of general obstacles, thereby improving the overall obstacle recognition accuracy, and ensuring the safety of automatic driving (or assisted driving) and automatic parking (or assisted parking).

[0123] Combination of the above Figure 2 and Figure 3a The obstacle detection method of the present application is described in general. Figure 4a , Figure 4b and Figure 5a The training process of the dense depth estimation model is explained in detail.

[0124] Next, we take the dense depth estimation model as the monodepth2 model as an example. First, we combine Figure 4a and Figure 4b The structure of the dense depth estimation model is introduced, and then combined with Figure 5a The training process of the dense depth estimation model is described in detail.

[0125] like Figure 4a As shown, the dense depth estimation model 400 is monodepth2, and monodepth2 includes a monocular depth estimation network 401 and a camera pose estimation network 402. In the training process of monodepth2, the training data of the input model is a continuous multi-frame image of a video, the input of the monocular depth estimation network 401 is a reference frame image, and the input of the camera pose estimation network 402 is a reference frame image and an adjacent frame image of the reference frame image, for example, the reference frame image is image 1, and the adjacent frame image of the reference frame image is image 2. The monocular depth estimation network 401 uses a convolutional neural network to process the reference frame image and outputs a depth map, and the camera pose estimation network 402 calculates the relative pose of the camera, that is, the change in motion pose, based on the reference frame image and the adjacent frame images of the reference frame image. Finally, the training device constructs a reprojected image based on the output of the monocular depth estimation network 401 and the camera pose estimation network 402, constructs a loss function based on the reprojection error of the reprojected image, and back-propagates and updates the parameters of monodepth2 through the loss function to optimize monodepth2.

[0126] In some possible embodiments, the adjacent frame image of the reference frame image may be an upper n frame image or a lower n frame image of the reference frame image, where n is a positive integer.

[0127] like Figure 4b As shown, the monocular depth estimation network 401 includes an encoder 403 and a decoder 404.

[0128] As a possible implementation, the encoder 403 can be a residual network such as ResNet18, ResNet50, etc., including a convolutional layer 405, a batch normalization (BN) layer 406, an activation function 407, a maximum pooling layer 408 and at least one residual layer 409.

[0129] The convolution layer 405 is used to extract features from the input image to obtain a feature map. The present application does not limit the size of the convolution layer 405. For example, the size of the convolution layer 405 can be 4*4, 7*7, 9*9, etc.

[0130] The batch normalization layer 406 is used to normalize the feature map extracted by the convolution layer 405 to prevent the gradient from disappearing or exploding, and to speed up the training.

[0131] The activation function 407 is used to perform nonlinear processing on the feature map. The present application does not limit the type of the activation function 407. For example, the activation function 407 is a linear rectification function (ReLU).

[0132] The maximum pooling layer 408 is used to compress the feature map and simplify the network complexity.

[0133] At least one residual layer 409 between different levels uses a convolution kernel with an increased step size to perform a convolution operation instead of a downsampling process, so that the size of the feature map is reduced by multiples, and the last residual layer 409 outputs the reduced feature map.

[0134] Optionally, the residual layer 409 is composed of two residual blocks, and the activation function used inside each residual block can be an exponential linear unit (ELU) activation function. The present application does not limit the number of residual layers 409, for example, the encoder 402 includes 3, 4, 5, 6, or 8 residual layers 409.

[0135] As a possible implementation manner, the decoder 404 includes a convolution layer 410 , an upsampling layer 411 and at least one up-convolution layer 412 .

[0136] The function of the convolution layer 410 is similar to that of the convolution layer 405, that is, feature extraction, which will not be described in detail here.

[0137] The upsampling layer 411 is used to enlarge the feature map.

[0138] The input of each upper convolution layer 412 in at least one upper convolution layer 412 is the output of the previous network layer and the feature map of the same size output by the residual layer 409 in the encoder 403. After the feature maps of the same size are fused, a depth map is output. In the embodiment of the present application, the number of upper convolution layers 412 is the same as the number of residual layers 409.

[0139] The camera pose estimation network 402 includes an encoder 413 and a decoder 414 .

[0140] As a possible implementation method, the structure of encoder 413 is similar to that of encoder 403, and the structure of decoder 414 is similar to that of decoder 404. The structures of encoder 413 and decoder 414 are not described in detail here, and only the functions of encoder 413 and decoder 414 are described.

[0141] The encoder 413 is used to estimate the pose of the input image and the previous frame image, and output pose features.

[0142] The decoder 414 is used to decode the pose features and regress the motion information of each pixel, so as to calculate the axis-angle matrix and the translation matrix according to the motion information of each pixel, and then use the axis-angle matrix and the translation matrix to predict the camera pose transformation. The camera pose transformation may include a translation matrix and a rotation matrix.

[0143] In combination Figure 4a and Figure 4b After taking monodepth2 as an example, the structure of the dense depth estimation model 400 is described in detail, and then monodepth2 is continued as an example. Figure 5a The process of training the dense depth estimation model 400 by the training device 220 is described in detail.

[0144] like Figure 5a As shown, the training process of the dense depth estimation model 400 includes the following steps 510 to 550.

[0145] Step 510: The training device 220 constructs at least one supervisory signal.

[0146] As a possible example, at least one supervision signal includes a structure-constrained supervision signal, and may also include a deep supervision signal and / or a self-supervision signal.

[0147] As a possible implementation manner, the training device 220 projects the point cloud corresponding to the reference frame image onto the reference frame image to obtain a third depth map, and uses the third depth map as a depth supervision signal.

[0148] Optionally, the training device 220 performs depth projection on the reference frame image and the point cloud corresponding to the reference frame image, and adjusts the resolution of the point cloud according to the resolution of the reference frame image, thereby obtaining a third depth map at the resolution of the reference frame image.

[0149] The point cloud corresponding to the reference frame image may be a single-frame point cloud or a multi-frame point cloud obtained by a laser radar or the like.

[0150] As a possible implementation, the training device 220 fits the depth values ​​of the pixels of the same semantic type in the third depth map, obtains the plane depth of at least one plane included in the third depth map, and uses the plane depth of at least one plane as a structural constraint supervision signal. Figure 5b The structural constraint supervision signal extracted from the parking lot scene is shown. The structural constraint supervision signal is used to indicate which pixels in the depth map form a plane, such as Figure 5b The white pixels in the figure are all pixels whose semantic categories belong to ground information.

[0151] Optionally, the semantic type of the pixel is extracted from the reference frame image by the training device 220 using a semantic segmentation model such as Mask2former.

[0152] As a possible implementation method, the training device 220 inputs the reference frame image and the adjacent frame images of the reference frame image into the camera pose estimation network 402 to obtain the camera pose transformation, and then interpolates the depth of the third depth map corresponding to the reference frame image to the reference frame image at the original resolution through bilinear transformation to obtain a first interpolated depth map. The training device 220 can determine the pixel correspondence between the first interpolated depth map and the adjacent frame image according to the camera pose transformation, and reproject the adjacent frame image. In this way, the adjacent frame image is used as the self-supervisory signal, and the reprojection error is used as the loss of the self-supervisory signal.

[0153] Step 520: The training device 220 inputs the reference frame image into the monocular depth estimation network to obtain a second depth map output by the monocular depth estimation network.

[0154] The principle of the monocular depth estimation network 401 outputting the second depth map according to the reference frame image, please refer to the description of the monocular depth estimation network 401 in the above text, which will not be repeated here.

[0155] Step 530: The training device 220 calculates the loss of each supervisory signal in at least one supervisory signal according to the second depth map.

[0156] As a possible example, at least one supervisory signal includes a loss of a structural constraint supervisory signal, and may also include a loss of a depth supervisory signal and / or a loss of a self-supervisory signal. The loss of the structural constraint supervisory signal may include at least one of a normal vector constraint loss of a plane and a depth constraint loss of a plane.

[0157] As a possible implementation method, since at least one plane depth in the structural constraint supervision signal contains the semantic type of each pixel, at least one plane is divided into multiple planes, and the training device 220 can calculate the normal vector constraint loss of one or more planes in the second depth map based on the plane division in the structural constraint supervision signal.

[0158] Optionally, the training device 220 first determines the pixel points belonging to the first plane in the second depth map according to the plane division in the structural constraint supervision signal, wherein the first plane is one of at least one plane contained in the second depth map, such as the ground, the wall, etc. Then, the training device 220 calculates the average normal vector of the pixel points of m sampling points in the first plane, where m is an integer greater than or equal to 3, and each of the m sampling points contains multiple pixel points, such as 100, 500, 681, 1000, 2005, etc. Finally, the training device 220 calculates the average normal vector difference between the normal vector of each pixel point contained in the first plane and the average normal vector, and uses the average normal vector difference as the normal vector constraint loss of the plane.

[0159] For example, the normal vector of the first plane containing the i-th pixel in the second depth map is The average normal vector of the pixel points of m sampling points in the first plane is n gt , the normal vector constraint loss of the plane can be expressed as follows (2).

[0160]

[0161] As a possible implementation method, since the structural constraint supervision signal is at least one plane depth obtained by fitting each plane based on the semantic type of the pixel points, the abnormal or missing depth values ​​of the pixel points contained in each plane are corrected during fitting. The training device 220 can use at least one plane depth in the structural constraint supervision signal to fill in the missing pixels in the third depth map of the reference frame image point cloud projection, and then calculate the depth constraint loss of the plane.

[0162] Optionally, the training device 220 first determines the pixel points belonging to the second plane in the second depth map according to the plane division in the structural constraint supervision signal, wherein the second plane is one of the at least one plane contained in the second depth map, such as the ground, the wall, etc. Then, the training device 220 uses the depth values ​​of the plane corresponding to the third plane and the pixel points at the corresponding position in the structural constraint supervision signal to fill the depth values ​​of the pixel points with missing depth values ​​in the third plane to obtain a fourth depth map. Among them, the second plane and the third plane correspond to the same plane in the reference frame image. The training device 220 then calculates the average depth difference between the depth values ​​of the second plane of the second depth map and each corresponding pixel point at the same position in the fourth depth map, and uses the average depth difference as the depth constraint loss of the plane.

[0163] For example, the depth value of the i-th pixel in the fourth depth map after filling is The depth value of the i-th pixel in the second depth map is The depth constraint loss of the plane can be expressed as follows:

[0164]

[0165] As a possible implementation method, the training device 220 calculates the predicted average difference of each pixel point between the depth supervision signal in the supervision signal, i.e., the third depth map and the second depth map output by the monocular depth estimation network 401, and uses the predicted average difference as the depth supervision loss.

[0166] For example, x represents a pixel in the image, yi represents the depth value of the i-th pixel in the depth supervision signal, f(xi) represents the depth value of the i-th pixel in the second depth map, and the depth supervision loss can be expressed as follows: formula (4).

[0167]

[0168] As a possible implementation method, the self-supervisory loss is obtained based on the self-supervisory signal in the above step 510. For its specific function structure, please refer to reprojection_loss in monodepth2, which will not be repeated here.

[0169] Step 540: The training device 220 determines a total loss function according to the loss of each supervisory signal in at least one supervisory signal.

[0170] As a possible example, the training device 220 performs weighted summation on the losses of each supervisory signal in at least one supervisory signal to obtain a total loss function.

[0171] Optionally, the weight of each supervisory signal in the at least one supervisory signal may be adjusted according to specific requirements.

[0172] Taking the plane normal vector constraint loss as Lplane1, the plane depth constraint loss as Lplane2, the depth supervision loss as Lgt, and the self-supervision loss as Lphoto as an example, the total loss function can be shown as follows: Formula (5).

[0173] L=γ1*Lgt+γ2*Lphoto+γ3*Lplane1+γ4*Lplane2 (5)

[0174] Among them, γ1, γ2, γ3, and γ4 represent weights.

[0175] Step 550: The training device 220 performs back propagation based on the total loss function to update the parameters of the dense depth estimation model.

[0176] The specific method in which the training device 220 updates parameters based on the total loss function is the same as the back propagation and parameter updating method of a common neural network, and will not be repeated here.

[0177] The training device 220 can repeat the above steps 510 to 550 to train the monocular depth estimation network 401 and the camera pose estimation network 402 until the model converges, thereby obtaining a trained dense depth estimation model 400.

[0178] In this way, the training device 220 uses the structural constraint supervision signal as the supervision of model training during the training process, and introduces the loss update parameter of the structural constraint supervision signal, so that the trained dense depth estimation model 400 can refer to the semantic type of each pixel point, that is, the plane, under the condition of structural constraints to more accurately predict the depth value of the pixel point.

[0179] The training device 220 obtains the trained dense depth estimation model 400 and uses the trained dense depth estimation model 400 as Figure 2 The neural network 201 in the embodiment is deployed to the execution device 210, so that the execution device 210 executes steps 310 to 320, and after obtaining the first depth map of the image acquired by the camera, it is also necessary to execute step 330 to segment the first depth map, thereby dividing the driving area and the non-driving area, so as to realize the recognition of the non-driving area, i.e., obstacles. Next, combined with Figure 6 Step 330 is described in detail. Figure 6 As shown, step 330 may include the following steps 331 to 334.

[0180] Step 331 : The execution device 210 converts the first depth map from the pixel coordinate system to the vehicle coordinate system to obtain a fifth depth map.

[0181] As a possible example, the execution device 210 converts the first depth map from a pixel coordinate system to an image coordinate system, then converts the depth map in the image coordinate system to a camera coordinate system, and then converts the depth map in the camera coordinate system to a vehicle coordinate system to obtain a fifth depth map.

[0182] Among them, the pixel coordinate system is a two-dimensional rectangular coordinate system that reflects the arrangement of pixels in the camera sensor chip. The origin is located in the upper left corner of the image, and the u axis and v axis are parallel to the image plane. The pixel coordinates and image coordinates are actually in a translation relationship. The image coordinate system is a two-dimensional rectangular coordinate system. The origin is the intersection of the optical axis and the image plane (also known as the principal point), that is, the center point of the image. The x axis and y axis are parallel to the two sides of the image plane. The camera coordinate system is a three-dimensional rectangular coordinate system. The origin is located at the optical center of the lens. The x axis and y axis are parallel to the two sides of the image plane, and the z axis is the optical axis of the lens, which is perpendicular to the image plane. The world coordinate system is a three-dimensional rectangular coordinate system, which can be used as a reference to describe the spatial position of the camera and the object to be measured. The world coordinate system can be freely determined according to actual conditions, and often coincides with the machine motion coordinate system.

[0183] As a possible implementation manner, the execution device 210 may combine the intrinsic parameters of the camera and use the following formula (6) to transform the first depth map from the pixel coordinate system to the image coordinate system.

[0184] u=αx+c x ,v=βy+c y (6)

[0185] Among them, (u, v) represents the coordinates of the pixel point in the pixel coordinate system, (x, y) represents the coordinates of the pixel point in the image coordinate system, and (c x , c y ) is the coordinate of the principal point position in the camera’s intrinsic parameters, α=fx / f, β=fy / f, and f is the focal length in the camera’s intrinsic parameters.

[0186] As a possible implementation manner, the execution device 210 may combine the camera intrinsic parameters and the depth value of each pixel point and use the following formula (7) to transform the first depth map from the image coordinate system to the camera coordinate system.

[0187]

[0188] Among them, x c is the x-axis coordinate of the pixel in the camera coordinate system, y c is the y-axis coordinate of the pixel in the camera coordinate system, z c is the depth value of the pixel, f is the focal length in the intrinsic parameter of the camera, and (x, y) represents the coordinates of the pixel in the image coordinate system.

[0189] As a possible implementation manner, the execution device 210 can combine the camera external parameters and use the following formula (8) to transform the first depth map from the camera coordinate system to the vehicle coordinate system to obtain the fifth depth map.

[0190]

[0191] Among them, (x c ,y c , z c ) is the coordinate of the pixel in the camera coordinate system, (x w ,y w , z w ) is the coordinate of the pixel point in the vehicle coordinate system, R is the rotation matrix in the camera's extrinsic parameters, and T is the translation matrix in the camera's extrinsic parameters.

[0192] Step 332: The execution device 210 adds a plurality of grids to the fifth depth map.

[0193] As an example, the number of grids can be adjusted according to the size of the fifth depth map and the accuracy requirement of obstacle recognition. For example, the number of grids can be 10*10, 25*25, 100*100, etc. The numbers before and after * represent the number of grids in a row / column respectively.

[0194] Step 333: The execution device 210 determines whether the area included in each grid in the fifth depth map is a driving area or a non-driving area, and obtains an area identification map.

[0195] As a possible example, the execution device 210 uses a ground segmentation algorithm to divide the area contained in each grid in the fifth depth map into a driving area or a non-driving area. Among them, all areas outside the driving area are regarded as non-driving areas, and the non-driving areas are regarded as obstacles in this embodiment. For example, when a vehicle uses automatic driving (or assisted driving) and automatic parking (or assisted parking) functions to perform automatic parking, the ground is the driving area, and the wall, sky, ground lock, fire hydrant, etc. are non-driving areas.

[0196] The above-mentioned automatic parking can be automatic parking assistance, remote control parking assistance, memory parking assistance, autonomous valet parking, etc.

[0197] The present application does not limit the type of ground segmentation algorithm. For example, the ground segmentation algorithm can be a plane grid method, a point cloud normal vector method, a model fitting method, a surface element grid method, etc.

[0198] As a possible implementation method, the execution device 210 has obtained the semantic type of each pixel in the image, such as ground information, sky information, and wall information, in the above-mentioned step 320. The execution device 210 can use the semantic type of each pixel to correct the depth value of the fifth depth map in the process of segmenting the fifth image using the ground segmentation algorithm. For example, the depth value with the semantic type of sky information can be corrected to infinity to improve the segmentation accuracy.

[0199] Step 334 : The execution device 210 performs multi-frame fusion on the division results of the region identification map based on the division results of the historical frame images to obtain a fused division result.

[0200] As a possible example, the execution device 210 determines the consistency between the division results of the historical frame image and the division results of the same area in the area identification map, determines whether to merge the division results of the same area in the historical frame image into the division results of the area identification map, and thereby determines the fused division results.

[0201] Please refer to the following Figure 7 The specific steps of multi-frame fusion may include the following steps 710 to 750.

[0202] Step 710: The execution device 210 compares whether the division result of the first historical frame image and the division result of the current frame image are the same.

[0203] As a possible example, the execution device 210 compares whether the division results of corresponding pixel points in the first historical frame image and the current frame image are the same.

[0204] Optionally, the execution device 210 first determines corresponding pixel points in the first historical frame image and the current frame image. If the corresponding pixel points are both driving areas or non-driving areas in the division results of the first historical frame image and the division results of the current frame image, the execution device 210 determines that the division results of the two are the same; if the corresponding pixel points are both driving areas and non-driving areas in the division results of the first historical frame image and the division results of the current frame image, the execution device 210 determines that the division results of the two are different.

[0205] Optionally, the historical frame image includes multiple frame images. For example, if the current frame image is the fifth depth map and the current frame is a T frame, the historical frame image may include the T-1 frame image, the T-2 frame image, ..., the To frame image frame, etc.

[0206] The corresponding pixel points refer to the pixel points corresponding to the same position in the first historical frame image and the current frame image in reality. The first historical frame image is any frame image among the multiple frame images contained in the historical frame image. The above-mentioned current frame image and historical frame images such as the T-th frame image, the T-1-th frame image, the T-2-th frame image, ..., the To-th frame image are all depth maps output by steps 310 to 330, and o is a positive integer greater than 2.

[0207] Step 720: If the first historical frame image does not include the T-th frame image, or the first historical frame image includes the T-th frame image and the division result of the first historical frame image is the same as the division result of the T-th frame image, the execution device 210 maintains the division result of the T-th frame image, and updates the division result of the historical frame image based on the division result of the T-th frame image.

[0208] As a possible example, the T-th frame image mentioned above refers to the area A included in the T-th frame image. When the first historical frame image does not include the area A, or the first historical frame image includes the area A and the division result of the first historical frame image for the area A is the same as the division result of the T-th frame image for the area A, the execution device 210 maintains the division result of the T-th frame image for the area A, and adds the division result of the T-th frame image for the area A to the division result of the historical frame image.

[0209] Optionally, region A may be all regions or a certain region included in the Tth frame image.

[0210] Step 730: If the first historical frame image includes the T-th frame image and the division result of the first historical frame image is different from the division result of the T-th frame image, the execution device 210 keeps the division result of the T-th frame image when the number of times t that the division results are different is greater than the preset number M, otherwise t=t+1, and uses the division result of the first historical frame image to replace the division result of the T-th frame image.

[0211] As a possible example, when the first historical frame image includes area A, and the division result of the first historical frame image for area A is different from the division result of the T-th frame image for area A, the execution device 210 determines the number of times t that the division results of the multiple frame image pairs in the historical frame image are different from the division results of the T-th frame image for area A. If t is greater than the preset number M, the division result of the T-th frame image for area A is maintained; otherwise, t=t+1, and the division result of the first historical frame image for area A is used to replace the division result of the T-th frame image for area A.

[0212] Step 740: If the Tth frame image does not include the first historical frame image, the execution device 210 deletes the first historical frame image when the number s of times the Tth frame image does not include the first historical frame image is greater than the preset number N, otherwise s=s+1, and the first historical frame image is retained.

[0213] As a possible example, when the first historical frame image includes area B and the T-th frame image does not include area B, the execution device 210 determines the number of times s that multiple frames of images in the historical frame image include area B and the T-th frame image does not include area B. If s is greater than a preset number of times N, area B in the historical frame image is deleted; otherwise, area B in the historical frame image is retained.

[0214] Step 750: The execution device 210 outputs the division result of the Tth frame image.

[0215] As a possible example, the segmentation result of the T-th frame image after being updated through steps 710 to 740 is a fused segmentation result.

[0216] Based on the above steps 710 to 750, the execution device 210 performs multi-frame fusion on the ground segmentation result of the single frame image and the ground segmentation result of the historical frame image to obtain a fused ground segmentation result, thereby further improving the accuracy of the ground segmentation.

[0217] Combination of the above Figure 2-Figure 7 The obstacle detection method of the present application is described in detail. In conjunction with the above obstacle detection method, the present application also provides an obstacle detection device. Figure 8 Schematic diagram of a possible obstacle detection device provided in this embodiment. The obstacle detection device can be used to implement the functions of the execution device 210 or the training device 220 in the above method embodiment, and thus also has the beneficial effects of the above method embodiment. In this embodiment, the obstacle detection device can be as follows: Figure 2 The execution device 210, training device 220 or other device with computing function shown may also be a module (such as a chip) applied to a server.

[0218] like Figure 8 As shown, the obstacle detection device 800 includes a transceiver module 810 , a depth estimation module 820 and a segmentation module 830 .

[0219] The obstacle detection device 800 can be used to implement the above Figure 3a In the method embodiment shown in FIG. 8 , the functions of the device 210 are executed, and the various modules included in the obstacle detection device 800 are specifically used to implement the following functions.

[0220] The transceiver module 810 is used to obtain an image. For example, the transceiver module 810 is used to perform the following steps: Figure 3a Step 310 is shown.

[0221] The depth estimation module 820 is used to construct a first depth map of the image based on the structural constraint information, where the structural constraint information includes the semantic type of each pixel in the image. Figure 3a Step 320 is shown.

[0222] The semantic type includes at least one of ground information, wall information and sky information.

[0223] The segmentation module 830 is used to process the first depth map to obtain a region identification map, where the region identification map includes multiple regions, any of which is a driving region or a non-driving region, and the non-driving region is an obstacle. Figure 3a Step 330 is shown.

[0224] As a possible implementation manner, the depth estimation module 820 is specifically used to: input the image into a dense depth estimation model to obtain a first depth map, where the dense depth estimation model is trained based on structural constraint information.

[0225] As a possible implementation method, the segmentation module 830 is specifically used to: convert the first depth map from a pixel coordinate system to a vehicle coordinate system to obtain a fifth depth map; add multiple grids to the fifth depth map; use a ground segmentation algorithm to determine whether the area contained in each grid in the fifth depth map is a driving area or a non-driving area, and obtain an area identification map.

[0226] Optionally, the segmentation module 830 is specifically used to: correct the depth value of each pixel in the fifth depth map according to the semantic type of each pixel in the depth map under the vehicle coordinate system to obtain a corrected depth map; use a ground segmentation algorithm to determine whether the area contained in each grid in the corrected depth map is a driving area or a non-driving area to obtain an area identification map.

[0227] The obstacle detection device 800 can also be used to implement the above Figure 3a In the method embodiment shown in FIG. 8 , the obstacle detection device 800 may further include a training module 840 for implementing the following functions.

[0228] As a possible implementation method, the training module 840 is used to: construct at least one supervisory signal, at least one supervisory signal including a structural constraint supervisory signal; input a reference frame image into a monocular depth estimation network to obtain a second depth map output by the monocular depth estimation network; calculate the loss of each supervisory signal in at least one supervisory signal according to the second depth map; determine a total loss function according to the loss of at least one supervisory signal; and perform back propagation based on the total loss function to update the parameters of the dense depth estimation model.

[0229] Optionally, the training module 840 is specifically used to: project the point cloud corresponding to the reference frame image onto the reference frame image to obtain a third depth map; respectively fit the depth values ​​of pixel points belonging to the same semantic type in the third depth map to obtain a structural constraint supervision signal, that is, the plane depth of at least one plane contained in the third depth map.

[0230] Optionally, the semantic types of pixels of the reference frame image are extracted from the reference frame image by a semantic segmentation model.

[0231] Optionally, the loss of the structural constraint supervision signal includes the normal vector constraint loss of the plane, and the training module 840 is specifically used to: determine the pixel points of the first plane contained in the second depth map according to the plane depth of at least one plane, the first plane is one of the at least one plane contained in the second depth map; calculate the average normal vector of the pixel points of m sampling points in the first plane, m is an integer greater than or equal to 3, and each of the m sampling points contains multiple pixel points; calculate the average normal vector difference between the normal vector of each pixel point contained in the first plane and the average normal vector, and the normal vector constraint loss of the plane includes the average normal vector difference.

[0232] Optionally, the structural constraint supervision signal also includes a third depth map, and the loss of the structural constraint supervision signal includes the depth constraint loss of the plane. The training module 840 is specifically used to: determine the pixel points of the second plane contained in the second depth map according to the plane depth of at least one plane, the second plane is one of at least one plane contained in the second depth map, and the plane corresponding to the second plane in the third depth map is the third plane; fill the depth values ​​of the pixel points with missing depth values ​​in the third plane based on the plane depth of at least one plane to obtain a fourth depth map; calculate the average depth difference between the depth values ​​of the second plane of the second depth map and all the pixels in the same position in the fourth depth map, and the depth constraint of the plane includes the average depth difference.

[0233] As a possible implementation, the obstacle detection device 800 also includes a notification module 850, which is used to: in response to a notification instruction, present obstacle information of the obstacle to the user, the obstacle information including distance information and category information, the distance information is determined by the depth value of the obstacle, and the category information includes a first category and a second category, the first category is used to indicate an obstacle with which a collision is expected, and the second category is used to indicate an obstacle with which a collision is not expected.

[0234] It should be understood that the obstacle detection device 800 of the embodiment of the present invention can be implemented by a CPU, or by a dedicated integrated circuit, or by a programmable logic device (PLD), wherein the PLD can be a complex programmable logical device (CPLD), a field programmable gate array, a generic array logic (GAL), or any combination thereof. Figure 3a In the obstacle detection method shown in FIG. 8 , the obstacle detection device 800 and its various modules may also be software modules.

[0235] It should be understood that the execution device 210 or the training device 220 in the embodiment of the present application may correspond to the obstacle detection device 800 in the embodiment of the application, and may correspond to the corresponding subject in executing the method according to the embodiment of the present application, and the above and other operations and / or functions of each module in the obstacle detection device 800 are respectively to achieve Figure 3a For the sake of brevity, the corresponding process of the method in will not be repeated here.

[0236] For example, Fig. 9 This is a schematic diagram of the structure of a computing device provided in an embodiment of the present application. The computing device may be a terminal device, a network device, a robot, a vehicle-mounted computer, etc., or may be a chip (system) or other parts or components that can be set in a terminal device, a network device, a robot, or a vehicle-mounted computer. Fig. 9 As shown, the computing device 900 may include a processor 901. Optionally, the computing device 900 may also include a memory 902 and / or a transceiver 903. The processor 901 is coupled with the memory 902 and the transceiver 903, such as being connected via a communication bus.

[0237] Combine the following Fig. 9 The components of the computing device 900 are described in detail:

[0238] The processor 901 is the control center of the computing device 900, and may be a processor or a general term for multiple processing elements. For example, the processor 901 is one or more central processing units, or may be a dedicated integrated circuit, or may be configured to implement one or more integrated circuits of the embodiments of the present application, such as one or more microprocessors (digital signal processor, DSP), or one or more field programmable gate arrays.

[0239] Optionally, the processor 901 may execute various functions of the computing device 900 by running or executing software programs stored in the memory 902 and calling data stored in the memory 902, such as Figure 3a Steps 310 to 330 shown in the figure or functions implemented by each module such as the transceiver module 810, the depth estimation module 820 and the segmentation module 830 in the obstacle detection device 800.

[0240] In a specific implementation, as an embodiment, the processor 901 may include one or more CPUs, such as Fig. 9 CPU0 and CPU1 are shown in FIG.

[0241] In a specific implementation, as an embodiment, the computing device 900 may also include multiple processors, such as Fig. 9 901 and processor 904 are shown in FIG. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). A processor herein may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0242] The memory 902 is used to store a software program for executing the solution of the present application, for example, to enable the processor 901 to execute the following Figure 3a The software program for the functions implemented by the transceiver module 810, the depth estimation module 820 and the segmentation module 830 in the obstacle detection device 800 is shown in step 310-step 330, and is controlled and executed by the processor 901. The specific implementation method can refer to the above method embodiment and will not be repeated here. In addition, the memory 902 can also be used to store the image acquired by the camera and the point cloud data corresponding to the image.

[0243] Optionally, the memory 902 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 902 may be integrated with the processor 901, or may exist independently and accessed through the interface circuit ( Fig. 9 (not shown) is coupled to the processor 901, which is not specifically limited in the embodiment of the present application.

[0244] The transceiver 903 is used for communication with other devices. For example, if the computing device 900 is a vehicle-mounted computer, the transceiver 903 can be used to communicate with a robot or another vehicle-mounted computer.

[0245] Optionally, the transceiver 903 may include a receiver and a transmitter ( Fig. 9 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0246] Optionally, the transceiver 903 may be integrated with the processor 901, or may exist independently and be connected to the processor 901 through the interface circuit ( Fig. 9 (not shown) is coupled to the processor 901, which is not specifically limited in the embodiment of the present application.

[0247] It should be noted that Fig. 9 The structure of the computing device 900 shown in the figure does not constitute a limitation on the device, and an actual device may include more or less components than those shown in the figure, or combine certain components, or arrange the components differently.

[0248] In addition, the technical effects of the computing device 900 can refer to the technical effects of the obstacle detection method described in the above method embodiment, which will not be repeated here.

[0249] It should be understood that the processor in the embodiments of the present application may be a central processing unit, and the processor may also be other general-purpose processors, digital signal processors, application-specific integrated circuits, field programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0250] It should also be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0251] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0252] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0253] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0254] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0255] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0256] In addition, each functional module in each embodiment of the present application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0257] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk or an optical disk.

[0258] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An obstacle detection method, characterized in that: include: Get the image; constructing a first depth map of the image based on the structural constraint information, wherein the structural constraint information includes a semantic type of each pixel in the image; The first depth map is processed to obtain a region identification map, where the region identification map includes multiple regions, any region of the multiple regions is a driving region or a non-driving region, and the non-driving region is an obstacle.

2. The method according to claim 1, characterized in that The step of constructing a first depth map of the image based on the structural constraint information includes: The image is input into a dense depth estimation model to obtain the first depth map, where the dense depth estimation model is trained based on structural constraint information.

3. The method according to claim 2, characterized in that The method further includes a training process of the dense depth estimation model, wherein the dense depth estimation model includes a monocular depth estimation network, and the training process includes: constructing at least one supervision signal, the at least one supervision signal comprising a structural constraint supervision signal; Inputting the reference frame image into the monocular depth estimation network to obtain a second depth map output by the monocular depth estimation network; Calculating a loss for each supervisory signal in the at least one supervisory signal according to the second depth map; Determining a total loss function based on the loss of each supervisory signal in the at least one supervisory signal; Back propagation is performed based on the total loss function to update parameters of the dense depth estimation model.

4. The method according to claim 3, characterized in that The constructing at least one supervisory signal includes constructing the structural constraint supervisory signal, and the constructing the structural constraint supervisory signal includes: Projecting the point cloud corresponding to the reference frame image onto the reference frame image to obtain a third depth map, wherein the third depth map includes depth values ​​of pixels of different semantic types; The depth values ​​of the pixels of different semantic types are fitted respectively to obtain the structural constraint supervision signal.

5. The method according to claim 4, characterized in that The calculating the loss of each supervisory signal in the at least one supervisory signal according to the second depth map includes calculating the normal vector constraint loss of the plane of the structural constraint supervisory signal in the at least one supervisory signal according to the second depth map, and the calculating the normal vector constraint loss of the plane of the structural constraint supervisory signal in the at least one supervisory signal according to the second depth map includes: Determine a pixel point of a first plane included in the second depth map according to the structural constraint supervision signal, where the first plane is one of at least one plane included in the second depth map; Calculate an average normal vector of pixel points of m groups of sampling points in the first plane, where m is an integer greater than or equal to 3, and each group of sampling points in the m groups of sampling points includes a plurality of pixel points; An average normal vector difference between the normal vector of each pixel point included in the first plane and the average normal vector is calculated, and the normal vector constraint loss of the plane includes the average normal vector difference.

6. The method according to claim 5, characterized in that The structure-constrained supervision signal further includes the third depth map, and the calculating of the loss of each supervision signal in the at least one supervision signal according to the second depth map includes calculating the depth-constrained loss of the plane of the structure-constrained supervision signal in the at least one supervision signal according to the second depth map, and the calculating of the depth-constrained loss of the plane of the structure-constrained supervision signal in the at least one supervision signal according to the second depth map includes: Determine a pixel point of a second plane included in the second depth map according to the structural constraint supervision signal, where the second plane is one of at least one plane included in the second depth map, and a plane corresponding to the second plane in the third depth map is a third plane; Filling the depth values ​​of the pixels whose depth values ​​are missing in the third plane based on the structural constraint supervision signal to obtain a fourth depth map; An average depth difference between the depth values ​​of all co-located pixels in the second plane of the second depth map and the fourth depth map is calculated, and the depth constraint of the plane includes the average depth difference.

7. The method according to any one of claims 1 to 6, characterized in that The step of processing the first depth map to obtain a region identification map includes: Converting the first depth map from a pixel coordinate system to a vehicle coordinate system to obtain a fifth depth map; adding a plurality of grids to the fifth depth map; A ground segmentation algorithm is used to determine whether the area included in each grid in the fifth depth map is the driving area or the non-driving area, and the area identification map is obtained.

8. The method according to claim 7, characterized in that The method of using a ground segmentation algorithm to determine whether the area included in each grid in the fifth depth map is the driving area or the non-driving area to obtain the area identification map includes: Correcting the depth value of each pixel in the fifth depth map according to the semantic type of each pixel in the depth map in the vehicle coordinate system to obtain a corrected depth map; The ground segmentation algorithm is used to determine whether the area included in each grid in the modified depth map is the driving area or the non-driving area, and the area identification map is obtained.

9. The method according to any one of claims 1 to 8, characterized in that The method further comprises: In response to the notification instruction, obstacle information of the obstacle is presented to the user, the obstacle information including distance information and category information, the distance information is determined by the depth value of the obstacle, the category information includes a first category and a second category, the first category is used to indicate an obstacle with which a collision is expected, and the second category is used to indicate an obstacle with which a collision is not expected.

10. The method according to any one of claims 1 to 9, characterized in that The semantic type includes at least one of ground information, wall information and sky information.

11. An obstacle detection device, characterized in that: The device comprises a module for executing the operation steps of any one of the methods of claims 1-10.

12. A computing device, characterized in that: The computing device includes a memory and a processor, wherein the memory is used to store a set of computer instructions; when the processor executes the set of computer instructions, the operation steps of the method described in any one of claims 1 to 10 are performed.

13. An obstacle detection system, characterized in that: The method comprises at least one camera and a computing device as claimed in claim 12, wherein the at least one camera is used to acquire an image, and the computing device is used to perform the operation steps of the method according to any one of claims 1 to 10 according to the image.

14. A readable storage medium, characterized in that: The readable storage medium includes a computer program or instructions. When the computer program or instructions are executed on a computer, the computer is caused to execute the operation steps of the method described in any one of claims 1 to 10.

Images