Method for determining spatial position of obstacle, method and device for training model

Abstract

The embodiment of the present disclosure discloses a method for determining the spatial position of an obstacle, a training method and device of a model, wherein the method for determining the spatial position of the obstacle comprises: acquiring a first image collected by a camera with a preset visual angle on a vehicle; extracting an image feature about the obstacle in the first image; determining a three-dimensional spatial feature of the obstacle based on the image feature; decoding the three-dimensional spatial feature to obtain occupancy grid information of the obstacle in the first image under a three-dimensional coordinate system; the three-dimensional coordinate system is a vehicle local coordinate system in which the vehicle is located; and determining the three-dimensional spatial position of the obstacle in the vehicle local coordinate system based on the occupancy grid information. The embodiment of the present disclosure can determine the spatial position of the obstacle during the driving of the vehicle based on the image of multiple visual angles, reduce the cost of determining the spatial position of the obstacle, thereby greatly improving the universality of the method for determining the spatial position of the obstacle, and being helpful for large-scale application.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the computer vision technology, and in particular, to a method for determining spatial position of an obstacle, a method for training a model and an apparatus. BACKGROUND

[0002] In the field of computer-aided driving, three-dimensional (3D) perception and modeling of the environment around a vehicle can effectively improve the driving safety of the vehicle. In the related art, the determination of the spatial position of an obstacle around a vehicle is usually based on a laser radar or a combination of a laser radar and an image sensor. However, due to the high cost of the laser radar, the determination method of the spatial position of the obstacle in the related art has poor versatility and is limited in scale. SUMMARY

[0003] To solve the above technical problems of poor versatility and limited application scale, embodiments of the present disclosure provide a method for determining the spatial position of an obstacle, a method for training a model and an apparatus. The spatial position of an obstacle during driving of a vehicle can be determined based on multi-view images, thereby reducing the cost of determining the spatial position of the obstacle, and greatly improving the versatility of the method for determining the spatial position of the obstacle, which is conducive to large-scale application.

[0004] In a first aspect, the present disclosure provides a method for determining the spatial position of an obstacle during driving of a vehicle, comprising: obtaining a first image captured by a camera at a preset viewing angle on the vehicle; extracting an image feature of an obstacle in the first image; determining a three-dimensional spatial feature of the obstacle based on the image feature; decoding the three-dimensional spatial feature to obtain occupancy grid information of the obstacle in the first image in a three-dimensional coordinate system; the three-dimensional coordinate system is a vehicle local coordinate system in which the vehicle is located; and determining the three-dimensional spatial position of the obstacle in the vehicle local coordinate system based on the occupancy grid information.

[0005] In a second aspect of the present disclosure, a method for training an obstacle space position determination model is provided, including: obtaining a first image sample set and camera poses corresponding to each sample image in the first image sample set, the first image sample set including at least one frame of image data collected by at least one camera with a preset view angle on a vehicle, each frame of image data including sample images of each preset view angle; determining, based on a to-be-trained obstacle space position determination model, occupancy grid information in a three-dimensional coordinate system corresponding to each frame of image data in the first image sample set; the three-dimensional coordinate system corresponding to each frame of image data being a local coordinate system of the vehicle when the vehicle collects the frame of image data; for each frame of image data, determining a depth map corresponding to each sample image in the frame of image data based on the occupancy grid information corresponding to the frame of image data and the camera poses corresponding to each sample image in the frame of image data; determining, based on the depth maps corresponding to each sample image, a photometric loss corresponding to each sample image; and determining a target obstacle space position determination model based on the photometric losses corresponding to each sample image in the first image sample set.

[0006] In a third aspect of the present disclosure, a device for determining a space position of an obstacle during driving of a vehicle is provided, including: an image obtaining module configured to obtain a first image collected by a camera with a preset view angle on the vehicle; a feature extracting module configured to extract image features of the obstacle in the first image; a space feature determining module configured to determine three-dimensional space features of the obstacle based on the image features; a space feature decoding module configured to decode the three-dimensional space features to obtain occupancy grid information of the obstacle in the first image in a three-dimensional coordinate system; the three-dimensional coordinate system being a local coordinate system of the vehicle; and a space position determining module configured to determine a three-dimensional space position of the obstacle in the local coordinate system of the vehicle based on the occupancy grid information.

[0007] In a fourth aspect of the present disclosure, a device for training an obstacle space position determination model is provided, including: a sample obtaining module configured to obtain a first image sample set and camera poses corresponding to each sample image in the first image sample set, the first image sample set including at least one frame of image data collected by at least one camera with a preset view angle on a vehicle, each frame of image data including sample images of each preset view angle;

[0008] The first processing module is configured to determine, based on the obstacle space position determination model to be trained, the occupancy grid information in a three-dimensional coordinate system corresponding to each frame of image data in the first image sample set. The three-dimensional coordinate system corresponding to each frame of image data is a local coordinate system of the vehicle when the vehicle collects the frame of image data. The second processing module is configured to, for each frame of image data, determine a depth map corresponding to each sample image in the frame of image data based on the occupancy grid information corresponding to the frame of image data and the camera pose corresponding to each sample image in the frame of image data. The third processing module is configured to determine the photometric loss corresponding to each sample image based on the depth map corresponding to each sample image. The fourth processing module is configured to determine the target obstacle space position determination model based on the photometric loss corresponding to each sample image in the first image sample set.

[0009] In a fifth aspect, the present disclosure provides a computer-readable storage medium storing a computer program, where the computer program is configured to execute the method for determining the obstacle space position during the driving of the vehicle according to any one of the above-mentioned embodiments of the present disclosure, or execute the method for training the obstacle space position determination model according to any one of the above-mentioned embodiments of the present disclosure.

[0010] In a sixth aspect, the present disclosure provides an electronic device, which includes a processor, a memory for storing executable instructions of the processor, and the processor is configured to read the executable instructions from the memory and execute the instructions to implement the method for determining the obstacle space position during the driving of the vehicle according to any one of the above-mentioned embodiments of the present disclosure, or implement the method for training the obstacle space position determination model according to any one of the above-mentioned embodiments of the present disclosure.

[0011] In a seventh aspect, the present disclosure provides a computer program product, when the instructions in the computer program product are executed by a processor, the method for determining the obstacle space position during the driving of the vehicle according to any one of the above-mentioned embodiments of the present disclosure is executed, or the method for training the obstacle space position determination model according to any one of the above-mentioned embodiments of the present disclosure is executed.

[0012] The method for determining the spatial position of an obstacle, the training method and the device of the model provided by the above embodiments of the present disclosure can extract image features related to the obstacle in the first image based on the first image collected by the camera with the preset perspective on the vehicle, determine the three-dimensional spatial features of the obstacle based on the image features, and further decode the three-dimensional spatial features to obtain the occupancy grid information of the obstacle in the first image in the three-dimensional coordinate system (i.e., the local coordinate system of the vehicle). Based on the occupancy grid information, the three-dimensional spatial position of the obstacle in the three-dimensional coordinate system can be determined. Thus, the spatial position of the obstacle in the image is determined based on the image with the preset perspective, avoiding the dependence on the laser radar, effectively reducing the cost of determining the spatial position of the obstacle, greatly improving the universality of the method for determining the spatial position of the obstacle, and facilitating large-scale application. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 is an exemplary application scenario of the method for determining the spatial position of an obstacle during driving of a vehicle provided by the present disclosure;

[0014] Figure 2 is a flowchart of the method for determining the spatial position of an obstacle during driving of a vehicle provided by an exemplary embodiment of the present disclosure;

[0015] Figure 3 is a schematic diagram of the camera with the preset perspective provided by an exemplary embodiment of the present disclosure;

[0016] Figure 4 is a flowchart of the method for determining the spatial position of an obstacle provided by another exemplary embodiment of the present disclosure;

[0017] Figure 5 is a network structure diagram of the model for determining the spatial position of an obstacle provided by an exemplary embodiment of the present disclosure;

[0018] Figure 6 is a flowchart of the training method of the model for determining the spatial position of an obstacle provided by an exemplary embodiment of the present disclosure;

[0019] Figure 7 is a flowchart of the method for obtaining the model for determining the spatial position of an obstacle provided by an exemplary embodiment of the present disclosure;

[0020] Figure 8 is a flowchart of the training method of the model for determining the spatial position of an obstacle provided by another exemplary embodiment of the present disclosure;

[0021] Figure 9 is a flowchart of the training method of the model for determining the spatial position of an obstacle provided by another exemplary embodiment of the present disclosure;

[0022] Figure 10is a flowchart of a method for training an obstacle space position determination model according to an example embodiment of the present disclosure;

[0023] Figure 11 is a flowchart of a method for training an obstacle space position determination model according to an example embodiment of the present disclosure;

[0024] Figure 12 is a flowchart of a method for training an obstacle space position determination model according to an example embodiment of the present disclosure;

[0025] Figure 13 is a flowchart of a method for training an obstacle space position determination model according to an example embodiment of the present disclosure;

[0026] Figure 14 is a flowchart of a method for training an obstacle space position determination model according to an example embodiment of the present disclosure;

[0027] Figure 15 is a schematic diagram of a network structure of an obstacle space position determination model according to an example embodiment of the present disclosure;

[0028] Figure 16 is a schematic diagram of an apparatus for determining an obstacle space position during driving of a vehicle according to an example embodiment of the present disclosure;

[0029] Figure 17 is a schematic diagram of an apparatus for determining an obstacle space position during driving of a vehicle according to an example embodiment of the present disclosure;

[0030] Figure 18 is a schematic diagram of a training apparatus for an obstacle space position determination model according to an example embodiment of the present disclosure;

[0031] Figure 19 is a schematic diagram of a training apparatus for an obstacle space position determination model according to an example embodiment of the present disclosure;

[0032] Figure 20 is a schematic diagram of a training apparatus for an obstacle space position determination model according to an example embodiment of the present disclosure;

[0033] Figure 21 is a structural diagram of an electronic device according to an example embodiment of the present disclosure. DETAILED DESCRIPTION

[0034] In order to explain the present disclosure, the example embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings, obviously, the described embodiments are only part of the embodiments of the present disclosure, not all the embodiments, it should be understood that the present disclosure is not limited by the example embodiments.

[0035] It should be noted that the relative arrangement, numerical expressions, and numerical values of the components and steps set forth in these embodiments do not limit the scope of the present disclosure unless otherwise specifically stated.

[0036] Summary of the Disclosure

[0037] In the process of implementing the present disclosure, the inventors found that in the field of computer-aided driving, three-dimensional (3D) perception and modeling of the vehicle's surrounding environment can effectively improve the safety of vehicle driving. In the related art, the spatial position of the vehicle's surrounding obstacles is usually determined based on a laser radar or in combination with a laser radar and an image sensor, but due to the high cost of the laser radar, the related art method for determining the spatial position of the obstacle has poor versatility and limited applicable scale.

[0038] Exemplary Overview

[0039] Figure 1 is an exemplary application scenario of the method for determining the spatial position of the obstacle in the process of driving the vehicle provided by the present disclosure.

[0040] As shown in Figure 1 , there can be m preset viewing angles, each preset viewing angle corresponding to a camera, the image captured by each camera serving as the first image of the preset viewing angle corresponding to the camera, or the image captured by the camera being preprocessed and then serving as the first image of the corresponding preset viewing angle, the preprocessing may, for example, include scaling, cropping, etc. In the assisted driving scenario, a device for determining the spatial position of the obstacle in the process of driving the vehicle (referred to as the device for determining the spatial position of the obstacle or the device) can be arranged on the vehicle-mounted computing platform of the vehicle, and the method for determining the spatial position of the obstacle in the process of driving the vehicle (which can be referred to as the method for determining the spatial position of the obstacle or the method) of the present disclosure can be run in the device. By using the method for determining the spatial position of the obstacle in the process of driving the vehicle of the present disclosure, the first image captured by the camera at the preset viewing angle on the vehicle can be obtained, the image features of the obstacle in the first image can be extracted, the three-dimensional spatial features of the obstacle can be determined based on the image features, the three-dimensional spatial features can be decoded, the occupancy grid information of the obstacle in the first image under the three-dimensional coordinate system (i.e., the vehicle local coordinate system in which the vehicle is located) can be obtained, and then the three-dimensional spatial position of the obstacle in the vehicle local coordinate system can be determined based on the occupancy grid information. The number m (m is a positive integer) of the preset viewing angles can be set according to the demand for the viewing angle of the vehicle, for example Figure 1The middle m can be set to 2, 3, 4, 5, 6, etc. The vehicle local coordinate system can be a self-vehicle coordinate system with the center of the rear axle of the vehicle as the coordinate origin, the length direction of the vehicle as the longitudinal axis (x-axis), and the width direction of the vehicle as the transverse axis (y-axis). The method of the embodiment of the present disclosure realizes the determination of the spatial position of the obstacle based on the image, can avoid the dependence on the laser radar, greatly reduces the cost of determining the spatial position of the obstacle, thereby effectively improving the universality of the determination method of the spatial position of the obstacle, and is helpful for large-scale application.

[0041] Exemplary Method

[0042] Figure 2 FIG. 1 is a flow diagram of a method for determining the spatial position of an obstacle during vehicle driving provided by an example embodiment of the present disclosure. The embodiment can be applied on an electronic device, specifically, for example, on a vehicle-mounted computing platform, such as Figure 2 as shown, comprising the following steps:

[0043] Step 201: acquiring a first image captured by a camera at a preset viewing angle on the vehicle.

[0044] The preset viewing angle can include one or more viewing angles, and the viewing angle refers to the camera viewing angle, such as one viewing angle corresponding to each camera on the vehicle. The number of preset viewing angles can be set according to actual needs, for example, Figure 3 FIG. 2 is a schematic diagram of a camera at a preset viewing angle provided by an example embodiment of the present disclosure. The surround-view cameras on the vehicle include a front-viewing-angle camera (corresponding to a preset viewing angle 1), a left-front-viewing-angle camera (corresponding to a preset viewing angle 2), a left-rear-viewing-angle camera (corresponding to a preset viewing angle 3), a right-front-viewing-angle camera (corresponding to a preset viewing angle 4), a right-rear-viewing-angle camera (corresponding to a preset viewing angle 5), and a rear-viewing-angle camera (corresponding to a preset viewing angle 6). There are a total of 6 preset viewing angles, and each camera at a preset viewing angle can capture an environmental image within the corresponding viewing angle range around the vehicle. The cameras at the 6 preset viewing angles can cover the surround-view angle around the vehicle, realizing the perception of the obstacles around the vehicle in the surround-view scene. The image captured by each camera at a preset viewing angle is taken as the first image at the preset viewing angle.

[0045] Step 202: extracting an image feature related to the obstacle in the first image.

[0046] The extraction of the image feature can be realized by any implementable feature extraction method, for example, can be realized based on any implementable feature extraction network. The feature extraction network can be set according to actual needs, for example, a backbone network based on Resnet50, U-NET, etc. can be used as the feature extraction network.

[0047] In some optional embodiments, the extracted image features about the obstacle can include image features of at least one scale, for example, at least one of 1 / 4 scale, 1 / 8 scale, 1 / 16 scale, and the like of the first image.

[0048] In some optional embodiments, the first image can also be subjected to semantic segmentation to obtain a semantic segmented image. The semantic segmented image includes not only the original image information of the first image, but also a semantic type corresponding to each pixel, which is helpful for the extraction of the image features about the obstacle.

[0049] In some optional embodiments, when the image features about the obstacle in the first image are extracted, the first image can be subjected to semantic segmentation to obtain a semantic type corresponding to each pixel in the first image, and then objects of a preset type in the first image are filtered out according to the semantic type corresponding to each pixel in the first image to obtain a second image corresponding to the first image after filtering out the objects of the preset type. The image features about the obstacle are obtained by performing feature extraction on the second image. The preset type can be, for example, the sky or the like, which cannot be represented by three-dimensional spatial features. By filtering out the objects of the preset type, the effectiveness of the image features can be further improved, thereby improving the accuracy of the determined spatial position of the obstacle.

[0050] Step 203: determining three-dimensional spatial features of the obstacle based on the image features.

[0051] The three-dimensional spatial features can be obtained by dimensionality upgrading of the image features. For example, the three-dimensional spatial features can be obtained based on a two-dimensional to three-dimensional dimensionality conversion network.

[0052] Step 204: decoding the three-dimensional spatial features to obtain occupancy grid information of the obstacle in the first image in a three-dimensional coordinate system; the three-dimensional coordinate system is a vehicle local coordinate system of the vehicle.

[0053] The vehicle local coordinate system can be a self-vehicle coordinate system with a center of a rear axle of the vehicle as a coordinate origin, a length direction of the vehicle as a longitudinal axis (x-axis), and a width direction of the vehicle as a transverse axis (y-axis).

[0054] In some optional embodiments, the occupancy grid information can include an occupied probability corresponding to a vertex of each grid, and can also include coordinates of the vertex of each grid in the three-dimensional coordinate system and other related information. The grid refers to a voxel or a small cube in the space of the three-dimensional coordinate system, i.e., a small cube with a preset length, width, and height. The vertex of the grid refers to a vertex of the small cube.

[0055] In some optional embodiments, the occupancy grid information can further include the types of obstacles occupying each grid vertex, and the types of obstacles can be set according to actual needs. For example, the types of obstacles can include types such as vehicles, pedestrians, buildings, traffic signs, traffic facilities, and the like, and are not limited in particular.

[0056] In some optional embodiments, the decoding of the three-dimensional spatial features can be implemented based on any implementable decoding network, for example, based on a convolutional network and an improved network thereof.

[0057] In step 205, based on the occupancy grid information, the three-dimensional spatial position of the obstacle in the local coordinate system of the vehicle is determined.

[0058] The occupancy grid information represents the probability of each small cube being occupied by an obstacle in the three-dimensional coordinate system space. Therefore, the three-dimensional spatial position of the obstacle in the three-dimensional coordinate system can be determined based on the probability of each small cube being occupied by an obstacle. For example, the occupancy grid information can be mapped to the three-dimensional spatial position of the obstacle based on a certain mapping rule. The specific mapping rule can be set according to actual needs. For example, the occupied state of each grid vertex and the type of obstacle occupying the grid vertex can be determined according to the probability of each grid vertex being occupied by different obstacles, and based on the occupied state of each grid vertex and the type of obstacle, combined with the coordinates of each grid vertex in the three-dimensional coordinate system, the three-dimensional spatial position of the obstacle in the three-dimensional coordinate system can be obtained.

[0059] In some optional embodiments, the three-dimensional spatial positions of different obstacles can be represented by different colors or different identifiers. For example, the grid space occupied by the obstacle belonging to the vehicle in the three-dimensional coordinate system can be marked as red, the grid space occupied by the obstacle belonging to the pedestrian can be marked as blue, the grid space belonging to the building can be marked as gray, and the like. Thus, accurate and effective three-dimensional spatial positions of obstacles can be provided for subsequent planning and control tasks.

[0060] The method for determining the spatial position of an obstacle during vehicle driving provided by the embodiments of the present disclosure can extract image features of the obstacle in the first image based on the first image collected by the camera with the preset view angle on the vehicle, determine the three-dimensional spatial features of the obstacle based on the image features, decode the three-dimensional spatial features, obtain the occupancy grid information of the obstacle in the first image in the three-dimensional coordinate system (i.e., the local coordinate system of the vehicle), and determine the three-dimensional spatial position of the obstacle in the three-dimensional coordinate system based on the occupancy grid information. Thus, the spatial position of the obstacle in the image is determined based on the image with the preset view angle, avoiding the dependence on the laser radar, effectively reducing the cost of determining the spatial position of the obstacle, greatly improving the universality of the method for determining the spatial position of the obstacle, and facilitating large-scale application.

[0061] Figure 4is a flowchart of a method for determining the spatial position of an obstacle provided by another example embodiment of the present disclosure.

[0062] In some optional embodiments, the extracting, in step 202, the image features about the obstacle in the first image comprises:

[0063] In step 2021, the feature extraction network in the obstacle spatial position determination model is used to perform feature extraction on the first image to obtain the image features.

[0064] The feature extraction network can be implemented by any implementable network, for example, a backbone network based on Resnet50, U-NET, etc.

[0065] In some optional embodiments, before step 2021, the first image can also be subjected to semantic segmentation by using a semantic segmentation network to obtain a semantic segmentation result, and a semantic segmented image is determined based on the first image and the semantic segmentation result. Correspondingly, step 2021 comprises using the feature extraction network in the obstacle spatial position determination model to perform feature extraction on the semantic segmented image to obtain the image features.

[0066] The semantic segmentation network can be a pre-trained network, the semantic segmentation result can comprise a semantic type corresponding to each pixel in the first image, the semantic segmentation result can be represented as a single-channel image with the same resolution as the first image, and the semantic segmented image can be obtained by splicing the single-channel image and the first image in the channel direction, so that the semantic segmented image comprises both the image information of the first image and the semantic type of each pixel. When performing feature extraction, the feature extraction network is used to perform feature extraction on the semantic segmented image to obtain the image features. It should be noted that, for the case of performing feature extraction on the semantic segmented image, the network construction and training process of the obstacle spatial position determination model should be constructed according to the data structure of the semantic segmented image, and the sample images used for training should also be semantic segmented sample images, so as to ensure the consistency of the inference logic in the model training stage and the application stage.

[0067] The feature extraction network in the obstacle spatial position determination model is used to perform feature extraction on the first image, thereby providing effective image features for the feature dimension conversion.

[0068] In some optional embodiments, the determining, in step 203, the three-dimensional spatial features of the obstacle based on the image features comprises:

[0069] In step 2031, the dimension conversion network in the obstacle spatial position determination model is used to perform dimension conversion on the image features to obtain the three-dimensional spatial features of the obstacle.

[0070] In some optional embodiments, the dimension conversion network can be implemented based on a three-dimensional convolutional network. Alternatively, it can also be implemented based on three-dimensional convolution and cross-attention mechanism, and the specific network structure is not limited as long as it can realize the corresponding feature dimension conversion function.

[0071] The dimension conversion network is used to convert the two-dimensional image features into three-dimensional space features of the obstacles in the embodiments of the present disclosure, which helps to obtain accurate and effective obstacle occupancy grid information through decoding, thereby improving the accuracy of the three-dimensional space position of the obstacles.

[0072] In some optional embodiments, the decoding of the three-dimensional space features in step 204 to obtain the occupancy grid information of the obstacles in the first image in the three-dimensional coordinate system comprises:

[0073] In step 2041, the decoding network in the obstacle space position determination model is used to decode the three-dimensional space features to obtain the occupancy grid information of the obstacles.

[0074] The network structure of the decoding network may, for example, comprise one or more decoders. Each decoder can realize the restoration of the three-dimensional space features corresponding to the low-resolution image features into high-resolution three-dimensional space features, and map the high-resolution three-dimensional space features into the occupancy probability of each grid vertex in the three-dimensional coordinate system space to obtain the occupancy grid information of the obstacles.

[0075] In some optional embodiments, the high-resolution three-dimensional space features can be obtained based on the upsampling manner. For example, the upsampling can be realized based on a three-dimensional deconvolutional network.

[0076] In some optional embodiments, the image features can comprise multi-scale image features, and the three-dimensional space features obtained through the dimension conversion can comprise multi-scale three-dimensional space features. When the low-resolution three-dimensional space features are restored into high-resolution three-dimensional space features, the three-dimensional space features obtained through the upsampling based on the low-resolution three-dimensional space features can be fused with the three-dimensional space features of the corresponding scale obtained through the dimension conversion, and the fusion result can be taken as the restored high-resolution three-dimensional space features, so that the final high-resolution three-dimensional space features are obtained through multiple times of upsampling and feature fusion.

[0077] The decoding network is used to decode the three-dimensional space features to obtain the occupancy grid information of the obstacles in the embodiments of the present disclosure, which helps to obtain the three-dimensional space position of the obstacles.

[0078] In some optional embodiments, Figure 5 is a network structure schematic diagram of the obstacle space position determination model provided by an exemplary embodiment of the present disclosure. As shown in Figure 5As shown, the network structure of the obstacle space position determination model includes a feature extraction network 21, a dimension conversion network 22, a decoding network 23, and an output mapping unit 24. The feature extraction network 21 is configured to perform feature extraction on the first image of the preset perspective, to obtain image features corresponding to the first image. The dimension conversion network 22 is configured to perform dimension conversion on the image features, to convert the image features into three-dimensional space features. The decoding network 23 is configured to decode the three-dimensional space features, to obtain the occupancy grid information of the obstacle in the three-dimensional coordinate system. The output mapping unit 24 is configured to determine the three-dimensional space position of the obstacle based on the occupancy grid information. The number of the feature extraction network 21 can be one or more. When there are multiple feature extraction networks 21, the number of the feature extraction networks 21 can be determined according to the number of the preset perspectives. The multiple feature extraction networks 21 can share network parameters, and the multiple feature extraction networks can perform feature extraction on the first images of the respective preset perspectives in parallel. When the number of the feature extraction network 21 is one, the feature extraction network 21 can perform feature extraction on the first images of the respective preset perspectives respectively, to obtain image features corresponding to the respective first images.

[0079] It should be noted that the obstacle space position determination model of the embodiments of the present disclosure needs to be trained in advance. The specific training process can be referred to the training method embodiments of the obstacle space position determination model of the present disclosure, which will not be repeated here.

[0080] The above-mentioned embodiments of the present disclosure can be implemented alone or in any combination without conflict. The specific implementation can be set according to the actual needs, and the present disclosure is not limited.

[0081] Any of the methods for determining the space position of the obstacle provided by the embodiments of the present disclosure can be executed by any appropriate device with data processing capability, including but not limited to terminal devices and servers, etc. Alternatively, any of the methods for determining the space position of the obstacle provided by the embodiments of the present disclosure can be executed by a processor, such as a processor executing any of the methods for determining the space position of the obstacle mentioned in the embodiments of the present disclosure by calling corresponding instructions stored in a memory. The following will not be repeated.

[0082] Figure 6 FIG. 1 is a flow diagram of a method for training an obstacle space position determination model according to an example embodiment of the present disclosure. The method can be applied on an electronic device, such as a server, a terminal, etc. Figure 6 As shown, the method includes the following steps:

[0083] In step 301, a first image sample set and camera poses corresponding to sample images in the first image sample set are obtained. The first image sample set includes at least one frame of image data collected by at least one camera of a vehicle at a preset perspective, and each frame of image data includes a sample image at the preset perspective.

[0084] wherein the camera pose is a pose of the camera in a reference coordinate system, which can be a world coordinate system or a relatively stable coordinate system rigidly connected with the world coordinate system, such as a camera coordinate system of the camera at an initial position (such as a position at which the first frame of image data is collected). The sample image in each frame of image data corresponds to the first image in the foregoing embodiments, and the first image represents an image of the model in an actual prediction situation, and the sample image represents an image of the model in a training phase, for distinguishing different phases.

[0085] In some optional embodiments, each sample image included in the first image sample set can be an image in which non-obstacle elements such as the sky are filtered out, so as to further improve the performance of the model.

[0086] In some optional embodiments, the camera pose corresponding to the sample image can be obtained through a preconfigured positioning manner in a sample collection process. For example, the camera pose corresponding to each sample image can be obtained through a positioning manner of SLAM (Simultaneous Localization and Mapping) based on a sequence of collected sample images or in combination with data of other sensors (such as data of a laser radar, an ultrasonic radar, a millimeter wave radar, etc.), and the specific manner is not limited.

[0087] In step 302, based on the to-be-trained obstacle space position determination model, the occupancy grid information of each frame of image data in the first image sample set in a three-dimensional coordinate system is determined; the three-dimensional coordinate system corresponding to each frame of image data is a vehicle local coordinate system in which the vehicle is located when the frame of image data is collected.

[0088] wherein for each frame of image data, the corresponding occupancy grid information can include an occupied probability corresponding to each vertex of each grid, and can also include coordinates of each vertex of each grid in the three-dimensional coordinate system. The grid refers to a voxel or a small cube in the space of the three-dimensional coordinate system, that is, a small cube with a preset length, width and height. The vertex of the grid refers to the vertex of the small cube.

[0089] In some optional embodiments, each sample image can also be a semantic segmentation image, i.e., the sample image includes not only image information but also a semantic type of each pixel, which is helpful for determining the three-dimensional spatial positions of different obstacles. Alternatively, the network structure of the obstacle spatial position determination model to be trained can include a semantic segmentation network, the network parameters of the semantic segmentation network being trained parameters, serving as hyperparameters of the obstacle spatial position determination model to be trained, and the network parameters of the semantic segmentation network not being updated in the training process. The semantic segmentation network is used to perform semantic segmentation on the sample image, and the semantic segmentation result is fused with the sample image to obtain a semantic segmentation image. For example, the semantic type of each pixel of the sample image included in the semantic segmentation result is spliced with the sample image in the channel direction as a single-channel image to obtain the semantic segmentation image. The semantic segmentation image is used as the sample image for training of the model. Further, the occupancy grid information is determined based on the semantic segmentation image.

[0090] In step 303, for each frame of image data, a depth map corresponding to each sample image in the frame of image data is determined based on the occupancy grid information corresponding to the frame of image data and the camera pose corresponding to each sample image in the frame of image data.

[0091] In some optional embodiments, the depth map corresponding to each sample image can be determined based on any implementable volume rendering manner.

[0092] In some optional embodiments, the depth map corresponding to each sample image can be determined based on a differentiable rendering manner.

[0093] In some optional embodiments, the depth map corresponding to each sample image can also be obtained based on a depth estimation network. The depth estimation network is used to perform depth estimation on the three-dimensional spatial features corresponding to each sample image to obtain the depth map corresponding to each sample image. The depth estimation network can be implemented by using a convolution-based or improved network.

[0094] In step 304, a photometric loss corresponding to each sample image is determined based on the depth map corresponding to each sample image.

[0095] The photometric loss refers to a photometric consistency loss, and specifically, can refer to a photometric consistency loss between a reconstructed image and an actual image.

[0096] In some optional embodiments, for each sample image, at least one of the adjacent frame image of the sample image and the adjacent view image of the sample image can be reconstructed based on the depth map corresponding to the sample image, and the luminosity loss corresponding to the sample image is determined based on the reconstructed adjacent frame image and the corresponding adjacent frame sample image and / or the reconstructed adjacent view image and the corresponding adjacent view sample image. The reconstruction of the adjacent frame image and / or the adjacent view image of the sample image can be achieved through coordinate system conversion based on the depth map, the sample image and the corresponding camera pose, the adjacent frame image and the corresponding camera pose, and / or the adjacent view image and the corresponding camera pose.

[0097] In step 305, a target obstacle space position determination model is determined based on the luminosity loss corresponding to each sample image in the first image sample set.

[0098] The network parameters of the to-be-trained obstacle space position determination model can be iteratively updated by using a preset gradient descent method according to the luminosity loss corresponding to each sample image, the training of the model is completed, and the target obstacle space position determination model is obtained.

[0099] In some optional embodiments, the preset gradient descent method can be, for example, a stochastic gradient descent method, a gradient descent method with adaptive learning rate, etc., and is not specifically limited.

[0100] In some optional embodiments, Figure 7 is a flowchart for obtaining an obstacle space position determination model provided by an example embodiment of the present disclosure. As shown in Figure 7 , the flowchart can include:

[0101] S1, a vehicle collects sensor data.

[0102] The sensor data can include image data collected by the vehicle during a period of driving, inertial measurement unit (IMU) data, global positioning system (GPS) data, and other sensor data.

[0103] S2, sensor data processing.

[0104] The sensor data processing can include semantic segmentation of the collected image data to obtain a semantic segmentation result, and can also include pose estimation of the camera pose of each image in the collected image data based on the image data, the IMU data, the GPS data, and the like, to obtain the camera pose corresponding to each image.

[0105] S3, training of an obstacle space position determination model.

[0106] The space position of the obstacle in the three-dimensional space of the road on which the vehicle travels can be obtained through the training of the model, thereby helping to achieve three-dimensional reconstruction of the obstacle of the road on which the vehicle travels, and obtaining the trained target obstacle space position determination model, which is used for real-time prediction of the space position of the obstacle when the vehicle is actually traveling, and provides accurate and reliable obstacle information for planning and control of the vehicle. The three-dimensional reconstruction of the obstacle of the road achieved in the training process also helps to perform three-dimensional labeling of the all-around view scene, such as semantic labeling and speed labeling of the three-dimensional space.

[0107] The training method of the obstacle space position determination model provided in the embodiments of the present disclosure can determine the photometric loss of each sample image based on the depth map of the sample image, and then perform self-supervised training on the obstacle space position determination model to be trained based on the photometric loss, which can effectively avoid labeling of the label data used for supervised training, thereby greatly reducing the labor cost and time cost of labeling, improving the efficiency of obtaining the model, and the target obstacle space position determination model obtained can determine the space position of the obstacle based on the image, which helps to apply the model on a large scale in vehicles.

[0108] Figure 8 FIG. 1 is a flowchart of a training method of an obstacle space position determination model provided by another example embodiment of the present disclosure.

[0109] In some optional embodiments, the step 301 of obtaining the first image sample set and the camera poses corresponding to each sample image in the first image sample set comprises:

[0110] The step 3011 obtains sensor data collected by the vehicle, and the sensor data comprises at least one frame of original image data, inertial measurement unit (IMU) data and positioning data.

[0111] The IMU data can comprise three-axis attitude angle (or angular rate) and acceleration and other related data of the vehicle, and can be used to determine the pose change of the vehicle. The positioning data can comprise GPS data and other related positioning data, which are not limited in specific.

[0112] The step 3012 determines the camera pose corresponding to each original image in each frame of original image data based on the sensor data.

[0113] The camera pose corresponding to each original image can be obtained based on a preset positioning manner, and the preset positioning manner can be determined according to actual requirements. For example, the preset positioning can be achieved by the SLAM manner and its series of improved manners.

[0114] Step 3013, performing semantic segmentation on each original image in each frame of original image data to obtain a semantic segmentation result corresponding to each original image respectively.

[0115] The semantic segmentation can be implemented based on a pre-trained semantic segmentation model, and the specific network structure of the semantic segmentation model can be, for example, a semantic segmentation model based on Transformer and its series, which is not limited in the present disclosure. The semantic segmentation result can include a semantic type corresponding to each pixel in the original image. The semantic type can include, for example, sky, lane line, road edge, other vehicles, pedestrians, cyclists, animals, etc. Specifically, the probability of each pixel belonging to each semantic type in the original image can be predicted based on the semantic segmentation model, and the probability of each pixel can be converted into a semantic type based on a preset conversion rule, for example, the semantic type corresponding to the maximum probability of each pixel belonging to each semantic type is taken as the semantic type to which the pixel belongs.

[0116] Step 3014, filtering out objects of a preset type in each original image based on the semantic segmentation result corresponding to each original image respectively to obtain a target image corresponding to each original image respectively.

[0117] For each original image, objects of a preset type can be filtered out according to the semantic type to which each pixel of the original image belongs to obtain a filtered target image.

[0118] In some optional embodiments, the preset type can include types such as sky that cannot be represented by three-dimensional spatial features.

[0119] In some optional embodiments, a mask image for filtering out objects of a preset type can be determined according to the semantic segmentation result, and objects of a preset type in the original image can be filtered out according to the mask image. The mask image has the same resolution as the original image, the pixel value of the region of the mask image where the object of the preset type needs to be filtered out is set to 0, and the pixel value of other regions is set to 1. Multiplying the mask image with the corresponding elements of the corresponding original image can obtain a target image in which objects of a preset type are filtered out.

[0120] In some optional embodiments, the image after filtering out objects of a preset type and the semantic type of each pixel of the image can be fused, and the fusion result can be taken as a target image, that is, the target image is a semantic segmented image, and the target image includes image information after filtering out objects of a preset type and the semantic type of each pixel.

[0121] Step 3015, determining a first image sample set based on each target image.

[0122] The set of target images in which objects of a preset type are filtered out can be taken as the first image sample set.

[0123] In some optional embodiments, the target image can also be determined based on the image in which the preset type of object is filtered out and the semantic type of each pixel after the preset type of object is filtered out, and the target image is taken as the sample image. In addition to the image information, the target image also includes the semantic type of each pixel. Specifically, the image in which the preset type of object is filtered out and the semantic type of each pixel after the preset type of object is filtered out can be fused to obtain the target image. For example, the fusion manner can be splicing in the channel direction, that is, the semantic type of each pixel is taken as a single-channel image and spliced with the image in which the preset type of object is filtered out in the channel direction, and the splicing result is taken as the target image.

[0124] In step 3016, the camera pose corresponding to each original image is taken as the camera pose corresponding to each image in the first image sample set.

[0125] In the above embodiments, the target image is obtained by filtering out the preset type of object from the original image, and the camera pose corresponding to the target image is the same as that of the original image. Therefore, the camera pose of each original image is taken as the camera pose of the corresponding target image, and the target image is taken as the sample image in the first image sample set, so as to obtain the camera pose corresponding to each sample image in the first image sample set.

[0126] In some optional embodiments, for the model obtained by training the model based on the sample image in which the preset type of object is filtered out, the first image collected at each preset view angle also needs to be filtered out of the preset type of object in the actual application of the model to ensure consistency with the training process.

[0127] The embodiments of the present disclosure can obtain accurate and effective camera poses of each sample image through the preset positioning manner, and can remove the object that cannot be represented by three-dimensional space information by filtering out the preset type of object in the original image, thereby improving the dimensionality of the image features of the sample image and helping to improve the convergence speed of the model.

[0128] Figure 9 FIG. 6 is a flowchart of a method for training an obstacle space position determination model according to an example embodiment of the present disclosure.

[0129] In some optional embodiments, the step 302 of determining the occupancy grid information of each frame of image data in the first image sample set under the three-dimensional coordinate system based on the to-be-trained obstacle space position determination model comprises:

[0130] In step 3021, the feature extraction network in the to-be-trained obstacle space position determination model is used to perform feature extraction on each frame of image data to obtain the sample image features about the obstacle corresponding to each frame of image data.

[0131] Step 3022: Use the dimension transformation network in the obstacle spatial location determination model to convert the sample image features corresponding to each frame of image data into three-dimensional sample space features.

[0132] Step 3023: Use the decoding network in the obstacle spatial location determination model to decode the spatial features of each three-dimensional sample to obtain the occupied grid information corresponding to each frame of image data.

[0133] The network structure of the obstacle spatial location determination model to be trained can be found in [reference needed]. Figure 5 The network structure shown, and the specific operating principles of steps 3021 to 3023 can be found in the aforementioned embodiments, and will not be repeated here.

[0134] In some optional embodiments, step 303, which determines the depth map corresponding to each sample image in the frame image data based on the occupancy grid information corresponding to the frame image data and the camera pose corresponding to each sample image in the frame image data, includes:

[0135] Step 3031: Based on the occupied grid information corresponding to the frame image data, determine the coordinates and occupancy probability of each grid vertex in the three-dimensional coordinate system.

[0136] The occupancy information includes the occupancy probability and coordinates in a 3D coordinate system for each vertex of the occupancy grid. The coordinates and occupancy probability of each vertex in the occupancy grid can be determined based on the image data of that frame. The number of occupancy grids can be set according to actual needs. For example, it can be a preset number of occupancy grids within a certain spatial range in a 3D coordinate system, such as occupancy grids divided within a certain spatial range around a vehicle.

[0137] Step 3032: Based on the camera pose corresponding to each sample image, the coordinates of each grid vertex in the three-dimensional coordinate system, and the occupancy probability, determine the depth map corresponding to each sample image in the frame image data.

[0138] Specifically, for each sample image in each frame of image data, the depth map corresponding to the sample image can be determined based on the camera pose corresponding to the sample image, the coordinates of each grid vertex corresponding to the frame of image data in the three-dimensional coordinate system, and the occupancy probability.

[0139] In some optional embodiments, the occupancy grid information can be rendered into each preset view based on a volume rendering manner to obtain a depth map corresponding to each preset view respectively. The volume rendering manner may, for example, be a differentiable rendering manner. The depth map corresponding to each sample image can also be obtained based on a differentiable rendering tool, in combination with the camera pose of each sample image, and the coordinates and occupied probability of each grid vertex in the three-dimensional coordinate system.

[0140] The embodiments of the present disclosure help to convert each sample image to an adjacent frame and / or an adjacent view, and facilitate calculation of photometric loss, by determining the depth map of each sample image based on the occupancy grid information of a frame of image data and the camera pose of each sample image in the frame of image data.

[0141] In some optional embodiments, the determination of the depth map corresponding to each sample image in the frame of image data based on the camera pose of each sample image, the coordinates and occupied probability of each grid vertex in the three-dimensional coordinate system, in step 3032, includes:

[0142] For any sample image, each pixel of the sample image is taken as a target pixel, and a target ray is determined based on the camera pose of the sample image, the target ray being a ray from the camera optical center to the target pixel. A plurality of sampling points are determined based on the coordinates of each grid vertex in the three-dimensional coordinate system and the target ray. The occupied probability of each sampling point is determined according to the occupied probability of each grid vertex. The depth map corresponding to the sample image is determined based on the occupied probability of each sampling point.

[0143] In the above method, each pixel of the sample image is taken as a target pixel, and the camera pose of the sample image is used to determine the position of the camera optical center in the three-dimensional coordinate system, the imaging plane of the camera in the three-dimensional coordinate system, and the target ray from the camera optical center to the target pixel on the imaging plane. The target ray will enter the grid in the three-dimensional coordinate system, and the coordinates and occupied probability of any point on the target ray can be determined based on the coordinates and occupied probability of each grid vertex in the direction of the target ray. For example, the occupied probability of any point in the grid can be obtained by interpolation based on the occupied probability of each grid vertex. The interpolation can be implemented by trilinear interpolation. The occupied probability of the point on the target ray represents the voxel density at the point, i.e., the probability that an infinitesimal particle is terminated at the point. Based on a plurality of sampling points on the target ray, the depth value corresponding to the target pixel in the three-dimensional coordinate system can be determined, and the depth value of each pixel in the sample image can be obtained, i.e., the depth map corresponding to the sample image can be obtained.

[0144] The embodiment of the present disclosure helps to provide an effective depth map for photometric loss calculation by determining a depth map corresponding to the sample image according to the occupied probability of the sampling points on the rays.

[0145] In some optional embodiments, the plurality of sampling points are determined based on the coordinates of the three-dimensional coordinate system corresponding to each mesh vertex and the target ray.

[0146] The sampling range is determined based on the coordinates of the three-dimensional coordinate system corresponding to each mesh vertex, and the plurality of sampling points are obtained by sampling along the target ray within the sampling range based on a preset sampling rule.

[0147] The sampling range can be determined based on the coordinates of the mesh vertices, the camera pose corresponding to the sample image, and the visible distance range of the camera. For example, the sampling range is the proximal boundary and the distal boundary of the mesh in the three-dimensional coordinate system on the target ray. The preset sampling rule can include a sampling interval. For example, the preset sampling rule is equal-interval sampling, and the sampling interval is n meters. Then, the plurality of sampling points are obtained by sampling along the target ray within the sampling range at a sampling interval of n meters.

[0148] The embodiment of the present disclosure can improve the effectiveness of the sampling points, avoid invalid sampling points, reduce the amount of calculation, and improve the processing efficiency of model training by determining the sampling range and obtaining the plurality of sampling points based on the preset sampling rule.

[0149] In some optional embodiments, the depth map corresponding to the sample image is determined based on the occupied probability corresponding to each sampling point.

[0150] The depth value corresponding to the target pixel is determined based on the occupied probability corresponding to each sampling point, and the depth map corresponding to the sample image is determined based on the depth value corresponding to each pixel of the sample image.

[0151] The cumulative transparency of the target ray can be determined based on the occupied probability corresponding to each sampling point and the distance between adjacent sampling points, and the depth value of the pixel is determined based on the cumulative transparency.

[0152] In some optional embodiments, the depth value corresponding to the pixel can be obtained based on any implementable rendering method, such as a voxel rendering method, a piecewise approximation rendering method, etc., and the embodiment of the present disclosure is not limited in this regard.

[0153] In some optional embodiments, the photometric loss corresponding to each sample image is determined based on the depth map corresponding to each sample image in step 304.

[0154] At step 3041, the target sample image is converted to a pixel coordinate system corresponding to the adjacent frame sample image based on the depth map corresponding to the target sample image and the camera pose, the camera pose corresponding to the at least one adjacent frame sample image of the target sample image, to obtain a projection image corresponding to the target sample image.

[0155] In the depth map, the depth value of each pixel of the target sample image can be determined. For example, the distance of the three-dimensional space point corresponding to any pixel to the camera can be obtained according to the depth value of the pixel. The coordinates of the three-dimensional space point corresponding to the pixel in the camera coordinate system can be obtained according to the coordinates of the pixel in the pixel coordinate system and the distance. The extrinsic parameters of the camera can be obtained according to the camera pose. Thus, the camera coordinate system of the target sample image can be converted to the reference coordinate system based on the extrinsic parameters of the camera. Then, the camera coordinate system of the adjacent frame sample image can be converted to the pixel coordinate system of the adjacent frame sample image based on the camera pose corresponding to the adjacent frame sample image. Thus, the projection image corresponding to the target sample image can be obtained.

[0156] In some optional embodiments, for each sample image, the sample image can be projected to at least one adjacent frame to obtain at least one projection image corresponding to the adjacent frame, respectively. For example, the sample image can be projected to the previous frame and the next frame to obtain the previous frame projection image and the next frame projection image, respectively.

[0157] At step 3042, the photometric loss corresponding to the target sample image is determined based on the projection image and the adjacent frame sample image.

[0158] In the depth map, the depth value of each pixel of the target sample image can be determined. For example, the distance of the three-dimensional space point corresponding to any pixel to the camera can be obtained according to the depth value of the pixel. The coordinates of the three-dimensional space point corresponding to the pixel in the camera coordinate system can be obtained according to the coordinates of the pixel in the pixel coordinate system and the distance. The extrinsic parameters of the camera can be obtained according to the camera pose. Thus, the camera coordinate system of the target sample image can be converted to the reference coordinate system based on the extrinsic parameters of the camera. Then, the camera coordinate system of the adjacent frame sample image can be converted to the pixel coordinate system of the adjacent frame sample image based on the camera pose corresponding to the adjacent frame sample image. Thus, the projection image corresponding to the target sample image can be obtained.

[0159] In some optional embodiments, the photometric loss corresponding to the target sample image can be determined based on any implementable loss function. For example, the L1 (i.e., Mean Absolute Error, MAE) loss function, the L2 (i.e., Mean Square Error, MSE) loss function, or the like can be used to calculate the photometric loss.

[0160] In some optional embodiments, if a plurality of adjacent frame projection images corresponding to the target sample image are obtained, the photometric loss can be calculated by synthesizing the plurality of adjacent frame projection images and the corresponding adjacent frame sample images. For example, the photometric loss of the previous frame and the photometric loss of the next frame are weighted and summed to obtain the photometric loss corresponding to the target sample image.

[0161] The embodiments of the present disclosure realize the self-supervised training of the model by converting the sample image to the corresponding adjacent frame to reconstruct the adjacent frame image (i.e., the projection image), and calculating the photometric loss with the actual adjacent frame sample image, effectively improving the accuracy of the model in predicting the occupancy grid information, thereby improving the accuracy of the determined spatial position of the obstacle.

[0162] Figure 10 FIG. 4 is a flowchart of a method for training an obstacle spatial position determination model according to an example embodiment of the present disclosure.

[0163] In some optional embodiments, the converting, in step 3041, each sample image to the pixel coordinate system corresponding to the at least one adjacent frame sample image of the target sample image includes:

[0164] In step 30411, the dynamic objects in each sample image are filtered out to obtain the filtered image corresponding to each sample image.

[0165] The dynamic objects can include moving objects such as vehicles and cyclists in the sample image. The filtering of the dynamic objects can be realized in combination with the semantic segmentation result of the sample image. According to the semantic segmentation result, the object type (semantic type) to which each pixel in the sample image belongs can be determined. Based on this, the pixel value of the pixel belonging to the dynamic object is updated to a preset value to realize the filtering of the dynamic object. The preset value can be set according to actual needs, for example, set to 0, which is not limited in particular.

[0166] In step 30412, each filtered image corresponding to each sample image is taken as a target sample image, and the target sample image is converted to the pixel coordinate system corresponding to the at least one adjacent frame sample image of the target sample image based on the depth map and the camera pose corresponding to the target sample image and the camera pose corresponding to the at least one adjacent frame sample image of the target sample image to obtain the projection image corresponding to the target sample image.

[0167] The coordinate system conversion of this step can be referred to the aforementioned embodiments, which will not be repeated here.

[0168] In some optional embodiments, for a case where the sample image after filtering out the dynamic object is taken as the target sample image, when calculating the photometric loss, the photometric loss can be determined based on the projection image and the sample image of the adjacent frame after filtering out the dynamic object.

[0169] In the embodiments of the present disclosure, since the state of the dynamic object in the adjacent frame can change, it is easy to cause a large photometric loss, which is not conducive to model training. Therefore, the calculation of the photometric loss after filtering out the dynamic object can improve the accuracy and effectiveness of the photometric loss, and is helpful to the fast convergence of the model.

[0170] Figure 11 FIG. 4 is a flowchart of a method for training an obstacle spatial position determination model according to an example embodiment of the present disclosure.

[0171] In some optional embodiments, the step 304 of determining the photometric loss corresponding to each sample image based on the depth map corresponding to each sample image comprises:

[0172] The step 304a takes each sample image as a target sample image, converts the target sample image to a pixel coordinate system corresponding to the adjacent view sample image based on the depth map corresponding to the target sample image, the camera pose of the target sample image, and the camera pose corresponding to the adjacent view sample image of the target sample image, and obtains a projection image corresponding to the target sample image.

[0173] The principle of converting the sample image to the adjacent view is similar to that of converting to the adjacent frame, and will not be described here.

[0174] The step 304b determines the photometric loss corresponding to the target sample image based on the projection image and the adjacent view sample image.

[0175] For the adjacent view, the photometric loss can be calculated based on the pixels in the overlapping region of the adjacent view when calculating the photometric loss, and the non-overlapping region is not used to calculate the photometric loss, so as to ensure the accuracy and effectiveness of the photometric loss.

[0176] In some optional embodiments, similar to the adjacent frame, the target sample image can also be projected to multiple adjacent views, and the photometric loss corresponding to the target sample image can be determined by comprehensively considering the projection images of the multiple adjacent views and the corresponding adjacent view sample images.

[0177] In the embodiments of the present disclosure, the sample image is used to reconstruct the image of the adjacent view, which is used to calculate the photometric loss with the actual adjacent view sample image, to perform self-supervised training on the model, and to improve the performance of the model.

[0178] Figure 12 FIG. 4 is a flowchart of a method for training an obstacle spatial position determination model according to an example embodiment of the present disclosure.

[0179] In some optional embodiments, the determining, in step 304, the photometric loss corresponding to each sample image based on the depth map corresponding to the sample image, comprises:

[0180] In step 304A, the sample image is converted into a pixel coordinate system corresponding to the adjacent frame sample image based on the depth map corresponding to the sample image, the camera pose of the sample image, and the camera pose corresponding to the adjacent frame sample image, to obtain a first projection image corresponding to the sample image.

[0181] In some optional embodiments, for the case of conversion to the adjacent frame, the dynamic object in the sample image can be filtered out before conversion, to improve the accuracy and effectiveness of the photometric loss (first photometric loss) of the adjacent frame. The filtering principle can be referred to the foregoing embodiments, and will not be described here.

[0182] In step 304B, the first photometric loss corresponding to each sample image is determined based on the first projection image corresponding to the sample image and the adjacent frame sample image.

[0183] In step 304C, the second photometric loss corresponding to each sample image is determined based on the depth map corresponding to the sample image, the camera pose of the sample image, and the camera pose corresponding to the adjacent view sample image.

[0184] The determination of the first photometric loss and the second photometric loss can be referred to the foregoing embodiments, and will not be described here.

[0185] In step 304D, the photometric loss corresponding to each sample image is determined based on the first photometric loss and the second photometric loss corresponding to the sample image.

[0186] For each sample image, the first photometric loss and the second photometric loss can be fused according to a preset rule to obtain the photometric loss corresponding to the sample image, for example, weighted summation according to a preset weight.

[0187] The disclosed embodiments can further improve the model performance by comprehensively performing self-supervised training on the photometric loss of the adjacent frame and the photometric loss of the adjacent view, so as to effectively improve the accuracy of the predicted spatial position of the obstacle.

[0188] Figure 13 FIG. 4 is a flowchart of a method for training an obstacle spatial position determination model according to an example embodiment of the present disclosure.

[0189] In some optional embodiments, the step 304C of determining the second photometric loss corresponding to each sample image based on the depth map and the camera pose corresponding to each sample image, and the camera pose corresponding to at least one adjacent-view sample image of each sample image, comprises:

[0190] The step 304C1 of converting each sample image to a pixel coordinate system corresponding to the adjacent-view sample image based on the depth map and the camera pose corresponding to each sample image, and the camera pose corresponding to the adjacent-view sample image of each sample image, to obtain a second projection image corresponding to each sample image.

[0191] The step 304C2 of determining the second photometric loss corresponding to each sample image based on the second projection image corresponding to each sample image and the adjacent-view sample image.

[0192] Wherein, the second projection image and the second photometric loss can refer to the aforementioned embodiments of converting to an adjacent view, which will not be repeated here.

[0193] Figure 14 is a flowchart of the training method of the obstacle space position determination model provided by another exemplary embodiment of the present disclosure.

[0194] In some optional embodiments, the step 305 of determining the target obstacle space position determination model based on the photometric loss corresponding to each sample image in the first image sample set comprises:

[0195] The step 3051 of updating the network parameters of the to-be-trained obstacle space position determination model according to the photometric loss to obtain an updated to-be-trained obstacle space position determination model.

[0196] Wherein, the iteration step of the network parameters can be determined according to the photometric loss by using a preset gradient descent method, and then the network parameters are updated based on the iteration step to obtain updated network parameters, and the updated to-be-trained obstacle space position determination model (referred to as an updated model) is obtained based on the updated network parameters.

[0197] The step 3052 of taking the updated to-be-trained obstacle space position determination model as the to-be-trained obstacle space position determination model and returning to the step 302.

[0198] After updating the network parameters of the model, the updated model can be taken as the aforementioned to-be-trained obstacle space position determination model (referred to as a to-be-trained model) to continue the iterative training. The steps 302 to 304 and the steps 3051 and 3052 are repeatedly executed to realize the continuous updating and optimization of the model.

[0199] In step 3053, in response to the luminosity loss satisfying the preset condition, the to-be-trained obstacle space position determination model is determined as the target obstacle space position determination model.

[0200] In the training process of the model, it can be determined at each iteration whether the luminosity loss satisfies the preset condition. If the luminosity loss satisfies the preset condition, the training is ended, and the to-be-trained model in which the luminosity loss satisfies the preset condition can be determined as the target obstacle space position determination model.

[0201] In some optional embodiments, in the training process, it can also be determined at each iteration whether the current iteration number reaches a preset number threshold. If the current iteration number reaches the preset number threshold, the training is ended, and the final to-be-trained model is determined as the target obstacle space position determination model.

[0202] In some optional embodiments, the first image sample set can be divided into multiple batches, and the sample images of each batch are used for one iteration training. A test sample set for testing the model performance can also be divided. After the training is ended, it is determined whether the model performance reaches an expectation based on the test sample set. If the model performance reaches the expectation, an applicable target obstacle space position determination model is obtained. If the model performance does not reach the expectation, the training can be continued.

[0203] The self-supervised iterative training based on the luminosity loss can continuously optimize the model, so that the target obstacle space position determination model obtained can predict the accurate three-dimensional space positions of the obstacles around the vehicle based on the multi-view images, and the effective determination of the obstacle space positions is realized at a small cost.

[0204] In some optional examples, Figure 15 is a network structure schematic diagram of the obstacle space position determination model provided by an example embodiment of the present disclosure. As shown in Figure 15 The number of preset perspectives is 6, the feature extraction network can extract multi-scale sample image features of the sample images of each preset perspective, Figure 15Taking three scales (including scale 1, scale 2 and scale 3) as an example, the upward arrow between the sample image features of each scale represents a 2D encoder, that is, an encoder of two-dimensional image features, and the extraction of multi-scale sample image features is realized through multiple encoders. 2D to 3D represents a conversion unit in the dimension conversion network, and x1, x3 and x6 represent a stack of 1, 3 and 6 conversion units respectively. Taking x6 as an example, it represents that the sample image features of scale 3 are sequentially converted by 6 conversion units to obtain the three-dimensional sample space features corresponding to the image features of scale 3. The network structure of the 2D to 3D conversion unit includes a cross-view attention layer (Cross-view Attention), a first addition and normalization layer (Add&Norm), a feed forward network layer (Feed Forward), a second addition and normalization layer (Add&Norm) and a 3D convolution layer (3D Convolution). In the decoding network, the decoding result of the three-dimensional sample space features of scale 3 (size HxWxZ, H represents the feature height, W represents the feature width, and Z represents the number of feature channels) is up-sampled to obtain the intermediate decoding result of scale 2 (size 2Hx2Wx2Z). The downward arrow in the decoding process represents a 3D deconvolution operation, that is, the up-sampling is realized by the 3D deconvolution operation, the intermediate decoding result of scale 2 is fused with the three-dimensional sample space features of scale 2 to obtain a fusion result, the fusion result is up-sampled to obtain the intermediate decoding result of scale 1 (size 4Hx4Wx4Z), the intermediate decoding result of scale 1 is fused with the three-dimensional sample space features of scale 1, and the fusion result is up-sampled to obtain the final decoding result (size 8Hx8Wx8Z). The decoding result of each scale can obtain the occupancy grid information of the corresponding obstacle, and then the spatial position of the obstacle is obtained according to the occupancy grid information. Figure 15 The network structure shown is only an exemplary structure of an embodiment, and in actual application, the specific network structure of each part such as the feature extraction network 21, the dimension conversion network 22, the decoding network 23, etc. is not limited, as long as the corresponding functions can be realized, and is not limited to the structure described above. After obtaining the occupancy grid information of the obstacle, the camera pose of each sample image can be combined to obtain the depth map corresponding to each sample image by using the volume rendering method, and then the photometric loss corresponding to each sample image is determined based on the depth map corresponding to each sample image, the target obstacle spatial position determination model is obtained by performing self-supervised training on the obstacle spatial position determination model to be trained based on the photometric loss. Since the volume rendering method has differentiability, it is helpful for the calculation of the network parameter iteration step, thereby helping the continuous iteration and optimization of the obstacle spatial position determination model to be trained, reducing the photometric loss, and finally obtaining the target obstacle spatial position determination model.

[0205] The above-mentioned embodiments of the present disclosure can be implemented independently or in any combination without conflict, and can be set according to actual needs. The present disclosure is not limited.

[0206] The training method of any of the obstacle space position determination models provided by the embodiments of the present disclosure can be executed by any appropriate device with data processing capability, including but not limited to terminal devices and servers, etc. Alternatively, the training method of any of the obstacle space position determination models provided by the embodiments of the present disclosure can be executed by a processor, such as a processor executing the training method of any of the obstacle space position determination models mentioned by the embodiments of the present disclosure by calling corresponding instructions stored in a memory. Details are not repeated hereinafter.

[0207] Exemplary Apparatus

[0208] Figure 16 is a structural schematic diagram of an apparatus for determining the spatial position of an obstacle during vehicle driving provided by an exemplary embodiment of the present disclosure. The apparatus of this embodiment can be used to implement the corresponding method embodiments of the present disclosure for determining the spatial position of an obstacle during vehicle driving, such as the method embodiments shown in Figure 16 The apparatus includes an image acquisition module 51, a feature extraction module 52, a spatial feature determination module 53, a spatial feature decoding module 54, and a spatial position determination module 55.

[0209] The image acquisition module 51 is configured to acquire a first image captured by a camera at a preset viewing angle on the vehicle.

[0210] The feature extraction module 52 is configured to extract image features of the obstacle in the first image.

[0211] The spatial feature determination module 53 is configured to determine the three-dimensional spatial features of the obstacle based on the image features.

[0212] The spatial feature decoding module 54 is configured to decode the three-dimensional spatial features to obtain the occupancy grid information of the obstacle in the first image under a three-dimensional coordinate system; the three-dimensional coordinate system is a vehicle local coordinate system in which the vehicle is located.

[0213] The spatial position determination module 55 is configured to determine the three-dimensional spatial position of the obstacle in the vehicle local coordinate system based on the occupancy grid information.

[0214] Figure 17 is a structural schematic diagram of an apparatus for determining the spatial position of an obstacle during vehicle driving provided by another exemplary embodiment of the present disclosure.

[0215] In some optional embodiments, the feature extraction module 52 includes:

[0216] The feature extraction unit 521 is configured to perform feature extraction on the first image by using a feature extraction network in the obstacle space position determination model, to obtain image features.

[0217] In some optional embodiments, the spatial feature determination module 53 includes:

[0218] The spatial feature determination unit 531 is configured to perform dimension conversion on the image features by using a dimension conversion network in the obstacle space position determination model, to obtain three-dimensional spatial features of the obstacle.

[0219] In some optional embodiments, the spatial feature decoding module 54 includes:

[0220] The decoding unit 541 is configured to decode the three-dimensional spatial features by using a decoding network in the obstacle space position determination model, to obtain the occupancy grid information of the obstacle.

[0221] The device example embodiment of the present application has corresponding beneficial technical effects, which can be referred to the corresponding beneficial technical effects of the above-mentioned example method part, and will not be described here.

[0222] Figure 18 is a structural schematic diagram of a training device of an obstacle space position determination model provided by an example embodiment of the present application. The device of this embodiment can be used to implement the training method embodiment of the corresponding obstacle space position determination model of the present application, as shown in Figure 18 The device includes a sample acquisition module 61, a first processing module 62, a second processing module 63, a third processing module 64, and a fourth processing module 65.

[0223] The sample acquisition module 61 is configured to acquire a first image sample set and camera poses corresponding to each sample image in the first image sample set. The first image sample set includes at least one frame of image data collected by at least one camera with a preset view angle on the vehicle, and each frame of image data includes sample images of each preset view angle.

[0224] The first processing module 62 is configured to determine, based on a to-be-trained obstacle space position determination model, occupancy grid information in a three-dimensional coordinate system corresponding to each frame of image data in the first image sample set. The three-dimensional coordinate system corresponding to each frame of image data is a local coordinate system of the vehicle when the vehicle collects the frame of image data.

[0225] The second processing module 63 is configured to, for each frame of image data, determine a depth map corresponding to each sample image in the frame of image data based on the occupancy grid information corresponding to the frame of image data and the camera poses corresponding to each sample image in the frame of image data.

[0226] The third processing module 64 is configured to determine the luminosity loss corresponding to each sample image based on the depth map corresponding to the sample image.

[0227] The fourth processing module 65 is configured to determine the target obstacle spatial position determination model based on the luminosity loss corresponding to each sample image in the first image sample set.

[0228] Figure 19 FIG. 1 is a structural schematic diagram of an obstacle spatial position determination model training apparatus provided by another example embodiment of the present disclosure.

[0229] In some optional embodiments, the sample obtaining module 61 includes an obtaining unit 611, a first determining unit 612, a first processing unit 613, a second processing unit 614, and a second determining unit 615.

[0230] The obtaining unit 611 is configured to obtain sensor data collected by the vehicle, wherein the sensor data includes at least one frame of original image data, inertial measurement unit data, and positioning data.

[0231] The first determining unit 612 is configured to determine the camera pose corresponding to each original image in each frame of original image data based on the sensor data.

[0232] The first processing unit 613 is configured to perform semantic segmentation on each original image in each frame of original image data to obtain a semantic segmentation result corresponding to each original image.

[0233] The second processing unit 614 is configured to filter out objects of a preset type in each original image based on the semantic segmentation result corresponding to each original image to obtain a target image corresponding to each original image.

[0234] The second determining unit 615 is configured to determine the first image sample set based on each target image, and determine the camera pose corresponding to each original image as the camera pose corresponding to each image in the first image sample set.

[0235] In some optional embodiments, the first processing module 62 includes a third processing unit 621 and a fourth processing unit 622.

[0236] The third processing unit 621 is configured to perform feature extraction on each frame of image data by using a feature extraction network in the obstacle spatial position determination model to be trained to obtain sample image features about obstacles corresponding to each frame of image data.

[0237] The fourth processing unit 622 is configured to convert the sample image features corresponding to each frame of image data into three-dimensional sample space features by using a dimension conversion network in the obstacle spatial position determination model to be trained.

[0238] The fifth processing unit 623 is configured to decode each three-dimensional sample space feature by using a decoding network in the obstacle space position determination model to be trained, to obtain the occupancy grid information corresponding to each frame of image data respectively.

[0239] In some optional embodiments, the second processing module 63 includes a third determination unit 631 and a fourth determination unit 632.

[0240] The third determination unit 631 is configured to determine, based on the occupancy grid information corresponding to the frame of image data, coordinates in a three-dimensional coordinate system and an occupied probability corresponding to each grid vertex of the occupancy grid respectively.

[0241] The fourth determination unit 632 is configured to determine, based on the camera pose corresponding to each sample image, the coordinates in the three-dimensional coordinate system and the occupied probability corresponding to each grid vertex, a depth map corresponding to each sample image in the frame of image data.

[0242] In some optional embodiments, the fourth determination unit 632 is specifically configured to:

[0243] For any sample image, each pixel of the sample image is taken as a target pixel respectively, and based on the camera pose corresponding to the sample image, a target ray is determined, where the camera optical center shoots at the target pixel; based on the coordinates in the three-dimensional coordinate system corresponding to each grid vertex and the target ray, a plurality of sampling points are determined; based on the occupied probability corresponding to each grid vertex, the occupied probability corresponding to each sampling point is determined; and based on the occupied probability corresponding to each sampling point, a depth map corresponding to the sample image is determined.

[0244] In some optional embodiments, the fourth determination unit 632 is specifically configured to:

[0245] Based on the coordinates in the three-dimensional coordinate system corresponding to each grid vertex, a sampling range is determined; and based on a preset sampling rule, sampling is performed along the target ray within the sampling range to obtain a plurality of sampling points.

[0246] In some optional embodiments, the fourth determination unit 632 is specifically configured to:

[0247] Based on the occupied probability corresponding to each sampling point, a depth value corresponding to the target pixel is determined; and based on the depth value corresponding to each pixel of the sample image respectively, a depth map corresponding to the sample image is determined.

[0248] In some optional embodiments, the third processing module 64 includes a sixth processing unit 641 and a seventh processing unit 642.

[0249] The sixth processing unit 641 is configured to convert each sample image as a target sample image to a pixel coordinate system corresponding to a neighboring frame sample image based on a depth map and a camera pose corresponding to the target sample image and a camera pose corresponding to at least one neighboring frame sample image of the target sample image, to obtain a projection image corresponding to the target sample image.

[0250] The seventh processing unit 642 is configured to determine a photometric loss corresponding to the target sample image based on the projection image and the neighboring frame sample image.

[0251] In some optional embodiments, the sixth processing unit 641 is specifically configured to:

[0252] filter out a dynamic object in each sample image to obtain a filtered image corresponding to each sample image; and convert each sample image as a target sample image to a pixel coordinate system corresponding to a neighboring frame sample image based on a depth map and a camera pose corresponding to the target sample image and a camera pose corresponding to at least one neighboring frame sample image of the target sample image, to obtain a projection image corresponding to the target sample image.

[0253] In some optional embodiments, the third processing module 64 includes the sixth processing unit 641 and the seventh processing unit 642.

[0254] The sixth processing unit 641 is configured to convert each sample image as a target sample image to a pixel coordinate system corresponding to a neighboring view sample image based on a depth map and a camera pose corresponding to the target sample image and a camera pose corresponding to at least one neighboring view sample image of the target sample image, to obtain a projection image corresponding to the target sample image.

[0255] The seventh processing unit 642 is configured to determine a photometric loss corresponding to the target sample image based on the projection image and the neighboring view sample image.

[0256] Figure 20 FIG. 7 is a structural schematic diagram of a device for training an obstacle space position determination model according to an example embodiment of the present disclosure.

[0257] In some optional embodiments, the third processing module 64 includes an eighth processing unit 64a, a fifth determination unit 64b, a ninth processing unit 64c, and a sixth determination unit 64d.

[0258] The eighth processing unit 64a is configured to convert each sample image to a pixel coordinate system corresponding to a neighboring frame sample image based on a depth map and a camera pose corresponding to each sample image and a camera pose corresponding to at least one neighboring frame sample image of each sample image, to obtain a first projection image corresponding to each sample image.

[0259] The fifth determining unit 64b is configured to determine, based on the first projection image corresponding to each of the sample images and the adjacent frame sample image, a first photometric loss corresponding to each of the sample images.

[0260] The ninth processing unit 64c is configured to determine, based on the depth map corresponding to each of the sample images and the camera pose, and the camera pose corresponding to at least one adjacent view sample image of each of the sample images, a second photometric loss corresponding to each of the sample images.

[0261] The sixth determining unit 64d is configured to determine, based on the first photometric loss and the second photometric loss corresponding to each of the sample images, a photometric loss corresponding to each of the sample images.

[0262] In some optional embodiments, the ninth processing unit 64c is specifically configured to:

[0263] convert each of the sample images to a pixel coordinate system corresponding to the adjacent view sample image based on the depth map corresponding to each of the sample images and the camera pose, and the camera pose corresponding to at least one adjacent view sample image of each of the sample images, to obtain a second projection image corresponding to each of the sample images; and determine, based on the second projection image corresponding to each of the sample images and the adjacent view sample image, a second photometric loss corresponding to each of the sample images.

[0264] In some optional embodiments, the fourth processing module 65 includes a parameter updating unit 651, a seventh determining unit 652, and an eighth determining unit 653.

[0265] The parameter updating unit 651 is configured to update the network parameter of the to-be-trained obstacle spatial position determination model according to the photometric loss, to obtain an updated to-be-trained obstacle spatial position determination model.

[0266] The seventh determining unit 652 is configured to transmit the updated to-be-trained obstacle spatial position determination model to the first processing module 62 as the to-be-trained obstacle spatial position determination model.

[0267] The eighth determining unit 653 is configured to, in response to the photometric loss satisfying a preset condition, take the to-be-trained obstacle spatial position determination model as a target obstacle spatial position determination model.

[0268] The beneficial technical effects corresponding to the exemplary embodiments of the device can be referred to the corresponding beneficial technical effects of the above-mentioned exemplary method, which will not be described herein again.

[0269] Exemplary Electronic Device

[0270] Figure 21is a structural diagram of an electronic device provided by an embodiment of the present disclosure, comprising at least one processor 11 and a memory 12.

[0271] The processor 11 can be a central processing unit (CPU) or other forms of processing unit having data processing capability and / or instruction execution capability, and can control other components in the electronic device 10 to perform desired functions.

[0272] The memory 12 can comprise one or more computer program products, which can comprise various forms of computer readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM), cache memory, and / or the like. The non-volatile memory may, for example, include read-only memory (ROM), hard disk, flash memory, and / or the like. One or more computer program instructions can be stored on the computer readable storage medium, and the processor 11 can run the one or more computer program instructions to implement the methods and / or other desired functions of various embodiments of the present disclosure described above.

[0273] In one example, the electronic device 10 can further comprise an input device 13 and an output device 14, which are interconnected through a bus system and / or other forms of connection mechanism (not shown).

[0274] The input device 13 can further comprise, for example, a keyboard, a mouse, and / or the like.

[0275] The output device 14 can output various information to the outside, which can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and / or the like.

[0276] Of course, in order to simplify, Figure 21 Only some of the components in the electronic device 10 related to the present disclosure are shown in the figure, and components such as buses, input / output interfaces, and / or the like are omitted. In addition, the electronic device 10 can further comprise any other appropriate components according to specific application cases.

[0277] Exemplary Computer Program Product and Computer-Readable Storage Medium

[0278] In addition to the above methods and devices, embodiments of the present disclosure can also provide a computer program product comprising computer program instructions, which, when run by a processor, cause the processor to perform the steps in the methods of various embodiments of the present disclosure described in the “Exemplary Methods” section above.

[0279] A computer program product can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++, or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server. The embodiments of the present disclosure can be a computer program product, which can include a computer program tangibly embodied in a machine readable storage medium.

[0280] Moreover, an embodiment of the present disclosure can also be a computer readable storage medium having stored thereon a computer program instructing, which can make a processor execute the steps of the methods of various embodiments of the present disclosure described in the above "Exemplary Methods" section when the computer program is run by the processor.

[0281] The computer readable storage medium can be any combination of one or more computer readable media. The computer readable medium can be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the above. More specific examples (a non-exhaustive list) of the computer readable storage medium include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0282] The above describes the basic principles of the present disclosure in combination with specific embodiments, but the advantages, benefits, effects and the like mentioned in the present disclosure are only examples and are not limiting, and it cannot be considered that each embodiment of the present disclosure must have them. In addition, the above specific details are only for the purpose of example and for the purpose of understanding, and are not limiting, and the above details do not limit the present disclosure to be necessarily implemented with the above specific details.

[0283] Those skilled in the art can make various modifications and variations to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is also intended to include these modifications and variations.

Claims

1. A method for determining the spatial location of an obstacle during vehicle driving, comprising: Acquire the first image captured by a camera with a preset viewing angle on the vehicle; Extract image features of obstacles from the first image; The three-dimensional spatial features of the obstacle are determined based on the image features; The three-dimensional spatial features are decoded to obtain the grid information occupied by the obstacle in the first image in a three-dimensional coordinate system; the three-dimensional coordinate system is the vehicle's local coordinate system. Based on the occupancy grid information, the three-dimensional spatial position of the obstacle in the vehicle's local coordinate system is determined; The step of determining the three-dimensional spatial features of the obstacle based on the image features includes: The image features are transformed using a dimension transformation network in the obstacle spatial location determination model to obtain the three-dimensional spatial features of the obstacle. The obstacle spatial location determination model is obtained in the following way: Based on the obstacle spatial location determination model to be trained, the occupied grid information in the three-dimensional coordinate system corresponding to each frame of image data in the first image sample set is determined; Based on the occupied grid information corresponding to each frame of image data and the camera pose corresponding to the sample image in each frame of image data, the depth map corresponding to the sample image in each frame of image data is determined. Based on the depth map corresponding to the sample image, determine the photometric loss corresponding to the sample image; Based on the photometric loss corresponding to the sample image, a target obstacle spatial location determination model is determined. The depth map corresponding to the sample image in any frame of image data is obtained in the following way: Based on the occupied grid information corresponding to the frame image data, determine the coordinates and occupancy probability of each grid vertex in the three-dimensional coordinate system. Based on the camera pose corresponding to the sample image, the coordinates of each of the mesh vertices in the three-dimensional coordinate system, and the probability of occupancy, the depth map corresponding to the sample image is determined.

2. The method according to claim 1, wherein, The step of extracting image features about obstacles from the first image includes: The first image is used to extract features by utilizing the feature extraction network in the obstacle spatial location determination model, thereby obtaining the image features.

3. The method according to claim 1, wherein, Decoding the three-dimensional spatial features to obtain the grid information occupied by the obstacle in the three-dimensional coordinate system in the first image includes: The three-dimensional spatial features are decoded using the decoding network in the obstacle spatial location determination model to obtain the occupancy grid information of the obstacle.

4. A training method for an obstacle spatial location determination model, comprising: A first image sample set and the camera poses corresponding to each sample image in the first image sample set are obtained. The first image sample set includes at least one frame of image data collected by at least one camera with a preset view on the vehicle. Each frame of image data includes sample images from each preset view. Based on the obstacle spatial location determination model to be trained, the occupied grid information in the three-dimensional coordinate system corresponding to each frame of image data in the first image sample set is determined; the three-dimensional coordinate system corresponding to each frame of image data is the vehicle local coordinate system where the vehicle was when the frame of image data was acquired; For each frame of image data, based on the occupied grid information corresponding to the frame of image data and the camera pose corresponding to each sample image in the frame of image data, the depth map corresponding to each sample image in the frame of image data is determined. Based on the depth map corresponding to each of the sample images, determine the photometric loss corresponding to each of the sample images; Based on the photometric loss corresponding to each of the sample images in the first image sample set, a target obstacle spatial location determination model is determined. The step of determining the depth map corresponding to each sample image in the frame of image data based on the occupied grid information corresponding to the frame of image data and the camera pose corresponding to each sample image in the frame of image data includes: Based on the occupied grid information corresponding to the frame image data, determine the coordinates and occupancy probability of each grid vertex in the three-dimensional coordinate system; Based on the camera pose corresponding to each of the sample images, the coordinates of each of the mesh vertices in the three-dimensional coordinate system, and the probability of occupancy, the depth map corresponding to each of the sample images in the frame image data is determined.

5. The method according to claim 4, wherein, The step of determining the depth map corresponding to each sample image in the frame image data based on the camera pose corresponding to each sample image, the coordinates of each mesh vertex in the three-dimensional coordinate system, and the occupancy probability includes: For any of the sample images, each pixel of the sample image is taken as a target pixel, and based on the camera pose corresponding to the sample image, a target ray from the camera optical center to the target pixel is determined; Based on the coordinates of each of the mesh vertices in the three-dimensional coordinate system and the target ray, multiple sampling points are determined; Based on the occupancy probability of each of the grid vertices, determine the occupancy probability of each of the sampling points. Based on the occupancy probability of each sampling point, the depth map corresponding to the sample image is determined.

6. The method according to claim 5, wherein, The determination of multiple sampling points based on the coordinates of each of the mesh vertices in the three-dimensional coordinate system and the target ray includes: The sampling range is determined based on the coordinates of each of the mesh vertices in the three-dimensional coordinate system. Based on preset sampling rules, sampling is performed along the target ray within the sampling range to obtain multiple sampling points.

7. The method according to claim 5, wherein, The step of determining the depth map corresponding to the sample image based on the occupancy probability of each of the sampling points includes: Based on the occupancy probability of each sampling point, the depth value corresponding to the target pixel is determined; Based on the depth values ​​corresponding to each pixel of the sample image, the depth map corresponding to the sample image is determined.

8. The method according to claim 4, wherein, The step of determining the photometric loss corresponding to each of the sample images based on the depth map corresponding to each of the sample images includes: Each of the sample images is taken as a target sample image. Based on the depth map and camera pose corresponding to the target sample image and the camera pose corresponding to at least one adjacent frame sample image of the target sample image, the target sample image is transformed to the pixel coordinate system corresponding to the adjacent frame sample image to obtain the projection image corresponding to the target sample image. Based on the projected image and the adjacent frame sample images, the photometric loss corresponding to the target sample image is determined.

9. The method according to claim 8, wherein, Also includes: Dynamic objects are filtered out from each of the sample images to obtain the filtered images corresponding to each of the sample images; The step of using each of the sample images as a target sample image includes: The filtered images corresponding to each of the sample images are respectively used as the target sample images.

10. The method according to claim 4, wherein, The step of determining the photometric loss corresponding to each of the sample images based on the depth map corresponding to each of the sample images includes: Each of the sample images is taken as a target sample image. Based on the depth map and camera pose corresponding to the target sample image and the camera pose corresponding to at least one adjacent view sample image of the target sample image, the target sample image is transformed to the pixel coordinate system corresponding to the adjacent view sample image to obtain the projection image corresponding to the target sample image. Based on the projected image and the adjacent viewpoint sample images, the photometric loss corresponding to the target sample image is determined.

11. The method according to claim 4, wherein, The step of determining the photometric loss corresponding to each of the sample images based on the depth map corresponding to each of the sample images includes: Based on the depth map and camera pose corresponding to each of the sample images, and the camera pose corresponding to at least one adjacent frame sample image of each of the sample images, each of the sample images is transformed to the pixel coordinate system corresponding to the adjacent frame sample image to obtain the first projection image corresponding to each of the sample images. Based on the first projected image and the adjacent frame sample image corresponding to each of the sample images, the first photometric loss corresponding to each of the sample images is determined; Based on the depth map and camera pose corresponding to each of the sample images, and the camera pose corresponding to at least one adjacent view sample image of each of the sample images, the second photometric loss corresponding to each of the sample images is determined. Based on the first photometric loss and the second photometric loss corresponding to each of the sample images, the photometric loss corresponding to each of the sample images is determined.

12. The method according to claim 11, wherein, The step of determining the second photometric loss corresponding to each of the sample images based on the depth map and camera pose corresponding to each of the sample images, and the camera pose corresponding to at least one adjacent view sample image of each of the sample images, includes: Based on the depth map and camera pose corresponding to each of the sample images, and the camera pose corresponding to at least one adjacent view sample image of each of the sample images, each of the sample images is transformed to the pixel coordinate system corresponding to the adjacent view sample image to obtain the second projection image corresponding to each of the sample images. Based on the second projected image and the adjacent view sample image corresponding to each of the sample images, the second photometric loss corresponding to each of the sample images is determined.

13. The method according to claim 4, wherein, The target obstacle spatial location determination model, based on the photometric loss corresponding to each of the sample images in the first image sample set, includes: The network parameters of the obstacle spatial location determination model to be trained are updated based on the photometric loss to obtain the updated obstacle spatial location determination model to be trained. The updated obstacle spatial location determination model is used as the obstacle spatial location determination model to be trained, and the step of determining the occupied grid information in the three-dimensional coordinate system corresponding to each frame image data of the first image sample set based on the obstacle spatial location determination model to be trained is repeated. In response to the light loss meeting a preset condition, the obstacle spatial location determination model to be trained is used as the target obstacle spatial location determination model.

14. The method according to claim 4, wherein, The step of obtaining the first image sample set and the camera pose corresponding to each sample image in the first image sample set includes: Acquire sensor data collected by the vehicle, the sensor data including at least one frame of original image data, inertial measurement unit data and positioning data; Based on the sensor data, determine the camera pose corresponding to each original image in each frame of original image data; Semantic segmentation is performed on each original image in each frame of original image data to obtain the semantic segmentation results corresponding to each original image. Based on the semantic segmentation results corresponding to each original image, objects of a preset type in each original image are filtered out to obtain the target images corresponding to each original image. Based on each of the target images, the first image sample set is determined; The camera poses corresponding to each original image are used as the camera poses corresponding to each image in the first image sample set.

15. The method according to claim 4, wherein, The obstacle spatial location determination model based on the object to be trained determines the grid occupancy information in the three-dimensional coordinate system corresponding to each frame of the first image sample set, including: The feature extraction network in the obstacle spatial location determination model is used to extract features from each frame of image data to obtain sample image features of the obstacle corresponding to each frame of image data. The dimension transformation network in the obstacle spatial location determination model is used to convert the sample image features corresponding to each frame of image data into three-dimensional sample space features. The decoding network in the obstacle spatial location determination model is used to decode the spatial features of each three-dimensional sample to obtain the occupied grid information corresponding to each frame of image data.

16. A device for determining the spatial position of an obstacle during vehicle driving, comprising: The image acquisition module is used to acquire the first image captured by a camera with a preset viewing angle on the vehicle; The feature extraction module is used to extract image features of obstacles from the first image; A spatial feature determination module is used to determine the three-dimensional spatial features of the obstacle based on the image features; A spatial feature decoding module is used to decode the three-dimensional spatial features to obtain the grid information occupied by the obstacle in the first image in a three-dimensional coordinate system; the three-dimensional coordinate system is the vehicle local coordinate system where the vehicle is located. A spatial location determination module is used to determine the three-dimensional spatial location of the obstacle in the vehicle's local coordinate system based on the occupied grid information. The spatial feature determination module includes: The spatial feature determination unit is used to perform dimensional transformation on the image features using the dimensional transformation network in the obstacle spatial location determination model to obtain the three-dimensional spatial features of the obstacle. The obstacle spatial location determination model is obtained in the following way: Based on the obstacle spatial location determination model to be trained, the occupied grid information in the three-dimensional coordinate system corresponding to each frame of image data in the first image sample set is determined; Based on the occupied grid information corresponding to each frame of image data and the camera pose corresponding to the sample image in each frame of image data, the depth map corresponding to the sample image in each frame of image data is determined. Based on the depth map corresponding to the sample image, determine the photometric loss corresponding to the sample image; Based on the photometric loss corresponding to the sample image, a target obstacle spatial location determination model is determined. The depth map corresponding to the sample image in any frame of image data is obtained in the following way: Based on the occupied grid information corresponding to the frame image data, determine the coordinates and occupancy probability of each grid vertex in the three-dimensional coordinate system. Based on the camera pose corresponding to the sample image, the coordinates of each of the mesh vertices in the three-dimensional coordinate system, and the probability of occupancy, the depth map corresponding to the sample image is determined.

17. A training device for an obstacle spatial location determination model, comprising: The sample acquisition module is used to acquire a first image sample set and the camera pose corresponding to each sample image in the first image sample set. The first image sample set includes at least one frame of image data collected by at least one camera with a preset view on the vehicle. Each frame of image data includes sample images from each preset view. The first processing module is used to determine the occupied grid information in the three-dimensional coordinate system corresponding to each frame of image data in the first image sample set based on the model for determining the spatial position of the obstacle to be trained; the three-dimensional coordinate system corresponding to each frame of image data is the vehicle local coordinate system where the vehicle was when the frame of image data was acquired; The second processing module is used to determine the depth map corresponding to each sample image in each frame of image data based on the occupied grid information corresponding to the frame of image data and the camera pose corresponding to each sample image in the frame of image data. The third processing module is used to determine the photometric loss corresponding to each of the sample images based on the depth map corresponding to each of the sample images. The fourth processing module is used to determine the spatial location determination model of the target obstacle based on the photometric loss corresponding to each of the sample images in the first image sample set. The second processing module includes: The third determining unit is used to determine the coordinates and occupancy probability of each grid vertex of the occupied grid in the three-dimensional coordinate system based on the occupied grid information corresponding to the frame image data. The fourth determining unit is used to determine the depth map corresponding to each of the sample images in the frame image data based on the camera pose corresponding to each of the sample images, the coordinates of each of the mesh vertices in the three-dimensional coordinate system, and the occupancy probability.

18. A computer-readable storage medium storing a computer program for performing the method for determining the spatial location of an obstacle during vehicle driving as described in any one of claims 1-3, or for performing the training method for an obstacle spatial location determination model as described in any one of claims 4-15.

19. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the method for determining the spatial position of an obstacle during vehicle driving as described in any one of claims 1-3, or to implement the training method for the obstacle spatial position determination model as described in any one of claims 4-15.

Images