Unmanned driving training data generation method, device and system and cloud platform

Abstract

The invention provides an unmanned driving training data generation method, device and system and a cloud platform, and the method comprises the steps: obtaining oblique photography data obtained by carrying out the oblique photography of a real scene, and constructing a mesh model of a static simulation scene according to the oblique photography data; obtaining a material model corresponding to a preset weather effect, and inputting the surface patch texture map and the surface patch normal of the mesh model into the material model to obtain a static simulation scene model with the preset weather effect; in a three-dimensional space region set by the simulator, acquiring sensing data acquired in a process of moving the simulated unmanned vehicle according to a preset track in a static simulation scene by utilizing the obtained simulation scene model, the unmanned vehicle model and a sensor model of a sensor configured by the unmanned vehicle; and obtaining target unmanned driving perception training data based on the perception data. According to the method, the long-tail scene is increased, and the data volume under different meteorological conditions is increased, so that the target unmanned driving perception training data is richer.

Description

Technical Field

[0001] The present disclosure relates to the field of autonomous driving, and in particular to a method, device, system and cloud platform for generating autonomous driving training data. Background Art

[0002] Existing autonomous driving perception training requires a large amount of training data. The existing methods of obtaining training data include directly collecting real data and synthesizing training data. However, the number of long-tail scenes in real data is small, which is difficult to meet the training needs; and the generated synthetic data has low fidelity and is difficult to meet the needs of high-precision perception training. The training data obtained by the above two methods also have the problems of low fidelity and small amount of data under different meteorological conditions. In order to solve the above problems, a new training data generation method is needed. Summary of the invention

[0003] The embodiments of the present disclosure provide a method, device, system and cloud platform for generating unmanned driving training data, so as to solve the problems in existing training data acquisition methods, such as small number of long-tail scenes, low realism, and small amount of data under different meteorological conditions.

[0004] Based on the above problems, in a first aspect, a method for generating unmanned driving training data is provided, comprising:

[0005] Obtaining oblique photography data obtained by oblique photography of a real scene, and constructing a mesh model of a static simulation scene according to the oblique photography data;

[0006] Obtain a material model corresponding to a preset weather effect, and input the surface texture map and surface normal of the mesh model into the material model to obtain a static simulation scene model with the preset weather effect;

[0007] In the three-dimensional space area set by the simulator, the simulation scene model with the preset weather effect, the unmanned vehicle model and the sensor model of the sensor configured by the unmanned vehicle are used to obtain the perception data collected by the simulation sensor of the simulated unmanned vehicle during the movement of the simulated unmanned vehicle according to the preset trajectory in the static simulation scene;

[0008] Based on the perception data, target unmanned driving perception training data is obtained.

[0009] In combination with the first aspect, in a possible implementation, a material model corresponding to a preset weather effect is obtained, and a patch texture map and a patch normal of the mesh model are input into the material model to obtain a static simulation scene model with the preset weather effect, including:

[0010] Get the material model corresponding to the preset static weather effect;

[0011] Inputting the patch texture map and patch normal of the mesh model into the material model of the preset static weather effect to obtain a static simulation scene model with the preset static weather effect;

[0012] The method further comprises:

[0013] Based on the preset static weather effect, particle special effects are used to add dynamic weather effects to the static simulation scene model with the preset static weather effect.

[0014] In combination with the first aspect, in a possible implementation, the patch texture map and the patch normal of the mesh model are input into the material model of the preset static weather effect to obtain a static simulation scene model with the preset static weather effect, including:

[0015] The mesh model is divided into regions according to the preset static weather effect, and corresponding effect processing is performed on different regions to obtain a static simulation scene model with the preset static weather effect; the effect includes at least one of the following: roughness, reflectivity and artistic effect.

[0016] In combination with the first aspect, in a possible implementation, the preset static weather effect includes: a precipitation weather effect; dividing the mesh model into regions according to the preset static weather effect, and performing corresponding effect processing on different regions to obtain a static simulation scene model with the preset static weather effect, including:

[0017] The area where the normal of the patch is approximately perpendicular to the horizontal plane is determined as the precipitation pit surface generation area;

[0018] Using a noise algorithm to set the size and shape of the precipitation pit surface for the precipitation pit surface generation area, obtaining a first texture of the precipitation pit surface, and adjusting the reflection coefficient of the first texture to a target reflection coefficient to obtain an adjusted first texture;

[0019] For the area outside the precipitation pit surface generation area, adjusting at least one of the roughness, reflectivity and color corresponding to the precipitation weather effect to generate a second map; and

[0020] The first map and the second map are used to cover the corresponding positions of the mesh model to obtain a static simulation scene model with precipitation weather effects.

[0021] In combination with the first aspect, in a possible implementation, based on the preset static weather effect, particle effects are used to add a dynamic weather effect to a static simulation scene model having the preset static weather effect, including:

[0022] Determining a dynamic weather effect display area in the three-dimensional space area;

[0023] Generating a preset number of three-dimensional coordinate points representing dynamic particles corresponding to the preset static weather effect in the display area;

[0024] Acquire a target map of the dynamic particle, determine the rate of change of the coordinate value of the coordinate representing the moving direction and moving speed in the three-dimensional coordinates of the dynamic particle, and add the target map and the rate of change of the coordinate value to the static simulation scene model to achieve the addition of the dynamic weather effect;

[0025] Among them, the display area includes a preset area in the three-dimensional space area, or an area through which the simulated laser rays emitted by the lidar model in the sensor model pass; the preset number and the rate of change of the coordinate value are determined based on the weather level corresponding to the dynamic weather effect.

[0026] In combination with the first aspect, in a possible implementation manner, the sensor model includes: a visible light camera model and / or a laser radar model;

[0027] Among them, the visible light camera model includes: a medium-focus lens simulated based on a physical camera model, a simulated telephoto lens, and a fisheye camera lens simulated by superimposing multiple physical camera model field of view textures based on a correction algorithm; the lidar model is used to render the static simulation scene model in depth to form a depth map based on a physical camera model with multiple preset field of view angles, so as to simulate the information in the depth map as point cloud data sampled by the lidar corresponding to the ray angle.

[0028] In combination with the first aspect, in a possible implementation manner, before constructing a mesh model of a static simulation scene according to the oblique photography data, the method further includes:

[0029] Preprocessing the oblique photography data includes at least one of the following:

[0030] Removing noise and spots introduced during the image data acquisition process;

[0031] Crop the edge of the image data to eliminate the portion with low edge resolution;

[0032] Use the preset correction algorithm to correct the image data distortion;

[0033] Detect shadows and reflections of image data using a preset image processing technology, and process the shadows and reflections using a preset shadow removal algorithm;

[0034] and / or,

[0035] After constructing the mesh model of the static simulation scene according to the oblique photography data, the method further includes:

[0036] Adjusting the mesh model of the generated static simulation scene includes at least one of the following:

[0037] Replace the attached scene model with the art model;

[0038] Correct the parts of the scene model that are inconsistent with the actual scene;

[0039] Standardizing road sign information, replacing the road sign information with standard indication information;

[0040] Add modeling details and refine the road surface to the ideal state.

[0041] In a second aspect, a device for generating unmanned driving training data is provided, comprising:

[0042] An acquisition module is used to acquire oblique photography data obtained by oblique photography of a real scene, and to construct a mesh model of a static simulation scene according to the oblique photography data; to acquire a material model corresponding to a preset weather effect, and to input a patch texture map and a patch normal of the mesh model into the material model to obtain a static simulation scene model with a preset weather effect;

[0043] A simulation module, for obtaining, in a three-dimensional space area set by the simulator, perception data collected by the simulated unmanned vehicle through the simulation sensor during the movement of the simulated unmanned vehicle along the preset trajectory in the static simulation scene, using the simulation scene model with the preset weather effect, the unmanned vehicle model, and the sensor model of the sensor configured for the unmanned vehicle;

[0044] The training data generation module is used to obtain target unmanned driving perception training data based on the perception data.

[0045] In a third aspect, a cloud platform is provided, comprising: an unmanned driving training data generating device as described in the second aspect.

[0046] In a fourth aspect, a system for generating unmanned driving training data is provided, comprising: a cloud platform and at least one oblique photography device;

[0047] The at least one oblique photography device is used to perform oblique photography of a real scene to obtain oblique photography data;

[0048] The cloud platform is used to construct a grid mesh model of a static simulation scene based on the oblique photography data; obtain a material model corresponding to a preset weather effect, and input the face texture map and face normal of the mesh model into the material model to obtain a static simulation scene model with a preset weather effect; in a three-dimensional space area set by a simulator, use the simulation scene model with the preset weather effect, as well as an unmanned vehicle model and a sensor model of a sensor configured for the unmanned vehicle to obtain perception data collected by the simulated unmanned vehicle through the simulation sensor during the movement of the simulated unmanned vehicle along a preset trajectory in the static simulation scene; based on the perception data, obtain target unmanned driving perception training data.

[0049] The beneficial effects of the embodiments of the present disclosure include:

[0050] The disclosed embodiments provide a method, device, system and cloud platform for generating unmanned driving training data, including: obtaining oblique photography data obtained by oblique photography of a real scene, and constructing a grid mesh model of a static simulation scene based on the oblique photography data; obtaining a material model corresponding to a preset weather effect, and inputting the face texture map and face normal of the mesh model into the material model to obtain a static simulation scene model with a preset weather effect; in a three-dimensional space area set by a simulator, obtaining perception data collected by the simulated unmanned vehicle through a simulation sensor during the movement of the simulated unmanned vehicle along a preset trajectory in the static simulation scene using the simulation scene model with the preset weather effect, as well as an unmanned vehicle model and a sensor model of a sensor configured for the unmanned vehicle; and obtaining target unmanned driving perception training data based on the perception data. The unmanned driving training data generation method in the real-time example disclosed in the present invention applies the scene data obtained by oblique photography to construct a simulation model, which is closer to the real scene and has a higher degree of realism than the model constructed by synthetic data. In the simulation scene model with weather effects constructed in the three-dimensional space area of ​​the simulator, driving scenes can be freely designed and different weather conditions can be adjusted to simulate different driving scenarios including long-tail scenes. This not only increases the long-tail scenes, but also increases the amount of data under different meteorological conditions, making the generated target unmanned driving perception training data richer. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A flow chart of a method for generating unmanned driving training data provided by an embodiment of the present disclosure;

[0052] Figure 2 A schematic diagram of a laser radar scanning range provided in an embodiment of the present disclosure;

[0053] Figure 3 A schematic diagram of a simulated fisheye camera provided in an embodiment of the present disclosure;

[0054] Figure 4 A schematic diagram of a simulated laser radar provided in an embodiment of the present disclosure;

[0055] Figure 5 A schematic diagram of a method for generating unmanned driving training data provided by an embodiment of the present disclosure;

[0056] Figure 6 A schematic diagram of the structure of an unmanned driving training data generating device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0057] The embodiments of the present disclosure provide a method, device, system and cloud platform for generating unmanned driving training data. The preferred embodiments of the present disclosure are described below in conjunction with the drawings of the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present disclosure, and are not used to limit the present disclosure. In addition, the embodiments and features in the embodiments of the present application can be combined with each other if there is no conflict.

[0058] The present disclosure provides a method for generating unmanned driving training data. Figure 1 As shown, including:

[0059] S101, obtaining oblique photography data obtained by oblique photography of a real scene, and constructing a mesh model of a static simulation scene according to the oblique photography data;

[0060] S102, obtaining a material model corresponding to a preset weather effect, and inputting a surface texture map and a surface normal of a mesh model into the material model to obtain a static simulation scene model with the preset weather effect;

[0061] S103, in a three-dimensional space area set by the simulator, using a simulation scene model with preset weather effects, an unmanned vehicle model, and a sensor model of a sensor configured for the unmanned vehicle, to obtain perception data collected by the simulated unmanned vehicle through the simulation sensor during the movement of the simulated unmanned vehicle along a preset trajectory in a static simulation scene;

[0062] S104: Based on the perception data, obtain target unmanned driving perception training data.

[0063] In the disclosed embodiment, in order to make the perception training data closer to the real scene, the real scene can be obliquely photographed by oblique photography equipment to obtain oblique photography data, and modeling can be performed based on the oblique photography data of the real scene. During implementation, a drone can be equipped with multiple cameras to fly over the target area along a preset path. The cameras continuously photograph the target area to obtain multi-angle photographic images of the target area. The working time of the drone is generally selected during sunny and cloudless daytime to reduce the impact of shadows and bad weather on drone photography.

[0064] Optionally, the obtained oblique photography image can be preprocessed, and the preprocessed oblique photography image can be registered. First, the feature points of the photographic image are detected using an image feature detection algorithm, and the image feature detection algorithm includes: Scale Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), etc. Taking the SIFT algorithm as an example, the oblique photography image is first filtered by a Gaussian difference function to obtain a Gaussian difference (DoG) map, and the pixels in the DoG map are detected. If the pixel value of a certain pixel is the maximum or minimum value of all adjacent pixels, then this pixel is marked as a feature point, and all feature points in the oblique photography image are marked in this way. With the obtained feature point as the center, the corresponding angle and amplitude of the pixels in the surrounding area are calculated to obtain the main direction of the feature point, and the position, scale and direction of the feature point are used to establish a description of the feature point.

[0065] The feature points with consistent descriptions in different oblique photography images are associated using the similarity criterion, so that a correspondence relationship based on the feature objects is established between the images. The plane coordinates and elevation three-dimensional coordinates of the feature points are calculated using the correspondence relationship of the feature objects in the image and the location information of the drone during oblique photography. The feature points are marked in the three-dimensional coordinate system to form a point cloud with coordinate data. A continuous irregular triangulated network is constructed using the points in the point cloud as vertices, and each vertex is used as a feature point to match the corresponding point of the oblique photography image. The corresponding oblique photography image is cut into multiple triangular facets according to the shape of the triangular network, and the triangular facets are attached to the corresponding triangular network as texture maps to form a mesh model.

[0066] A material library is established, which contains preset material models and corresponding parameters corresponding to different weather effects. The texture map of the mesh model and the normal of the texture map in the mesh model are imported into the material model. The parameters are adjusted to output the texture map that meets the corresponding weather effect, and the map is then attached to the corresponding position of the mesh model to replace the original map, so as to obtain a static simulation scene model with preset weather effects.

[0067] In a static simulation scene model with preset weather effects established using real-scene oblique photography data, the perception data collected by the simulated unmanned vehicle through the simulated sensors is obtained through the unmanned vehicle model and the sensor model of the sensors configured by the unmanned vehicle, and target unmanned driving perception training data is further generated. This allows for flexibly configuring long-tail scenes with higher realism, increases the amount of data under different meteorological conditions, and obtains richer target unmanned driving perception training data.

[0068] In another embodiment provided by the present disclosure, the above step S102 "obtaining a material model corresponding to a preset weather effect, inputting the surface texture map and surface normal of the mesh model into the material model to obtain a static simulation scene model with a preset weather effect" includes the following steps:

[0069] Step 1: Obtain the material model corresponding to the preset static weather effect;

[0070] Step 2: Input the surface texture map and surface normal of the mesh model into the material model of the preset static weather effect to obtain a static simulation scene model with the preset static weather effect;

[0071] The method further comprises the steps of:

[0072] Step 3: Based on the preset static weather effect, use particle effects to add dynamic weather effects to the static simulation scene model with the preset static weather effect.

[0073] In the embodiment of the present disclosure, a material library with material models of different weather effects is preset in the simulator. The patch texture map and patch normal of the mesh model are input into the corresponding material model of the material library, and the corresponding parameters are adjusted so that the patch texture map obtains the corresponding weather effect.

[0074] The obtained patch texture map corresponding to the weather effect is input into the mesh model to replace the original patch texture map, thereby obtaining a static simulation scene model with static weather effects.

[0075] In the disclosed embodiment, in order to make the simulation scene more realistic, particle effects may also be used to add dynamic weather effects with corresponding weather effects to the static simulation scene model.

[0076] In another embodiment provided by the present disclosure, in the above step 2, “inputting the surface texture map and surface normal of the mesh model into the material model of the preset static weather effect to obtain a static simulation scene model with the preset static weather effect” includes the following steps:

[0077] Step 1: Divide the mesh model into regions according to preset static weather effects, and perform corresponding effect processing on different regions to obtain a static simulation scene model with preset static weather effects; the effects include at least one of the following: roughness, reflectivity, and artistic effects.

[0078] In the disclosed embodiment, a mesh model is identified by a recognition algorithm, and different regions are identified, including road surface, grass, land, etc. Different material models are used to process the patch texture map of different regions to obtain a patch texture map with corresponding weather effects.

[0079] Taking the target effect of rainy road surface as an example, part of the road surface patch texture map is input into the material library, and the preset material model of rainy road surface in the material library is combined with the patch texture map. The transparency of the material model is improved to make the patch texture map present a wet effect, the noise of the material model is reduced to match the visual effect of the lower roughness of the wet road surface, and the reflectivity of the material model is adjusted to match the visual effect of the wet road surface.

[0080] In another embodiment provided by the present disclosure, the preset static weather effect includes: precipitation weather effect; then the above step 1 "dividing the mesh model into regions according to the preset static weather effect, and performing corresponding effect processing on different regions to obtain a static simulation scene model with the preset static weather effect" includes the following steps:

[0081] Step 1: determine the area where the normal line of the surface is approximately perpendicular to the horizontal plane as the precipitation pit surface generation area;

[0082] Step 2: Use a noise algorithm to set the size and shape of the precipitation pit surface for the precipitation pit surface generation area, obtain a first map of the precipitation pit surface, and adjust the reflection coefficient of the first map to the target reflection coefficient to obtain the adjusted first map;

[0083] Step 3: For the area outside the precipitation pit surface generation area, adjust at least one of the roughness, reflectivity and color corresponding to the precipitation weather effect to generate a second map; and

[0084] Step 4: Use the first map and the second map to cover the corresponding positions of the mesh model to obtain a static simulation scene model with precipitation weather effects.

[0085] In the disclosed embodiment, scene changes caused by precipitation are added to the static simulation scene model, including: accumulated water, accumulated snow, etc.

[0086] During implementation, the average value of the three-dimensional coordinate height value of the point cloud corresponding to the identified road surface area can be calculated, and the area below this average value and approximately perpendicular to the horizontal plane can be determined as the precipitation pit surface generation area, and the noise algorithm can be used to generate objects in the precipitation pit surface. Taking the use of the Perlin noise algorithm to increase water accumulation as an example, multiple vertices are defined in the precipitation pit surface, and each vertex contains a vector. The vertices are connected to form a three-dimensional network, and the gradient vectors of these vertices are randomly changed through hash operations to form a distorted effect to simulate the water surface. Add the transparent effect map as the first map to the outer surface of the generated three-dimensional network. The transparent effect map adds a dynamic reflection coefficient calculation formula in the physical camera rendering channel to calculate the mixed reflection and refraction effects.

[0087] The texture around the precipitation pit can also be modified according to the corresponding weather effects. Taking the road surface pit as an example, the wetness of the non-water area in the pit and the area around the water pit should be greater than the wetness of the general road surface. The transparency, noise, reflectivity and other parameters of the material model combined with the patch texture map of this part of the area can be adjusted on the basis of the material model combined with the general road surface to reduce the transparency of the material model and reduce the noise. Combine this part of the patch texture map with the adjusted material model to form a second map.

[0088] The first texture is added to the outer surface of the three-dimensional grid increased by the noise algorithm, and the second texture replaces the texture at the original position, so as to obtain a static simulation scene model with precipitation weather effect.

[0089] In another embodiment provided by the present disclosure, the above step 3 "based on the preset static weather effect, using particle special effects to add dynamic weather effects to the static simulation scene model with the preset static weather effect" includes the following steps:

[0090] Step 1, determining a dynamic weather effect display area in a three-dimensional space area;

[0091] Step 2: generating a preset number of three-dimensional coordinate points representing dynamic particles corresponding to the preset static weather effect in the display area;

[0092] Step 3, obtaining the target map of the dynamic particles, and determining the coordinate value change rate of the coordinates representing the movement direction and movement speed in the three-dimensional coordinates of the dynamic particles, and adding the target map and the coordinate value change rate to the static simulation scene model to achieve the addition of dynamic weather effects;

[0093] Among them, the display area includes a preset area in the three-dimensional space area, or an area through which the simulated laser rays emitted by the lidar model in the sensor model pass; the preset number and the rate of change of the coordinate value are determined based on the weather level corresponding to the dynamic weather effect.

[0094] In the disclosed embodiment, the display area of ​​the dynamic weather effect can be a preset area of ​​the entire three-dimensional simulated space area; or it can be an area where the simulated laser rays emitted by the laser radar model pass through, and this area is determined by the scanning area of ​​the laser radar corresponding to the laser radar model. Figure 2As shown, the scanning area 2 of the laser radar 1 is determined by the vertical scanning area 4 and the horizontal scanning area 3. The vertical scanning area 4 of the laser radar 1 is a sector with the laser radar 1 as the center, the maximum detection distance of the laser radar 1 as the radius, and the vertical field of view (FOV) as the angular radian; the horizontal scanning area 2 of the laser radar 1 is a sector with the laser radar 1 as the center, the maximum detection distance of the laser radar 1 as the radius, and the horizontal FOV as the angular radian. The simulated weather effect can be displayed only in Figure 2 The three-dimensional area enclosed by the corresponding fan shape shown.

[0095] Optionally, a particle generation system can be established on top of a dynamic weather effect display area in a three-dimensional simulation space based on a corresponding dynamic weather effect particle model generated by a static weather effect. The particle generation system can continuously generate multiple three-dimensional coordinate points in the dynamic weather display area. The three-dimensional coordinate points move with a preset vector and speed. When the three-dimensional coordinate points move out of the display range of the dynamic weather effect, the three-dimensional coordinate points are deleted. The dynamic particles corresponding to the static weather effect are represented by the three-dimensional coordinate points. The falling trajectory and falling speed of the dynamic particles are represented by the rate of change of the coordinate values ​​of the coordinates representing the direction and speed of movement in the three-dimensional coordinates of the dynamic particles. The corresponding weather particle effects include raindrops, snowflakes, etc.

[0096] Taking the dynamic particles as raindrops as an example, a raindrop map is added to each three-dimensional coordinate point. Adjusting the longitudinal movement speed of the three-dimensional coordinate point can represent the amount of rainfall. The faster the longitudinal movement speed of the three-dimensional coordinate point, the greater the rainfall. Adjusting the lateral movement speed of the three-dimensional coordinate point can represent the size of the wind speed. The faster the lateral movement speed of the three-dimensional coordinate point, the faster the wind speed. Adjusting the generation speed of the three-dimensional coordinate point of the particle generation system can represent the amount of rainfall. The faster the generation speed of the three-dimensional coordinate point, the greater the rainfall.

[0097] In another embodiment provided by the present disclosure, the sensor model includes: a visible light camera model and / or a laser radar model;

[0098] Among them, the visible light camera model includes: a medium-focus lens simulated based on the physical camera model, a simulated telephoto lens, and a fisheye camera lens simulated by superimposing multiple physical camera model field of view textures based on a correction algorithm; the lidar model is used for physical camera models based on multiple preset field of view angles, and the depth rendering of the static simulation scene model to form a depth map, so as to simulate the information in the depth map as point cloud data sampled by the lidar corresponding ray angle.

[0099] In the disclosed embodiment, the mid-focus lens effect can use the preset physical camera model in the simulation space, and adjust the focal length of the corresponding physical camera model between 85mm and 135mm; the telephoto lens effect can use the preset physical camera model in the simulation space, and adjust the focal length of the corresponding physical camera model to be above 135mm; the fisheye camera lens effect can be simulated using a large field of view physical camera model and a medium field of view physical camera model at the same position and in the same direction, and during the rendering process, the textures rendered by the large FOV physical camera model and the medium FOV physical camera model are superimposed, and the superimposed textures are converted into fisheye camera images using a fisheye distortion model; the lidar effect can use multiple physical camera models of the same specification to form a circle to form a camera model group with the lens facing outward, so that the horizontal FOV combination of the camera model group is 360°, and during the rendering process, the cameras in the camera model group only render the vertices within the field of view, thereby simulating the point cloud of the lidar.

[0100] like Figure 3 As shown in FIG. 1 , it is a schematic diagram of simulating a fisheye lens using a physical camera model 6 with a horizontal FOV of 160° and a physical camera model 5 with a horizontal FOV of 90°. Figure 4 As shown, it is a schematic diagram of using 4 physical camera models with a horizontal FOV of 90° to form a camera model group 7.

[0101] In another embodiment provided by the present disclosure, before constructing a mesh model of a static simulation scene according to oblique photography data, the following steps are also included:

[0102] Preprocessing the oblique photography data includes at least one of the following:

[0103] Remove noise and spots introduced during the image data acquisition process;

[0104] Crop the edge of the image data to eliminate the portion with low edge resolution;

[0105] Use the preset correction algorithm to correct the image data distortion;

[0106] Detect shadows and reflections of image data using a preset image processing technology, and process the shadows and reflections using a preset shadow removal algorithm;

[0107] and / or,

[0108] After constructing the mesh model of the static simulation scene based on the oblique photography data, it also includes:

[0109] Adjusting the mesh model of the generated static simulation scene includes at least one of the following:

[0110] Replace the attached scene model with the art model;

[0111] Correct the parts of the scene model that are inconsistent with the actual scene;

[0112] Standardizing road sign information, replacing the road sign information with standard indication information;

[0113] Add modeling details and refine the road surface to the ideal state.

[0114] In the disclosed embodiment, before constructing a mesh model of a static simulation scene based on oblique photography data, the oblique photography data needs to be preprocessed to adjust the data obtained from the oblique photography to a picture suitable for mesh model building. Use a denoising algorithm to eliminate noise and noise in the image data; crop the edges of the image data and retain the high-resolution parts in the center of the image data; use a correction algorithm to correct the distortion of the image data; use a preset image processing technology to detect the shadows and reflections of the image data, and process the shadows and reflections using a preset shadow elimination algorithm;

[0115] After constructing the mesh model of the static simulation scene based on the oblique photography data, the mesh model needs to be adjusted; the subsidiary scene model is replaced with the art model, the subsidiary scene model generated by oblique photography in the mesh model is deleted, and the preset model in the model library is placed in the same position. The subsidiary scene models include tree models, street light models, telephone pole models, traffic sign models, etc.; compared with the real scene, modify the part of the mesh model that does not match the real environment; according to the traffic regulations of the target area, adjust the road traffic instruction information to standard information. The road traffic instruction information includes: road surface signs, traffic signs, etc.; add vertices to the mesh model to increase the number of facets, and trim the mesh model of the road surface to make the road surface conform to the real road surface.

[0116] To sum up, Figure 5 A schematic diagram of a method for generating unmanned driving training data provided by an embodiment of the present disclosure is shown in FIG. Figure 5 As shown. A static simulation scene model with preset weather effects is imported into the simulated three-dimensional space constructed in the simulator, and the corresponding dynamic weather effects are imported, and a simulated unmanned vehicle model is configured therein. The unmanned vehicle model is built according to the actual vehicle appearance and performance parameters, and a simulated sensor model is configured at the same position as the actual vehicle. The simulation function of the simulated sensor model is realized through physical camera simulation. During the movement of the simulated unmanned vehicle in the simulation scene, the simulated sensor model can collect perception data. The collected perception data is converted into data that can be used in the autonomous driving data set for the perception algorithm to train and obtain perception training data.

[0117] In addition, other dynamic object models can be added to the simulated three-dimensional space, such as high-precision models of mining trucks, excavators, inspection vehicles, pedestrians, etc., and made to move along preset trajectories to simulate the long-tail scenarios in real mine unmanned driving.

[0118] Based on the same disclosed concept, the embodiments of the present disclosure also provide an unmanned driving training data generation device, system and cloud platform. Since the principles of the problems solved by these devices, systems and cloud platforms are similar to those of the aforementioned unmanned driving training data generation method, the implementation of the device, system and cloud platform can refer to the implementation of the aforementioned method, and the repeated parts will not be repeated.

[0119] The present disclosure provides a device for generating unmanned driving training data. Figure 6 As shown, including:

[0120] The model building module 601 is used to obtain oblique photography data obtained by oblique photography of a real scene, and to build a mesh model of a static simulation scene according to the oblique photography data;

[0121] The weather effect simulation module 602 is used to obtain a material model corresponding to a preset weather effect, and input the surface texture map and surface normal of the mesh model into the material model to obtain a static simulation scene model with the preset weather effect;

[0122] The scene simulation module 603 is used to obtain the perception data collected by the simulated unmanned vehicle through the simulation sensor during the movement of the simulated unmanned vehicle along the preset trajectory in the static simulation scene by using the simulation scene model with the preset weather effect, the unmanned vehicle model and the sensor model of the sensor configured by the unmanned vehicle in the three-dimensional space area set by the simulator;

[0123] The training data generation module 604 is used to obtain target unmanned driving perception training data based on the perception data.

[0124] In another embodiment provided by the present disclosure, the weather effect simulation module 602 is used to obtain a material model corresponding to a preset static weather effect; input the patch texture map and patch normal of the mesh model into the material model of the preset static weather effect to obtain a static simulation scene model with the preset static weather effect;

[0125] The weather effect simulation module 602 is also used to add dynamic weather effects to the static simulation scene model with the preset static weather effects by using particle effects based on the preset static weather effects.

[0126] In another embodiment provided by the present disclosure, the weather effect simulation module 602 is used to divide the mesh model into regions according to the preset static weather effect, and perform corresponding effect processing on different regions to obtain a static simulation scene model with a preset static weather effect; the effect includes at least one of the following: roughness, reflectivity and artistic effect.

[0127] In another embodiment provided by the present disclosure, the weather effect simulation module 602 is used to determine the area where the normal of the face is approximately perpendicular to the horizontal plane as the precipitation pit surface generation area; use a noise algorithm to set the size and shape of the precipitation pit surface for the precipitation pit surface generation area, obtain a first map of the precipitation pit surface, and adjust the reflection coefficient of the first map to the target reflection coefficient to obtain the adjusted first map; for the area outside the precipitation pit surface generation area, adjust at least one of the roughness, reflectivity and color corresponding to the precipitation weather effect to generate a second map; and use the first map and the second map to cover the corresponding position of the mesh model to obtain a static simulation scene model with precipitation weather effect; the preset static weather effects include: precipitation weather effect.

[0128] In another embodiment provided by the present disclosure, a weather effect simulation module 602 is used to determine a dynamic weather effect display area in the three-dimensional space area; generate a preset number of three-dimensional coordinate points representing dynamic particles corresponding to the preset static weather effect in the display area; obtain a target map of the dynamic particles, and determine the rate of change of coordinate values ​​of the coordinates representing the direction and speed of movement in the three-dimensional coordinates of the dynamic particles, and add the target map and the rate of change of coordinate values ​​to the static simulation scene model to achieve the addition of the dynamic weather effect; wherein the display area includes a preset area in the three-dimensional space area, or an area through which the simulated laser rays emitted by the laser radar model in the sensor model pass; the preset number and the rate of change of coordinate values ​​are determined based on the weather level corresponding to the dynamic weather effect.

[0129] In another embodiment provided by the present disclosure, the sensor model includes: a visible light camera model and / or a lidar model; wherein the visible light camera model includes: a medium-focus lens simulated based on a physical camera model, a simulated telephoto lens, and a fisheye camera lens simulated by superimposing a plurality of physical camera model field of view textures based on a correction algorithm; the lidar model is used to deeply render the static simulation scene model to form a depth map based on a physical camera model with a plurality of preset field of view angles, so as to simulate the information in the depth map as point cloud data sampled by the lidar corresponding to the ray angle.

[0130] The model building module 601 is also used to pre-process the oblique photography data before building the mesh model of the static simulation scene according to the oblique photography data, including at least one of the following:

[0131] Removing noise and spots introduced during the image data acquisition process;

[0132] Crop the edge of the image data to eliminate the portion with low edge resolution;

[0133] Use the preset correction algorithm to correct the image data distortion;

[0134] Detect shadows and reflections of image data using a preset image processing technology, and process the shadows and reflections using a preset shadow removal algorithm;

[0135] and / or,

[0136] After constructing the mesh model of the static simulation scene according to the oblique photography data, the generated mesh model of the static simulation scene is further adjusted, including at least one of the following:

[0137] Replace the attached scene model with the art model;

[0138] Correct the parts of the scene model that are inconsistent with the actual scene;

[0139] Standardizing road sign information, replacing the road sign information with standard indication information;

[0140] Add modeling details and refine the road surface to the ideal state.

[0141] An embodiment of the present disclosure provides a cloud platform, including: an unmanned driving training data generating device provided by any of the above embodiments.

[0142] The disclosed embodiment provides an unmanned driving training data generation system, comprising: a cloud platform and at least one oblique photography device;

[0143] The at least one oblique photography device is used to perform oblique photography of a real scene to obtain oblique photography data;

[0144] The cloud platform is used to construct a grid mesh model of a static simulation scene based on the oblique photography data; obtain a material model corresponding to a preset weather effect, and input the face texture map and face normal of the mesh model into the material model to obtain a static simulation scene model with a preset weather effect; in a three-dimensional space area set by a simulator, use the simulation scene model with the preset weather effect, as well as an unmanned vehicle model and a sensor model of a sensor configured for the unmanned vehicle to obtain perception data collected by the simulated unmanned vehicle through the simulation sensor during the movement of the simulated unmanned vehicle along a preset trajectory in the static simulation scene; based on the perception data, obtain target unmanned driving perception training data.

[0145] Through the description of the above implementation methods, those skilled in the art can clearly understand that the embodiments of the present disclosure can be implemented by hardware, or by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present disclosure.

[0146] Those skilled in the art will appreciate that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the modules or processes in the accompanying drawings are not necessarily required for implementing the present disclosure.

[0147] Those skilled in the art can understand that the modules in the device in the embodiment can be distributed in the device in the embodiment according to the description of the embodiment, or can be changed accordingly and located in one or more devices different from the present embodiment. The modules in the above embodiment can be combined into one module, or can be further divided into multiple sub-modules.

[0148] The serial numbers of the above-mentioned embodiments of the present disclosure are only for description and do not represent the advantages or disadvantages of the embodiments.

[0149] Obviously, those skilled in the art can make various changes and modifications to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is also intended to include these modifications and variations.

Claims

1. A method for generating unmanned driving training data, characterized in that: include: Obtaining oblique photography data obtained by oblique photography of a real scene, and constructing a mesh model of a static simulation scene according to the oblique photography data; Obtain a material model corresponding to a preset weather effect, and input the surface texture map and surface normal of the mesh model into the material model to obtain a static simulation scene model with the preset weather effect; In the three-dimensional space area set by the simulator, the simulation scene model with the preset weather effect, the unmanned vehicle model and the sensor model of the sensor configured by the unmanned vehicle are used to obtain the perception data collected by the simulation sensor of the simulated unmanned vehicle during the movement of the simulated unmanned vehicle according to the preset trajectory in the static simulation scene; Based on the perception data, target unmanned driving perception training data is obtained.

2. The method according to claim 1, characterized in that Obtaining a material model corresponding to a preset weather effect, and inputting the surface texture map and surface normal of the mesh model into the material model to obtain a static simulation scene model with the preset weather effect, including: Get the material model corresponding to the preset static weather effect; Inputting the patch texture map and patch normal of the mesh model into the material model of the preset static weather effect to obtain a static simulation scene model with the preset static weather effect; The method further comprises: Based on the preset static weather effect, particle special effects are used to add dynamic weather effects to the static simulation scene model with the preset static weather effect.

3. The method according to claim 2, characterized in that Inputting the patch texture map and patch normal of the mesh model into the material model of the preset static weather effect to obtain a static simulation scene model with the preset static weather effect, including: The mesh model is divided into regions according to the preset static weather effect, and corresponding effect processing is performed on different regions to obtain a static simulation scene model with the preset static weather effect; the effect includes at least one of the following: roughness, reflectivity and artistic effect.

4. The method according to claim 3, characterized in that The preset static weather effect includes: precipitation weather effect; the mesh model is divided into regions according to the preset static weather effect, and corresponding effect processing is performed on different regions to obtain a static simulation scene model with the preset static weather effect, including: The area where the normal of the patch is approximately perpendicular to the horizontal plane is determined as the precipitation pit surface generation area; Using a noise algorithm to set the size and shape of the precipitation pit surface for the precipitation pit surface generation area, obtaining a first texture of the precipitation pit surface, and adjusting the reflection coefficient of the first texture to a target reflection coefficient to obtain an adjusted first texture; For the area outside the precipitation pit surface generation area, adjusting at least one of the roughness, reflectivity and color corresponding to the precipitation weather effect to generate a second map; and The first map and the second map are used to cover the corresponding positions of the mesh model to obtain a static simulation scene model with precipitation weather effects.

5. The method according to claim 2, characterized in that Based on the preset static weather effect, particle effects are used to add dynamic weather effects to the static simulation scene model having the preset static weather effect, including: Determining a dynamic weather effect display area in the three-dimensional space area; Generating a preset number of three-dimensional coordinate points representing dynamic particles corresponding to the preset static weather effect in the display area; Acquire a target map of the dynamic particle, determine the rate of change of the coordinate value of the coordinate representing the moving direction and moving speed in the three-dimensional coordinates of the dynamic particle, and add the target map and the rate of change of the coordinate value to the static simulation scene model to achieve the addition of the dynamic weather effect; Among them, the display area includes a preset area in the three-dimensional space area, or an area through which the simulated laser rays emitted by the lidar model in the sensor model pass; the preset number and the rate of change of the coordinate value are determined based on the weather level corresponding to the dynamic weather effect.

6. The method according to claim 1, characterized in that The sensor model includes: a visible light camera model and / or a laser radar model; Among them, the visible light camera model includes: a medium-focus lens simulated based on a physical camera model, a simulated telephoto lens, and a fisheye camera lens simulated by superimposing multiple physical camera model field of view textures based on a correction algorithm; the lidar model is used to render the static simulation scene model in depth to form a depth map based on a physical camera model with multiple preset field of view angles, so as to simulate the information in the depth map as point cloud data sampled by the lidar corresponding to the ray angle.

7. The method according to claim 1, characterized in that Before constructing the mesh model of the static simulation scene according to the oblique photography data, the method further includes: Preprocessing the oblique photography data includes at least one of the following: Removing noise and spots introduced during the image data acquisition process; Crop the edge of the image data to eliminate the portion with low edge resolution; Use the preset correction algorithm to correct the image data distortion; Detect shadows and reflections of image data using a preset image processing technology, and process the shadows and reflections using a preset shadow removal algorithm; and / or, After constructing the mesh model of the static simulation scene according to the oblique photography data, the method further includes: Adjusting the mesh model of the generated static simulation scene includes at least one of the following: Replace the attached scene model with the art model; Correct the parts of the scene model that are inconsistent with the actual scene; Standardizing road sign information, replacing the road sign information with standard indication information; Add modeling details and refine the road surface to the ideal state.

8. An unmanned driving training data generating device, characterized in that: include: A model building module, used to obtain oblique photography data obtained by oblique photography of a real scene, and to build a mesh model of a static simulation scene according to the oblique photography data; A weather effect simulation module, used to obtain a material model corresponding to a preset weather effect, and input the surface texture map and surface normal of the mesh model into the material model to obtain a static simulation scene model with the preset weather effect; A scene simulation module is used to obtain, in a three-dimensional space area set by the simulator, perception data collected by the simulated unmanned vehicle through the simulation sensor during the movement of the simulated unmanned vehicle along the preset trajectory in the static simulation scene by using the simulation scene model with the preset weather effect, the unmanned vehicle model and the sensor model of the sensor configured by the unmanned vehicle; The training data generation module is used to obtain target unmanned driving perception training data based on the perception data.

9. A cloud platform, characterized in that: include: An unmanned driving training data generating device as claimed in claim 8.

10. An unmanned driving training data generation system, characterized in that: include: A cloud platform and at least one oblique photography device; The at least one oblique photography device is used to perform oblique photography of a real scene to obtain oblique photography data; The cloud platform is used to construct a mesh model of a static simulation scene according to the oblique photography data; obtain a material model corresponding to a preset weather effect, and input a patch texture map and a patch normal of the mesh model into the material model to obtain a static simulation scene model with a preset weather effect; in a three-dimensional space area set by a simulator, use the simulation scene model with the preset weather effect, as well as an unmanned vehicle model and a sensor model of a sensor configured for the unmanned vehicle to obtain perception data collected by the simulated unmanned vehicle through a simulation sensor during the movement of the simulated unmanned vehicle along a preset trajectory in the static simulation scene; Based on the perception data, target unmanned driving perception training data is obtained.

Images