Three-dimensional model generation method and system based on multi-data fusion

Abstract

The application relates to the technical field of three-dimensional reconstruction, and particularly discloses a three-dimensional model generation method and system based on multi-data fusion, which comprises the following steps: controlling a UAV to perform tilt photography on a real scene to be reconstructed, generating a first point cloud based on data obtained through the tilt photography, collecting a second point cloud on a road of the real scene to be reconstructed through a handheld laser radar, controlling a vehicle carrying the laser radar to drive on the road to collect a third point cloud, fusing the first point cloud, the second point cloud and the third point cloud to obtain a multi-data fusion point cloud, generating a three-dimensional model of the real scene to be reconstructed based on the multi-data fusion point cloud, scanning a road surface through the handheld laser radar and scanning a surrounding environment of the road through the vehicle-mounted laser radar to obtain point clouds, fusing the first point cloud generated by collecting data through the tilt photography to generate a high-precision three-dimensional model, and accurately reproducing details such as road ups and downs, slopes, corners, side slopes, guardrails and traffic signs of the real scene, so that the simulation effect of a real scene of a game is improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of three-dimensional reconstruction, and particularly relates to a three-dimensional model generation method and system based on multi-data fusion. BACKGROUND

[0002] When a game scene is made by referring to a real scene, a three-dimensional model of the game scene is usually made by referring to a photo or a video of the real scene. For example, a three-dimensional model of a racing game scene is made by referring to a three-dimensional satellite image, an image or a video of a real scene. However, the three-dimensional model made by the photo, the video or the satellite image of the real scene is difficult to reflect the ups and downs, the slope and the corner of the ground in the real scene, and the user is difficult to experience the real simulation effect in the game. SUMMARY

[0003] The embodiment of the present application aims to provide a three-dimensional model generation method and system based on multi-data fusion, and aims to solve the problem that the existing three-dimensional model generation is difficult to reflect the ups and downs, the slope and the corner of the ground in the real scene, and affects the simulation experience of the user in the game.

[0004] To achieve the above-mentioned purpose, the embodiment of the present application provides the following technical scheme:

[0005] A three-dimensional model generation method based on multi-data fusion, specifically comprising the following steps:

[0006] Controlling a UAV carrying a camera to take oblique photography of a real scene to be reconstructed to obtain oblique photography data, and generating a first point cloud of the real scene to be reconstructed based on the oblique photography data;

[0007] Collecting a second point cloud of a road of the real scene to be reconstructed by a handheld laser radar;

[0008] Controlling a vehicle carrying a laser radar to drive on the road of the real scene to be reconstructed to collect a third point cloud;

[0009] Fusing the first point cloud, the second point cloud and the third point cloud to obtain a multi-data fusion point cloud;

[0010] Generating a three-dimensional model of the real scene to be reconstructed based on the multi-data fusion point cloud.

[0011] As a further technical scheme of the present application, before controlling a UAV carrying a camera to take oblique photography of a real scene to be reconstructed to obtain oblique photography data, the following steps are further included:

[0012] Obtaining a satellite image of the real scene to be reconstructed;

[0013] Generate a flight path of the unmanned aerial vehicle based on the satellite image, the flight path comprising unmanned aerial vehicle waypoints and image acquisition points.

[0014] As a further technical solution of the present application, the unmanned aerial vehicle carrying the camera controls the oblique photography of the real scene to be reconstructed to obtain oblique photography data, and generates a first point cloud of the real scene to be reconstructed based on the oblique photography data, specifically comprising the following steps:

[0015] The unmanned aerial vehicle carrying the camera controls the oblique photography of the real scene to be reconstructed to obtain oblique photography data, and generates a first point cloud of the real scene to be reconstructed based on the oblique photography data, specifically comprising the following steps:

[0016] When flying to the image acquisition point, control the camera to acquire images to obtain oblique photography images;

[0017] Generate a first point cloud of the real scene to be reconstructed based on the oblique photography images and the pose data of the unmanned aerial vehicle at the image acquisition point.

[0018] As a further technical solution of the present application, before collecting a second point cloud of the road of the real scene to be reconstructed by the handheld laser radar, specifically comprising the following steps:

[0019] Generate an intermediate three-dimensional model of the real scene to be reconstructed based on the first point cloud;

[0020] Input the intermediate three-dimensional model into a road recognition model to obtain at least one target road segment to be reconstructed.

[0021] As a further technical solution of the present application, collect a second point cloud of the road of the real scene to be reconstructed by the handheld laser radar, specifically comprising the following steps:

[0022] Scan the target road segment to be reconstructed by the handheld laser radar to obtain a second point cloud.

[0023] As a further technical solution of the present application, fuse the first point cloud, the second point cloud and the third point cloud to obtain a multi-data fusion point cloud, specifically comprising the following steps:

[0024] Preprocess the first point cloud, the second point cloud and the third point cloud to obtain preprocessed first point cloud, second point cloud and third point cloud;

[0025] ICP registration fusion is performed on the preprocessed first point cloud, second point cloud and third point cloud to obtain a multi-data fusion point cloud.

[0026] As a further technical solution of the present application, ICP registration fusion is performed on the preprocessed first point cloud, second point cloud and third point cloud to obtain a multi-data fusion point cloud, specifically comprising the following steps:

[0027] ICP registration fusion is performed on the preprocessed first point cloud and the second point cloud, and a fourth point cloud is obtained;

[0028] Road surface point clouds are removed from the preprocessed third point cloud, and a fifth point cloud is obtained;

[0029] ICP registration fusion is performed on the fourth point cloud and the fifth point cloud, and a multi-data fusion point cloud is obtained.

[0030] As a further technical solution of the present application, a three-dimensional model of the real scene to be reconstructed is generated based on the multi-data fusion point cloud, specifically including the following steps:

[0031] A triangular mesh is generated based on the multi-data fusion point cloud, and an initial three-dimensional surface model of the real scene to be reconstructed is obtained;

[0032] The initial three-dimensional surface model is smoothed to obtain a three-dimensional model of the real scene to be reconstructed.

[0033] A three-dimensional model generation system based on multi-data fusion, specifically including the following units:

[0034] A first point cloud acquisition unit is configured to control a UAV equipped with a camera to perform oblique photography on a real scene to be reconstructed to obtain oblique photography data, and generate a first point cloud of the real scene to be reconstructed based on the oblique photography data;

[0035] A second point cloud acquisition unit is configured to collect a second point cloud of a road of the real scene to be reconstructed by a handheld laser radar;

[0036] A third point cloud acquisition unit is configured to control a vehicle equipped with a laser radar to drive on the road of the real scene to be reconstructed to collect a third point cloud;

[0037] A point cloud fusion unit is configured to fuse the first point cloud, the second point cloud and the third point cloud to obtain a multi-data fusion point cloud;

[0038] A three-dimensional model generation unit is configured to generate a three-dimensional model of the real scene to be reconstructed based on the multi-data fusion point cloud.

[0039] As a further technical solution of the present application, the following units are further included:

[0040] A satellite image acquisition unit is configured to acquire a satellite image of the real scene to be reconstructed;

[0041] A flight path generation unit is configured to generate a flight path of a UAV based on the satellite image, the flight path including UAV waypoints and image acquisition points.

[0042] Compared with the prior art, the present application has the following advantages:

[0043] The embodiment of the present application generates a first point cloud based on the data obtained by the tilt photography of the real scene to be reconstructed by the unmanned aerial vehicle, collects a second point cloud by the handheld laser radar on the road of the real scene to be reconstructed, and controls the vehicle carrying the laser radar to drive on the road of the real scene to be reconstructed to collect a third point cloud, and after the first point cloud, the second point cloud and the third point cloud are fused to obtain a multi-data fusion point cloud, generates a three-dimensional model of the real scene to be reconstructed based on the multi-data fusion point cloud, which realizes that the first point cloud of the whole and rough region to be reconstructed is obtained by the high-altitude tilt photography of the unmanned aerial vehicle, the second point cloud is collected on the road by the handheld laser radar to collect the details such as the ups and downs, the slope and the corner of the road, and the third point cloud is collected by the vehicle-mounted laser radar to drive on the road to collect the details such as the side slope, the guardrail and the traffic sign on the road, so that the multi-data fusion point cloud after fusion can represent the real scene of the real scene to be reconstructed, and the high-precision three-dimensional model applied in the game can simulate the ups and downs, the road bump, the slope, the bump, the corner, the side slope, the guardrail, the depth of the ditch and the traffic sign on the road in the real scene, and improve the simulation effect of the game. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application.

[0045] Figure 1 A flow chart of the three-dimensional model generation method based on multi-data fusion provided by an embodiment of the present application is shown.

[0046] Figure 2 A schematic diagram of the aerial tilt photography of the unmanned aerial vehicle is shown.

[0047] Figure 3 A schematic diagram of the true color point cloud obtained by the handheld laser radar is shown.

[0048] Figure 4 A schematic diagram of the point cloud obtained by the vehicle-mounted laser radar is shown.

[0049] Figure 5 A schematic diagram of the three-dimensional model generated after the multi-data fusion point cloud generates a three-dimensional grid is shown.

[0050] Figure 6 An application architecture diagram of the three-dimensional model generation system based on multi-data fusion provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0051] In order to make the objects, technical solutions and advantages of the present application clearer, further detailed description will be given to the present application in combination with the accompanying drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and should not be used to limit the present application.

[0052] Figure 1 A flow chart of the method for generating a three-dimensional model based on multi-data fusion provided by the embodiment of the present application is shown. Specifically, the method for generating a three-dimensional model based on multi-data fusion of the embodiment of the present application specifically includes the following steps:

[0053] In step S101, the unmanned aerial vehicle carrying the camera is controlled to take oblique photography of the real scene to be reconstructed to obtain oblique photography data, and a first point cloud of the real scene to be reconstructed is generated based on the oblique photography data.

[0054] One application scenario of the embodiment is to establish a three-dimensional model of a real scene, and apply the three-dimensional model in a game scene. For example, the game can be a racing game, and the real scene can be a real scene with roads suitable for racing, such as a mountainous area with roads or a city street. The real scene to be reconstructed is the real scene to be simulated in the game, and the embodiment can obtain a first point cloud of the real scene to be reconstructed by taking aerial oblique photography of the real scene to be reconstructed by the unmanned aerial vehicle carrying the camera.

[0055] Specifically, a satellite image of the real scene to be reconstructed can be obtained first, and a flight path of the unmanned aerial vehicle is generated based on the satellite image. The flight path includes the unmanned aerial vehicle waypoints and image acquisition points. The image acquisition points can be points for controlling the camera on the unmanned aerial vehicle to acquire images. The unmanned aerial vehicle carrying the camera is controlled to fly over the real scene to be reconstructed according to the flight path. When the unmanned aerial vehicle flies to the image acquisition point, the camera is controlled to acquire images to obtain oblique photography images. Based on the oblique photography images and the pose data of the unmanned aerial vehicle at the image acquisition point, a first point cloud of the real scene to be reconstructed is generated.

[0056] The satellite image can be a stereoscopic image of the real scene to be reconstructed, or a height map of the real scene to be reconstructed, the stereoscopic image including the height of an object on the ground surface of the real scene to be reconstructed, the satellite image, the flight height, the flight speed, the flight path overlap rate of the unmanned aerial vehicle can be input into a path planning program of the unmanned aerial vehicle to obtain an initial flight path of the unmanned aerial vehicle when the unmanned aerial vehicle performs oblique photography on the real scene to be reconstructed, the initial flight path including a waypoint and an image acquisition point, the waypoint being used to control the direction and height of flight of the unmanned aerial vehicle, and the image acquisition point being used to control the camera on the unmanned aerial vehicle to acquire images, the initial path is adjusted by a surveying personnel to obtain a final flight path, for example, the surveying personnel can add, delete or adjust the image acquisition point in the initial flight path to acquire images at the best image acquisition point. Of course, the surveying personnel can also select a target area on a two-dimensional map of the real scene to be reconstructed, and manually set a flight path in the target area, and the embodiment does not limit the way of generating the flight path.

[0057] The flight height of the flight path of the unmanned aerial vehicle can change along with the terrain of the real scene to be reconstructed, so that the unmanned aerial vehicle can fly along the terrain in the real scene to be reconstructed to perform oblique photography.

[0058] The unmanned aerial vehicle can be equipped with two or more cameras, and the angles of view of the cameras are different to acquire images from different angles, wherein the cameras can be calibrated in advance to obtain calibration parameters and then stored.

[0059] As shown in FIG. 1, the unmanned aerial vehicle flies along the flight path to acquire images at the image acquisition points. Figure 2 As shown in FIG. 2, the flight path is a plurality of parallel paths, and the unmanned aerial vehicle controls the camera to expose and acquire images when reaching each image acquisition point to obtain oblique photography data, the oblique photography data including images, pose data of the unmanned aerial vehicle when the images are acquired, pose data of the camera, etc., and the first point cloud of the real scene to be reconstructed can be calculated based on the acquired images, the pose data of the unmanned aerial vehicle and the pose data of the camera.

[0060] Specifically, for a plurality of images acquired at each image acquisition point, the images can be preprocessed, such as lens distortion correction, color correction, brightness and contrast adjustment, and then feature points in each image are recognized, the recognized feature points can be matched in the plurality of images to establish a corresponding relationship between the images, and further, the feature points extracted from different images are matched to establish a relative positional relationship of the feature points of different images in space, and further, a preliminary three-dimensional structure, i.e., a sparse point cloud, is obtained through triangulation based on the matched feature points, and a dense point cloud is obtained by further estimating more scene details based on the sparse point cloud, the dense point cloud is the first point cloud, and details are not described herein again, which can refer to related technologies of generating a three-dimensional model by oblique photography technology.

[0061] In step S102, a second point cloud is collected by a handheld laser radar on the road of the real scene to be reconstructed.

[0062] In actual applications, the unmanned aerial oblique photography may be affected by obstructions or photography accuracy, and the first point cloud may not reflect the slope, corner, ups and downs, etc. of the road on the ground, and thus a second point cloud needs to be collected by a handheld laser radar on the road of the real scene to be reconstructed.

[0063] In one embodiment, the second point cloud can be obtained by walking on the road of the real scene to be reconstructed by a handheld laser radar, and scanning the road surface by the handheld laser radar, so as to reflect the slope, bumps, road pits, corners, ups and downs, etc. of the road surface of all roads of the real scene to be reconstructed.

[0064] In another embodiment, an intermediate three-dimensional model of the real scene to be reconstructed can be generated based on the first point cloud, the intermediate three-dimensional model is input into a road recognition model to obtain at least one target road section to be reconstructed, and the target road section to be reconstructed is scanned by a handheld laser radar to obtain a second point cloud.

[0065] Specifically, the road recognition model can be pre-trained so that the road recognition model can recognize the slope section, corner section, and obstructed section of the road. For example, a three-dimensional model sample of other scenes in the prior art can be labeled with the slope section, corner section, and obstructed section, and then the three-dimensional model sample is input into the road recognition model to obtain the position of the detection frame and the type of the road section. The loss rate is calculated by the position of the detection frame and the type of the road section, and the parameters of the road recognition model are adjusted by the loss rate. The training is continued until the loss rate is less than a preset value or the number of training reaches a preset number.

[0066] After obtaining the intermediate three-dimensional model of the real scene to be reconstructed by the unmanned aerial oblique photography, the intermediate three-dimensional model can be input into the road recognition model to obtain the slope section, corner section, and obstructed section of the road in the real scene to be reconstructed as the target road section. The road recognition model outputs the position of the detection frame (such as the latitude and longitude coordinates of the detection frame) and the type of various road sections, so that the survey personnel can hand the laser radar to reach each target road section according to the position of the detection frame, and then scan and collect a second point cloud on each target road section by the handheld laser radar. The scanning and collecting of the point cloud by the laser radar can refer to the prior art, and will not be described in detail here.

[0067] In yet another embodiment, the intermediate three-dimensional model can also be displayed, and the target road section is determined in response to a selection operation of the survey personnel on a slope road section, a corner road section, a sheltered road section, etc. in the intermediate three-dimensional model.

[0068] The embodiment identifies the slope road section, the corner road section, the sheltered road section, etc. as the target road section from the intermediate model generated from the first point cloud, and further scans and collects the second point cloud for each target road section by the handheld laser radar, so that the point cloud can be re-collected for the slope road section, the corner road section, the sheltered road section, etc. On the one hand, the handheld laser radar does not need to scan and collect the point cloud for the entire road section, reducing the workload of the survey personnel, and on the other hand, the data volume of the second point cloud can be reduced, and the complexity of subsequent point cloud fusion can be reduced.

[0069] As shown in Figure 3 The schematic diagram of the true color point cloud collected for the road surface is shown, and since the handheld laser radar mainly scans the road surface, the point cloud of the road surface and the objects (such as guardrails and road edges) near the road is more prominent in the true color point cloud.

[0070] In step S103, the vehicle carrying the laser radar is controlled to drive on the road of the real scene to be reconstructed to collect the third point cloud.

[0071] In the embodiment, the laser radar can be installed on the roof of the survey vehicle, and the vehicle is controlled to drive on the road of the real scene to be reconstructed. The vehicle-mounted laser radar scans the surrounding environment of the road where the vehicle is located at a preset frequency to obtain the third point cloud. The vehicle-mounted laser radar scans the surrounding environment of the road, and the point cloud of objects such as guardrails, traffic signs, and slopes on both sides of the road can be scanned to make up for the details that cannot be collected by unmanned aerial oblique photography.

[0072] As shown in Figure 4 The schematic diagram of the point cloud obtained by scanning and solving by the vehicle-mounted laser radar is shown. From the point cloud in Figure 4 The point cloud can reflect the details such as the guardrails of the road, the ditches beside the road, and the slopes.

[0073] In step S104, the first point cloud, the second point cloud, and the third point cloud are fused to obtain a multi-data fusion point cloud.

[0074] Specifically, the first point cloud, the second point cloud, and the third point cloud are preprocessed to obtain the preprocessed first point cloud, the preprocessed second point cloud, and the preprocessed third point cloud. The preprocessed first point cloud, the preprocessed second point cloud, and the preprocessed third point cloud are ICP registered and fused to obtain a multi-data fusion point cloud.

[0075] Before the first point cloud, the second point cloud and the third point cloud are fused, the first point cloud, the second point cloud and the third point cloud can be preprocessed, such as denoising, smoothing and other preprocessing of the point cloud, to remove abnormal points in the point cloud, to improve the quality of the first point cloud, the second point cloud and the third point cloud, to reduce the fusion error, and to improve the fusion accuracy.

[0076] In one embodiment, the preprocessed first point cloud and the second point cloud can be fused by ICP registration to obtain a fourth point cloud, the road surface point cloud is removed from the preprocessed third point cloud to obtain a fifth point cloud, and the fourth point cloud and the fifth point cloud are fused by ICP registration to obtain a multi-data fusion point cloud. The road surface point cloud collected by the handheld laser radar is first fused into the first point cloud collected by the unmanned aerial vehicle aerial oblique photography to obtain the fourth point cloud, so that the road surface in the fourth point cloud can reflect the detailed features of the road, such as the ups and downs, corners and slopes, and then the fifth point cloud is obtained by removing the road surface point cloud from the third point cloud collected by the vehicle-mounted laser radar, and then the fifth point cloud is fused into the fourth point cloud, so that the point cloud of the road edge, guardrail and traffic sign collected by the vehicle-mounted laser radar is fused into the fourth point cloud, so that the final multi-data fusion point cloud not only includes the global features of the scene to be reconstructed collected by the unmanned aerial vehicle aerial oblique photography, but also includes the detailed features of the road surface collected by the handheld laser radar, such as the ups and downs, corners and slopes, and the features of the road edge, guardrail, traffic sign and slope beside the road collected by the vehicle-mounted laser radar.

[0077] Among them, the point cloud registration fusion can be ICP (Iterative Closest Point, Iterative Closest Point) registration, which mainly determines the rigid body change relationship between two point clouds. Taking the first point cloud and the second point cloud as an example, the first point cloud can be taken as the reference, and for a point q i in the first point cloud, n nearest points p i are found in the second point cloud according to the predetermined constraint condition, the optimal matching parameters R and t are calculated, and the error function is calculated, as follows: ;

[0078] After each iteration, the parameters R and t are used to perform coordinate rotation and translation conversion on the second point cloud to obtain a new second point cloud, the average distance between the point q i and the n nearest points p i is calculated, and if the average distance is less than a threshold value or the number of iterations reaches a predetermined number, the iteration is stopped, the final parameters R and t are obtained, and the second point cloud is converted using the final parameters R and t, that is, the coordinates of the first point cloud and the second point cloud in the same coordinate system can be obtained, and the point cloud registration is completed. When the optimization is minimized, the point q i and the point pi Distance can also be calculated using the coordinates of point q. i and point p i The difference is calculated by fitting the normal vector to the adjacent points. When finding the nearest neighbor, the KD-tree (k-dimensional tree) algorithm can be used. KD-tree is a data structure that divides the k-dimensional data space. It is mainly used for finding the nearest neighbor and approximate nearest neighbor of key data in multi-dimensional space.

[0079] Similarly, after fusing the first point cloud and the second point cloud using the first point cloud as a reference through ICP registration to obtain the fourth point cloud, when fusing the fourth point cloud and the fifth point cloud, the fourth point cloud, which includes the first point cloud, can be used as a reference to further fuse the fourth point cloud and the fifth point cloud through ICP registration to obtain a multi-data fused point cloud.

[0080] Step S105: Generate a 3D model of the real scene to be reconstructed based on the multi-data fusion point cloud.

[0081] Specifically, a triangular mesh can be generated based on multi-data fusion point cloud to obtain an initial 3D surface model of the real scene to be reconstructed. The initial 3D surface model is then smoothed to obtain a 3D model of the real scene to be reconstructed. For details, please refer to the method of generating a 3D model from point cloud, which will not be elaborated here.

[0082] like Figure 5 The diagram shows the generation of triangular meshes from multi-data fusion point clouds. This involves connecting three adjacent points in the multi-data fusion point cloud to obtain a triangular mesh. Multiple triangular meshes form the initial 3D surface model of the real scene to be reconstructed. After smoothing, the 3D model of the real scene to be reconstructed is obtained.

[0083] The embodiment of the application obtains the first point cloud by tilting photography of the real scene to be reconstructed by the unmanned aerial vehicle, collects the second point cloud by the handheld laser radar on the road of the real scene to be reconstructed, and collects the third point cloud by controlling the vehicle carrying the laser radar to drive on the road of the real scene to be reconstructed. After the first point cloud, the second point cloud and the third point cloud are fused to obtain the multi-data fusion point cloud, the three-dimensional model of the real scene to be reconstructed is generated based on the multi-data fusion point cloud, the overall and rough first point cloud of the region to be reconstructed is obtained by high-altitude tilting photography of the unmanned aerial vehicle, the second point cloud is collected on the road by the handheld laser radar to collect the details such as the ups and downs, the slope and the corner of the road, and the third point cloud is collected by the vehicle-mounted laser radar to drive on the road to collect the details such as the side slope, the guardrail and the traffic sign on the road, so that the multi-data fusion point cloud after fusion can represent the real scene of the real scene to be reconstructed. After the three-dimensional model is applied to the game, the ups and downs, the road bumps, the bumps, the slope, the corner, the side slope, the guardrail, the ditch depth and the traffic sign of the road in the real scene can be simulated truly and accurately, and the simulation effect of the game is improved.

[0084] Figure 6 The application architecture diagram of the three-dimensional model generation system based on multi-data fusion provided by the embodiment of the application is shown. The three-dimensional model generation system based on multi-data fusion of the embodiment specifically includes the following units:

[0085] The first point cloud acquisition unit 601 is configured to control the unmanned aerial vehicle carrying the camera to perform tilting photography on the real scene to be reconstructed to obtain tilting photography data, and generate the first point cloud of the real scene to be reconstructed based on the tilting photography data.

[0086] The second point cloud acquisition unit 602 is configured to collect the second point cloud on the road of the real scene to be reconstructed by the handheld laser radar.

[0087] The third point cloud acquisition unit 603 is configured to control the vehicle carrying the laser radar to drive on the road of the real scene to be reconstructed to collect the third point cloud.

[0088] The point cloud fusion unit 604 is configured to fuse the first point cloud, the second point cloud and the third point cloud to obtain the multi-data fusion point cloud.

[0089] The three-dimensional model generation unit 605 is configured to generate the three-dimensional model of the real scene to be reconstructed based on the multi-data fusion point cloud.

[0090] As a further technical solution of the application, the following units are further included:

[0091] The satellite image acquisition unit is configured to acquire the satellite image of the real scene to be reconstructed.

[0092] A flight path generation unit is configured to generate a flight path of the UAV based on the satellite image, the flight path comprising UAV waypoints and image collection points.

[0093] As a further technical solution of the present application, the first point cloud acquisition unit 601 specifically comprises the following modules:

[0094] A UAV flight control module is configured to control the UAV carrying the camera to fly above the real scene to be reconstructed according to the flight path;

[0095] An image collection control module is configured to control the camera to collect images to obtain oblique photography images when flying to the image collection points;

[0096] A first point cloud generation module is configured to generate a first point cloud of the real scene to be reconstructed based on the oblique photography images and pose data of the UAV at the image collection points.

[0097] As a further technical solution of the present application, the present application specifically further comprises the following units:

[0098] An intermediate three-dimensional model generation unit is configured to generate an intermediate three-dimensional model of the real scene to be reconstructed based on the first point cloud;

[0099] An intermediate three-dimensional model detection unit is configured to input the intermediate three-dimensional model into a road recognition model to obtain at least one target road segment to be reconstructed.

[0100] As a further technical solution of the present application, the second point cloud acquisition unit 602 specifically comprises the following modules:

[0101] A target road segment scanning module is configured to scan the target road segment to be reconstructed by a handheld laser radar to obtain a second point cloud.

[0102] As a further technical solution of the present application, the point cloud fusion unit 604 specifically comprises the following modules:

[0103] A point cloud preprocessing module is configured to preprocess the first point cloud, the second point cloud and the third point cloud to obtain preprocessed first point cloud, second point cloud and third point cloud;

[0104] A point cloud fusion module is configured to perform ICP registration fusion on the preprocessed first point cloud, second point cloud and third point cloud to obtain a multi-data fusion point cloud.

[0105] As a further technical solution of the present application, the point cloud fusion module specifically comprises the following sub-modules:

[0106] A first fusion sub-module is configured to perform ICP registration fusion on the preprocessed first point cloud and second point cloud to obtain a fourth point cloud.

[0107] A road surface point cloud removing submodule is configured to remove road surface point clouds from the third point cloud after preprocessing to obtain a fifth point cloud.

[0108] A second fusion submodule is configured to perform ICP registration fusion on the fourth point cloud and the fifth point cloud to obtain a multi-data fusion point cloud.

[0109] As a further technical solution of the present application, the three-dimensional model generation unit 605 specifically includes the following modules:

[0110] An initial three-dimensional model generation module is configured to generate a triangular mesh based on the multi-data fusion point cloud to obtain an initial three-dimensional surface model of the real scene to be reconstructed.

[0111] A model smoothing module is configured to perform smoothing processing on the initial three-dimensional surface model to obtain a three-dimensional model of the real scene to be reconstructed.

[0112] It should be understood that, although each step in the flowchart of each embodiment of the present application is displayed in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, at least a part of the steps in each embodiment can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these sub-steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least a part of other steps or sub-steps or stages of other steps.

[0113] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The program can be stored in a non-volatile computer readable storage medium, and when the program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, storage, database or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0114] Any combination of the technical features of the above embodiments can be made, and in order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the present application.

[0115] The above embodiments only express several embodiments of the present application, and the description is more specific and detailed, but it should not be understood as limiting the scope of the patent of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

[0116] The above is only the preferred embodiment of the present application, and does not limit the present application, any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for generating a 3D model based on multi-data fusion, characterized in that, The process of creating a 3D model of a road for a racing game includes the following steps: The drone equipped with a camera is controlled to perform oblique photography of the real scene to be reconstructed to obtain oblique photography data, and a first point cloud of the real scene to be reconstructed is generated based on the oblique photography data. The second point cloud is collected by handheld lidar on the road in the real scene to be reconstructed. The second point cloud reflects the slope, bumps, potholes, corners and elevation changes of all roads in the real scene to be reconstructed. The vehicle equipped with lidar is controlled to drive on the road in the real scene to be reconstructed to collect a third point cloud, which includes the point clouds of the road guardrails, road traffic signs, and slopes on both sides of the road. The first point cloud, the second point cloud, and the third point cloud are fused to obtain a multi-data fused point cloud; A 3D model of the real-world scene to be reconstructed is generated based on the multi-data fusion point cloud. The first point cloud, the second point cloud, and the third point cloud are fused to obtain a multi-data fused point cloud, including: The first point cloud, the second point cloud, and the third point cloud are preprocessed to obtain the preprocessed first point cloud, the second point cloud, and the third point cloud. ICP registration and fusion are performed on the preprocessed first and second point clouds to obtain a fourth point cloud. The road surface in the fourth point cloud reflects the detailed features of the road's undulations, corners, and slopes. Remove the road surface point cloud from the preprocessed third point cloud to obtain the fifth point cloud; The fourth point cloud and the fifth point cloud are registered and fused by ICP to obtain a multi-data fused point cloud. The fused point cloud includes global features of the scene to be reconstructed, as well as features of road surface undulations, corners, and slopes, and features of road curbs, guardrails, traffic signs, and slopes on both sides of the road. Before acquiring a second point cloud of the road in the real-world scene to be reconstructed using a handheld lidar, the following steps are also included: An intermediate 3D model of the real-world scene to be reconstructed is generated based on the first point cloud; The intermediate 3D model is input into the road recognition model to obtain the latitude and longitude coordinates of the detection box of at least one target road segment to be reconstructed; The second point cloud of the road in the real scene to be reconstructed is collected using a handheld lidar, specifically including the following steps: The target road segment to be reconstructed is scanned using a handheld lidar based on the latitude and longitude coordinates of the detection frame to obtain a second point cloud; The road recognition model is trained through the following steps: Obtain 3D model samples and mark slope sections, corner sections, and obstructed sections in the 3D model samples; Inputting 3D model samples into the road recognition model yields various types of bounding box locations and road segment types; The loss rate is calculated by determining the location of the detection box and the road segment type. Adjust the parameters of the road recognition model by adjusting the loss rate, and continue training until the loss rate is less than the preset value or the number of training iterations reaches the preset number.

2. The method for generating a 3D model based on multi-data fusion according to claim 1, characterized in that, Before controlling a camera-equipped drone to perform oblique photography of the real-world scene to be reconstructed to obtain oblique photography data, the following steps are also included: Acquire satellite images of the real-world scene to be reconstructed; The flight path of the UAV is generated based on the satellite imagery, and the flight path includes UAV waypoints and image acquisition points.

3. The method for generating a 3D model based on multi-data fusion according to claim 2, characterized in that, The process involves controlling a drone equipped with a camera to perform oblique photography of the real-world scene to be reconstructed, obtaining oblique photography data, and generating a first point cloud of the real-world scene to be reconstructed based on the oblique photography data. Specifically, this includes the following steps: Control the drone equipped with a camera to fly over the real-world scene to be reconstructed along the flight path; When the aircraft flies to the image acquisition point, it controls the camera to acquire images to obtain oblique photographic images; Based on the oblique photographic images and the pose data of the UAV at the image acquisition points, a first point cloud of the real scene to be reconstructed is generated.

4. The method for generating a 3D model based on multi-data fusion according to any one of claims 1-3, characterized in that, The process of generating a 3D model of the real-world scene to be reconstructed based on the multi-data fusion point cloud includes the following steps: Based on the multi-data fusion point cloud, a triangular mesh is generated to obtain the initial three-dimensional surface model of the real scene to be reconstructed; The initial three-dimensional surface model is smoothed to obtain the three-dimensional model of the real scene to be reconstructed.

5. A 3D model generation system based on multi-data fusion, characterized in that, The three-dimensional model used to create roads in racing games includes the following units: The first point cloud acquisition unit is used to control a drone equipped with a camera to perform oblique photography of the real scene to be reconstructed to obtain oblique photography data, and to generate the first point cloud of the real scene to be reconstructed based on the oblique photography data. The second point cloud acquisition unit is used to acquire a second point cloud of the road in the real scene to be reconstructed by a handheld lidar. The third point cloud acquisition unit is used to control a vehicle equipped with lidar to drive on the road in the real scene to be reconstructed and collect the third point cloud. The point cloud fusion unit is used to fuse the first point cloud, the second point cloud, and the third point cloud to obtain a multi-data fused point cloud; A 3D model generation unit is used to generate a 3D model of the real scene to be reconstructed based on the multi-data fusion point cloud. The point cloud fusion unit is specifically used for: The first point cloud, the second point cloud, and the third point cloud are fused to obtain a multi-data fused point cloud, including: The first point cloud, the second point cloud, and the third point cloud are preprocessed to obtain the preprocessed first point cloud, the second point cloud, and the third point cloud. ICP registration and fusion are performed on the preprocessed first and second point clouds to obtain a fourth point cloud. The road surface in the fourth point cloud reflects the detailed features of the road's undulations, corners, and slopes. Remove the road surface point cloud from the preprocessed third point cloud to obtain the fifth point cloud; The fourth point cloud and the fifth point cloud are registered and fused by ICP to obtain a multi-data fused point cloud. The fused point cloud includes global features of the scene to be reconstructed, as well as features of road surface undulations, corners, and slopes, and features of road curbs, guardrails, traffic signs, and slopes on both sides of the road. Specifically, it also includes: An intermediate 3D model generation unit is used to generate an intermediate 3D model of the real scene to be reconstructed based on the first point cloud. The intermediate 3D model detection unit is used to input the intermediate 3D model into the road recognition model to obtain the latitude and longitude coordinates of the detection box of at least one target road segment to be reconstructed. The second cloud acquisition unit specifically includes the following modules: The target road segment scanning module is used to scan the target road segment to be reconstructed using a handheld lidar based on the latitude and longitude coordinates of the detection frame to obtain a second point cloud; The road recognition model is trained through the following steps: Obtain 3D model samples and mark slope sections, corner sections, and obstructed sections in the 3D model samples; Inputting 3D model samples into the road recognition model yields various types of bounding box locations and road segment types; The loss rate is calculated by determining the location of the detection box and the road segment type. Adjust the parameters of the road recognition model by adjusting the loss rate, and continue training until the loss rate is less than the preset value or the number of training iterations reaches the preset number.

6. The 3D model generation system based on multi-data fusion according to claim 5, characterized in that, It also includes the following units: A satellite image acquisition unit is used to acquire satellite images of the real-world scene to be reconstructed; The flight path generation unit is used to generate a flight path for the UAV based on the satellite image, the flight path including UAV waypoints and image acquisition points.

Images