A real scene three-dimensional virtual reality scene construction method and system based on a generative adversarial neural network and oblique photography

Abstract

The application discloses a kind of based on generation confrontation neural network and oblique photography real scene three-dimensional virtual reality scene construction method and system, it is related to three-dimensional modeling technical field.The method is in based on unmanned aerial vehicle oblique photography and obtains the initial real scene three-dimensional scene of target field area, first, the model surface initial two-dimensional image of each field object in initial real scene is obtained by cutting out, then these model surface initial two-dimensional image is respectively modeled hollow area identification processing, and the modeling hollow area identification result is obtained, then based on GAN, the model surface initial two-dimensional image with modeling hollow area is carried out image restoration processing, and the model surface complete two-dimensional image is obtained, finally, according to model surface complete two-dimensional image, re-rendering and model replacement can be carried out, and the final real scene three-dimensional scene can be obtained, so the purpose of repairing three-dimensional model hollow area by reprocessing based on unmanned aerial vehicle oblique photography real scene three-dimensional modeling preliminary achievement can be realized.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of three-dimensional modeling, and particularly relates to a real scene three-dimensional virtual reality scene construction method and system based on a generative adversarial neural network and oblique photography. BACKGROUND

[0002] Oblique photography measurement technology is a new and popular surveying and mapping technology. It obtains high-precision images by carrying multiple sensors on an aircraft to shoot from multiple angles such as vertical and oblique. There are many oblique photography platforms at home and abroad, such as ADS series cameras, RCD30 oblique aerial camera, UCO-P aerial camera of Microsoft Corporation, Pictometry oblique photography system, and SWDC-5 oblique camera of Four Dimensions. The common oblique aerial camera is a 5-patch camera, that is, 5 cameras are respectively directed forward, backward, left, right, and vertically downward. At each exposure point, different angle images are obtained by multiple lenses at the same time. The same specific object can be imaged on multiple different angle images at different exposure points. In order to facilitate the processing of data later, the exposure time, plane position, flight attitude, and other data are obtained at the same time when the image is shot. The combination of unmanned aerial vehicles and oblique photography technology is an effective way to realize low-cost and rapid establishment of urban real scene three-dimensional model. Because the unmanned aerial vehicle flies at a low height, the resolution of the oblique photograph taken is high, and the color is closer to the color observed by the human eye, which can significantly improve the realism of the urban three-dimensional model.

[0003] At present, the software for real scene three-dimensional modeling based on unmanned aerial vehicle oblique photography mainly includes ContextCapture, etc. The predecessor of the software is Smart3D of Acute3D company, which is a revolutionary fully automatic three-dimensional modeling software. It can generate super-high-density point clouds using continuous multi-angle images without human intervention, and generate high-resolution three-dimensional scenes with real image texture on this basis.

[0004] At present, the real scene three-dimensional modeling technology based on unmanned aerial vehicle oblique photography can observe the same object from multiple angles, making the object texture more abundant and the effect more realistic, which is the mainstream direction of future three-dimensional city modeling. However, due to the influence of aerial photography blind area and feature point matching error, etc., the automatically generated three-dimensional model will have a hollow area, which cannot meet the high requirements of some specific projects (such as driving training simulation projects) on model quality. Therefore, how to reprocess the preliminary results of real scene three-dimensional modeling based on unmanned aerial vehicle oblique photography to repair the hollow area of the three-dimensional model and obtain a cloned real scene three-dimensional scene of the target scene area is a subject that needs to be studied by those skilled in the art. SUMMARY

[0005] The application aims to provide a real scene three-dimensional virtual reality scene construction method and system based on a generative adversarial neural network and oblique photography, a driving training simulator system, a computer device and a computer readable storage medium, so as to solve the problem that the existing real scene three-dimensional modeling technology based on unmanned aerial vehicle oblique photography will cause the automatically generated three-dimensional model to have a hollow area and cannot meet the higher requirements of some specific projects on model quality.

[0006] In order to achieve the above-mentioned purpose, the application adopts the following technical solutions:

[0007] In a first aspect, a real scene three-dimensional virtual reality scene construction method based on a generative adversarial neural network and oblique photography is provided, comprising:

[0008] Obtaining oblique photography data collected by an unmanned aerial vehicle oblique photography device on a target site area and unmanned aerial vehicle attitude measurement data or image control measurement data recorded synchronously with the oblique photography data;

[0009] According to the oblique photography data and the unmanned aerial vehicle attitude measurement data or the image control measurement data, an unmanned aerial vehicle oblique photography real scene three-dimensional modeling software is used to construct an initial real scene three-dimensional scene of the target site area, wherein the initial real scene three-dimensional scene contains initial three-dimensional models of a plurality of site objects;

[0010] For each site object in the plurality of site objects, a corresponding model surface initial two-dimensional image is obtained by cutting according to the corresponding initial three-dimensional model;

[0011] The model surface initial two-dimensional images of the respective site objects are respectively subjected to modeling hollow area identification processing to obtain modeling hollow area identification results of the respective site objects;

[0012] For the respective site objects, if the corresponding modeling hollow area identification result indicates that there is at least one modeling hollow area in the corresponding model surface initial two-dimensional image, the model surface initial two-dimensional image is subjected to image restoration processing based on a generative adversarial neural network (GAN) to obtain a corresponding model surface complete two-dimensional image, otherwise the corresponding model surface initial two-dimensional image is directly taken as the corresponding model surface complete two-dimensional image;

[0013] For the respective site objects, the corresponding model surface complete two-dimensional image is rendered onto the surface of the corresponding initial three-dimensional model to obtain a corresponding final three-dimensional model;

[0014] In the initial real scene three-dimensional scene, the initial three-dimensional models of the respective site objects are updated to the corresponding final three-dimensional models to obtain a final real scene three-dimensional scene of the target site area.

[0015] Based on the above invention content, a new scheme for repairing the hollow area of a three-dimensional model based on a generative adversarial neural network (GAN) is provided, that is, after obtaining an initial real scene three-dimensional scene of a target site area based on unmanned aerial vehicle oblique photography, the model surface initial two-dimensional image of each site object in the initial real scene is first cut out, then the modeling hollow area identification processing is performed on these model surface initial two-dimensional images respectively to obtain the modeling hollow area identification result, then the model surface initial two-dimensional image with the modeling hollow area is processed by image repair based on the generative adversarial neural network (GAN) to obtain the model surface complete two-dimensional image, and finally the model surface complete two-dimensional image is re-rendered and replaced to obtain the final real scene three-dimensional scene. In this way, the purpose of reprocessing the real scene three-dimensional modeling preliminary results based on unmanned aerial vehicle oblique photography to repair the hollow area of the three-dimensional model can be achieved, and a cloned real scene three-dimensional scene of the target site area can be obtained, which meets the high requirements of some specific projects on model quality and is convenient for practical application and promotion.

[0016] In one possible design, for a certain site object in the plurality of site objects, if the corresponding modeling hollow area identification result indicates that there is at least one modeling hollow area in the corresponding model surface initial two-dimensional image, the model surface initial two-dimensional image is processed by image repair based on the generative adversarial neural network (GAN) to obtain the corresponding model surface final two-dimensional image, including:

[0017] For a certain site object in the plurality of site objects, if the corresponding modeling hollow area identification result indicates that there is at least one modeling hollow area in the corresponding model surface initial two-dimensional image, the at least one modeling hollow area is arranged in order from small to large according to the area to obtain a modeling hollow area sequence;

[0018] For the kth modeling hollow area in the modeling hollow area sequence, the model surface repair two-dimensional image corresponding to the (k-1)th modeling hollow area in the modeling hollow area sequence is processed by image repair based on the generative adversarial neural network (GAN) to obtain the corresponding model surface repair two-dimensional image, where k represents a positive integer, and the model surface initial two-dimensional image of the certain site object is taken as the model surface repair two-dimensional image corresponding to the zeroth modeling hollow area;

[0019] The model surface repair two-dimensional image corresponding to the last modeling hollow area in the modeling hollow area sequence is taken as the model surface final two-dimensional image of the certain site object.

[0020] In one possible design, for the kth modeling hollow region in the sequence of modeling hollow regions, a model surface repaired two-dimensional image corresponding to the (k-1)th modeling hollow region in the sequence of modeling hollow regions is generated by performing image inpainting processing on the model surface repaired two-dimensional image corresponding to the (k-1)th modeling hollow region in the sequence of modeling hollow regions based on a generative adversarial neural network (GAN), including the following steps S521-S525:

[0021] S521. An image generator in a complete image generation model based on a generative adversarial neural network (GAN) and having completed pre-training is applied to generate a new image, and then step S522 is performed;

[0022] S522. An image discriminator in the complete image generation model is applied to determine whether the new image is a complete image, if yes, step S523 is performed, otherwise, the image generator is applied again to generate a new image, and then step S522 is performed;

[0023] S523. Color difference values of each pixel point in a non-modeling hollow region of the new image and the model surface repaired two-dimensional image corresponding to the (k-1)th modeling hollow region are calculated, and then step S524 is performed, where k represents a positive integer, and the model surface initial two-dimensional image of the certain field object is taken as the model surface repaired two-dimensional image corresponding to the zeroth modeling hollow region;

[0024] S524. It is determined whether a standard deviation of the color difference values of each pixel point of the two images reaches a preset standard deviation threshold, if yes, the new image is taken as the model surface repaired two-dimensional image corresponding to the kth modeling hollow region in the sequence of modeling hollow regions, otherwise, step S525 is performed;

[0025] S525. The color difference values of each pixel point of the two images are imported into the image generator as content loss penalty term data, and the image generator is applied again to generate a new image, and then step S522 is performed.

[0026] In one possible design, the training process of the complete image generation model includes:

[0027] A plurality of object surface two-dimensional images are obtained;

[0028] The plurality of object surface two-dimensional images are applied to train a generative adversarial neural network (GAN) including an image generator and an image discriminator, to obtain the complete image generation model.

[0029] In one possible design, the model surface initial two-dimensional images of the respective field objects are subjected to modeling hollow region identification processing respectively, to obtain modeling hollow region identification results of the respective field objects, including:

[0030] For each of the on-site objects, the corresponding model surface initial two-dimensional image is introduced into a modeling hollow area recognition model based on the YOLO target detection algorithm and pre-trained, and the corresponding modeling hollow area recognition result is output.

[0031] In one possible design, the model surface initial two-dimensional image of each of the on-site objects is subjected to modeling hollow area recognition processing respectively, and the modeling hollow area recognition result of each of the on-site objects is obtained, including:

[0032] For each of the on-site objects, the corresponding model surface initial two-dimensional image is introduced into a modeling hollow area recognition model based on the YOLO target detection algorithm and pre-trained, and the corresponding modeling hollow area recognition result is output.

[0033] If the modeling hollow area recognition result of the on-site object indicates that there is at least one modeling hollow area bounding box in the model surface initial two-dimensional image of the on-site object, at least one modeling hollow area image corresponding to the at least one modeling hollow area bounding box is intercepted from the model surface initial two-dimensional image of the on-site object according to the at least one modeling hollow area bounding box.

[0034] The at least one modeling hollow area image is subjected to image denoising processing, gray scale conversion processing and binarization processing based on a preset gray scale threshold value in sequence respectively, and at least one binarization image corresponding to the at least one modeling hollow area image is obtained, wherein the preset gray scale threshold value is pre-set according to a modeling hollow area gray scale value.

[0035] For each of the at least one binarization image, a corresponding center connected domain is extracted based on the Canny algorithm, and the center connected domain is taken as a modeling hollow area within the corresponding modeling hollow area bounding box.

[0036] All the modeling hollow areas are summarized to obtain the final modeling hollow area recognition result of the on-site object.

[0037] In a second aspect, a real scene three-dimensional virtual reality scene construction system based on a generative adversarial neural network and oblique photography is provided, including data acquisition modules, initial modeling modules, image extraction modules, hollow area recognition modules, image repair modules, image rendering modules and model replacement modules which are sequentially and communicatively connected.

[0038] The data acquisition module is configured to acquire oblique photography data collected by an unmanned aerial vehicle oblique photography device on a target on-site region and unmanned aerial vehicle attitude measurement data or image control measurement data recorded synchronously with the oblique photography data.

[0039] The initial modeling module is configured to construct an initial real scene three-dimensional scene of the target site area by using a real scene three-dimensional modeling software for unmanned aerial vehicles based on the oblique photography data and the unmanned aerial vehicle attitude measurement data or the image control measurement data, wherein the initial real scene three-dimensional scene comprises initial three-dimensional models of a plurality of site objects.

[0040] The image extraction module is configured to extract an initial two-dimensional image of a model surface of each site object in the plurality of site objects based on the corresponding initial three-dimensional model.

[0041] The cavity identification module is configured to perform modeling cavity region identification processing on the initial two-dimensional image of the model surface of each site object to obtain a modeling cavity region identification result of the site object.

[0042] The image repairing module is configured to, for each site object, if the corresponding modeling cavity region identification result indicates that there is at least one modeling cavity region in the initial two-dimensional image of the model surface, perform image repairing processing on the initial two-dimensional image of the model surface based on a generative adversarial neural network (GAN) to obtain a complete two-dimensional image of the model surface, or directly use the initial two-dimensional image of the model surface as the complete two-dimensional image of the model surface.

[0043] The image rendering module is configured to, for each site object, render the complete two-dimensional image of the model surface onto the surface of the corresponding initial three-dimensional model to obtain a final three-dimensional model of the site object.

[0044] The model replacing module is configured to replace the initial three-dimensional model of each site object with the final three-dimensional model of the site object in the initial real scene three-dimensional scene to obtain a final real scene three-dimensional scene of the target site area.

[0045] In a third aspect, the present application provides a driving training simulator system, comprising a driving component, a VR display, a motion control card, a four-degree-of-freedom motion platform and a visual information processing device, wherein the driving component comprises a steering wheel, a gas pedal and a brake pedal.

[0046] The driving component is in communication with the visual information processing device and is configured to generate a driving signal in response to an operation of a driving trainee and transmit the driving signal to the visual information processing device.

[0047] The view information processing device is respectively connected in communication with the VR display and the motion control card, and is configured to determine a driving simulation virtual image of a vehicle driven by the driving trainee and vehicle motion posture information in a final real three-dimensional scene according to the driving signal and the final real three-dimensional scene obtained by applying the real three-dimensional virtual reality scene construction method of the first aspect or any possible design of the first aspect to the driving test area, and transmit the driving simulation virtual image to the VR display and the vehicle motion posture information to the motion control card.

[0048] The VR display is configured to output the driving simulation virtual image to the driving trainee.

[0049] The motion control card is connected in communication with the four-degree-of-freedom motion platform, and is configured to obtain motor pulse quantity according to the vehicle motion posture information, and then control the four-degree-of-freedom motion platform to perform driving simulation motion based on the motor pulse quantity.

[0050] In a fourth aspect, the present application provides a computer device, comprising a memory, a processor and a transceiver connected in sequence in communication, wherein the memory is configured to store a computer program, the transceiver is configured to receive and send messages, and the processor is configured to read the computer program and execute the real three-dimensional virtual reality scene construction method of the first aspect or any possible design of the first aspect.

[0051] In a fifth aspect, the present application provides a computer readable storage medium, wherein instructions are stored on the computer readable storage medium, and when the instructions are executed on a computer, the real three-dimensional virtual reality scene construction method of the first aspect or any possible design of the first aspect is executed.

[0052] In a sixth aspect, the present application provides a computer program product comprising instructions, which, when executed on a computer, cause the computer to execute the real three-dimensional virtual reality scene construction method of the first aspect or any possible design of the first aspect.

[0053] The above-mentioned scheme has the following beneficial effects:

[0054] (1) The application creatively provides a new scheme for repairing a three-dimensional model hollow area based on a generative adversarial neural network (GAN), that is, after obtaining an initial real scene three-dimensional scene of a target site area based on unmanned aerial vehicle oblique photography, the model surface initial two-dimensional image of each site object in the initial real scene is first cut out, then the modeling hollow area identification processing is performed on the model surface initial two-dimensional image to obtain a modeling hollow area identification result, then the model surface initial two-dimensional image with the modeling hollow area is processed by image repair based on the generative adversarial neural network (GAN) to obtain a model surface complete two-dimensional image, and finally, the final real scene three-dimensional scene can be obtained by re-rendering and model replacement according to the model surface complete two-dimensional image, so that the purpose of reprocessing the real scene three-dimensional modeling preliminary results based on unmanned aerial vehicle oblique photography to repair the three-dimensional model hollow area can be achieved, and a cloned real scene three-dimensional scene of the target site area can be obtained, which meets the high requirements of some specific projects on model quality and is convenient for practical application and promotion. BRIEF DESCRIPTION OF DRAWINGS

[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0056] Figure 1 The flowchart of the real scene three-dimensional virtual reality scene construction method provided by the embodiments of the present application.

[0057] Figure 2 The structural diagram of the real scene three-dimensional virtual reality scene construction system provided by the embodiments of the present application.

[0058] Figure 3 The structural diagram of the driving training simulator system provided by the embodiments of the present application.

[0059] Figure 4 The structural diagram of the computer device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0060] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings. It should be noted that the description of these embodiment modes is used to help understand the present application, but does not constitute a limitation on the present application.

[0061] It should be understood that, although the terms first and second, etc. can be used herein to describe various objects, the objects should not be limited by these terms. These terms are only used to distinguish one object from another. For example, a first object could be termed a second object, and, similarly, a second object could be termed a first object, without departing from the scope of example embodiments of the present application.

[0062] It should be understood that, for the term "and / or" possibly occurring herein, it merely describes an association relationship of associated objects, and indicates that three relationships can exist, for example, A and / or B can indicate that A exists alone, B exists alone, or A and B exist simultaneously, and the like; for example, A, B and / or C can indicate that any one of A, B and C exists or any combination thereof; for the term " / " possibly occurring herein, it describes another association relationship of another associated object, and indicates that two relationships can exist, for example, A / and B can indicate that A exists alone or A and B exist simultaneously; in addition, for the character " / " possibly occurring herein, it generally indicates that the associated objects before and after the character " / " are in an "or" relationship.

[0063] Embodiments:

[0064] As shown in Figure 1 , the first aspect of the present embodiment provides a real scene three-dimensional virtual reality scene construction method based on a generative adversarial neural network and oblique photography, which can be executed by a computer device with certain computing resources, such as an information processing device, a platform server, a personal computer (PC, which refers to a multi-purpose computer suitable for personal use in size, price and performance; desktop computers, notebook computers to small notebook computers and tablet computers, and ultrabooks, etc.), a smart phone, a personal digital assistant (PDA), or a wearable device, etc. Figure 1 As shown in

[0065] S1. Obtain oblique photography data collected by an unmanned aerial vehicle oblique photography device on a target site area and unmanned aerial vehicle attitude measurement data or image control measurement data recorded synchronously with the oblique photography data.

[0066] In step S1, specifically, the UAV oblique photography device preferably adopts a typical five-lens oblique gimbal. Since in its working state, the optical axis of the central camera is perpendicular to the horizontal plane, and four cameras are distributed in the four directions with their optical axes at a 45° angle to the horizontal plane, it can simultaneously complete the coverage of three or more images of the same ground object or feature point from different angles during a single flight of the UAV. (At the same time, since the higher the coverage and overlap of the images of the same ground object from different angles, the more refined the calculated model, the overlap of the images during flight will be increased as much as possible when acquiring real-scene 3D modeling data; however, considering that a higher overlap means an additional workload, considering efficiency and the tilt of the aircraft during flight, the flight path is generally set to have a forward overlap of more than 80% and a lateral overlap of more than 60%).

[0067] S2. Based on the oblique photography data and the UAV attitude measurement data or the image control measurement data, an initial real-world 3D scene of the target site area is constructed using UAV oblique photography real-world 3D modeling software, wherein the initial real-world 3D scene contains initial 3D models of multiple site objects.

[0068] In step S2, specifically, the UAV oblique photogrammetry real-scene 3D modeling software preferably uses ContextCapture software.

[0069] S3. For each of the multiple field objects, based on the corresponding initial three-dimensional model, extract the initial two-dimensional image of the corresponding model surface.

[0070] S4. Perform modeling void region identification processing on the initial two-dimensional images of the model surfaces of each of the field objects to obtain the modeling void region identification results of each of the field objects.

[0071] In the step S4, specifically includes but not limited to: for each object, the corresponding model surface initial two-dimensional image is introduced into the modeling hollow area recognition model based on the YOLO target detection algorithm and has completed the pre-training, and the corresponding modeling hollow area recognition result is output. The YOLO (You only look once) target detection algorithm is a kind of existing artificial intelligence recognition algorithm for identifying objects inside and marking object positions in pictures, and the specific model structure of YOLO V4 version is composed of three parts, backbone, neck and head. The backbone can adopt CSPDarknet53 (CSP represents Cross Stage Partial) network for extracting features. The neck is composed of SPP (Spatial Pyramid Pooling block) block and PANet (Path Aggregation Network) network, the former is used to increase the receptive field and separate the most important features, and the latter is used to ensure that semantic features are received from high-level layers and fine-grained features are received from low-level layers of the transverse backbone network. The head network is based on anchor frame detection, and three different size (i.e. 13x13, 26x26 and 52x52) feature maps are detected, which are used to detect large to small targets (here, the feature map with large size contains more information, therefore, the feature map with 52x52 size is used to detect small targets, and vice versa). The foregoing modeling hollow area recognition model can be trained by conventional sample training method, so as to output the recognition result of modeling hollow area and their confidence prediction value and other information after inputting test image.

[0072] In the step S4, in order to accurately determine the modeling hollow area in the model surface initial two-dimensional image, preferably, the model surface initial two-dimensional image of each object is subjected to modeling hollow area recognition processing, to obtain the modeling hollow area recognition result of each object, including but not limited to the following steps S41-S45.

[0073] S41. For a certain object in the plurality of objects, the corresponding model surface initial two-dimensional image is introduced into the modeling hollow area recognition model based on the YOLO target detection algorithm and has completed the pre-training, and the corresponding modeling hollow area recognition result is output.

[0074] S42. If the modeling hollow region identification result of the certain field object indicates that there is at least one modeling hollow region marking box in the model surface initial two-dimensional image of the certain field object, at least one modeling hollow region image corresponding to the at least one modeling hollow region marking box is cut from the model surface initial two-dimensional image of the certain field object according to the at least one modeling hollow region marking box.

[0075] S43. The at least one modeling hollow region image is subjected to image denoising processing, gray scale conversion processing and binarization processing based on a preset gray scale threshold in sequence respectively, to obtain at least one binarization image corresponding to the at least one modeling hollow region image, wherein the preset gray scale threshold is preset according to a modeling hollow region gray scale value.

[0076] S44. For each binarization image in the at least one binarization image, a corresponding center connected domain is extracted based on a Canny algorithm, and the center connected domain is taken as a modeling hollow region in the corresponding modeling hollow region marking box.

[0077] S45. All the modeling hollow regions are summarized to obtain the certain field object and final modeling hollow region identification result.

[0078] S5. For the each field object, if the corresponding modeling hollow region identification result indicates that there is at least one modeling hollow region in the corresponding model surface initial two-dimensional image, the model surface initial two-dimensional image is subjected to image inpainting processing based on a generative adversarial neural network (GAN), to obtain a corresponding model surface complete two-dimensional image, otherwise the corresponding model surface initial two-dimensional image is directly taken as the corresponding model surface complete two-dimensional image.

[0079] In the step S5, the generative adversarial nets (GAN) is a new framework for estimating a generative model: two models are trained simultaneously, one is a generative model for capturing the data distribution, and the other is a discriminative model for distinguishing whether the data is real data or generated data (pseudo data); the image processing task uses a neural network such as CNN to analyze and process the input image to obtain information related to the content of the input image; in contrast to the image processing task, in the image generation task, the image generation model generates an image according to the input information related to the content of the input image; for the image generation task, the input of the image generation model is uncertain, depending on the scene and the specific model design, and the style transfer belongs to one of the scenes; the generative adversarial nets (GAN) can be applied to the style transfer. In the generative adversarial nets, there are two core components: the generator (Generator, the aforementioned generative model) and the discriminator (Discriminator, the aforementioned discriminative model); the discriminator and the generator can be composed of a multi-layer perceptron (which can be regarded as a fully connected neural network, FC), and in the training process, the key step of "adversarial" is: first fix the parameters of the generator, train and optimize the discriminator so that the discriminator can accurately distinguish "real image" and "fake image"; then fix the parameters of the discriminator, train and optimize the generator to make the discriminator unable to accurately distinguish "real image" and "fake image", so that after the model training is completed, the trained generator (Generator) can be used to generate images. In addition, the discriminator can also use a convolutional neural network (Strided Convolution, ordinary convolution operation, which will reduce the channel if not padded), and the generator can also use transposed convolution (Transposed Convolution) (which can also be regarded as deconvolution) to achieve.

[0080] In the step S5, in order to quickly and effectively complete the image repair processing, the following easy-to-difficult repair scheme is preferably adopted, that is: for a certain on-site object in the plurality of on-site objects, if the corresponding modeling hollow region identification result indicates that there is at least one modeling hollow region in the corresponding model surface initial two-dimensional image, the model surface initial two-dimensional image is processed based on the generative adversarial nets (GAN) to obtain the corresponding model surface final two-dimensional image, including but not limited to the following steps S51-S53.

[0081] S51. For a certain live object in the plurality of live objects, if the corresponding modeling hollow region identification result indicates that there is at least one modeling hollow region in the corresponding model surface initial two-dimensional image, the at least one modeling hollow region is arranged in order of area from small to large to obtain a modeling hollow region sequence.

[0082] S52. For the kth modeling hollow region in the modeling hollow region sequence, an image inpainting process is performed on the model surface repair two-dimensional image corresponding to the (k-1)th modeling hollow region in the modeling hollow region sequence based on a generative adversarial neural network (GAN) to obtain a corresponding model surface repair two-dimensional image, wherein k represents a positive integer, and the model surface initial two-dimensional image of the certain live object is taken as the model surface repair two-dimensional image corresponding to the zeroth modeling hollow region.

[0083] In the step S52, the following steps S521-S525 are included but not limited thereto.

[0084] S521. An image generator in a complete image generation model based on a generative adversarial neural network (GAN) and having completed pre-training is applied to generate a new image, and then step S522 is performed.

[0085] In the step S521, in detail, the training process of the complete image generation model includes but is not limited to the following: first, a plurality of real object surface two-dimensional images are obtained; then, the plurality of object surface two-dimensional images are applied to train a generative adversarial neural network (GAN) including an image generator and an image discriminator to obtain the complete image generation model.

[0086] S522. An image discriminator in the complete image generation model is applied to determine whether the new image is a complete image, if yes, step S523 is performed, otherwise, the image generator is applied again to generate a new image, and then step S522 is performed.

[0087] S523. According to the new image and the model surface repair two-dimensional image corresponding to the (k-1)th modeling hollow region, color difference values of each pixel point in the non-modeling hollow region of the two images are calculated, and then step S524 is performed, wherein k represents a positive integer, and the model surface initial two-dimensional image of the certain live object is taken as the model surface repair two-dimensional image corresponding to the zeroth modeling hollow region.

[0088] S524. It is determined whether the standard deviation of the color difference values of the two images at the pixel points reaches a preset standard deviation threshold, if yes, the new image is taken as the model surface repair two-dimensional image corresponding to the kth modeling hollow region in the modeling hollow region sequence, otherwise, step S525 is performed.

[0089]

[0089] S525. The color difference values of the two images at the respective pixels are imported into the image generator as content loss penalty term data, and the image generator is applied again to generate a new image, and then step S522 is performed.

[0090] S53. The model surface repair two-dimensional image corresponding to the last modelled hollow region in the sequence of modelled hollow regions is taken as the final model surface two-dimensional image of the certain field object.

[0091] S6. For the respective field objects, the corresponding model surface complete two-dimensional image is rendered onto the surface of the corresponding initial three-dimensional model to obtain the corresponding final three-dimensional model.

[0092] S7. In the initial real scene three-dimensional scene, the initial three-dimensional model of the respective field object is updated to the corresponding final three-dimensional model to obtain the final real scene three-dimensional scene of the target field region.

[0093] The real scene three-dimensional virtual reality scene construction method described in the foregoing steps S1-S7 provides a new scheme for repairing three-dimensional model hollow regions based on a generative adversarial neural network GAN, that is, after obtaining an initial real scene three-dimensional scene of a target field region based on unmanned aerial vehicle oblique photography, model surface initial two-dimensional images of respective field objects in the initial real scene are first cut out, then the model surface initial two-dimensional images are subjected to respective modelled hollow region identification processing to obtain modelled hollow region identification results, then the model surface initial two-dimensional images with modelled hollow regions are subjected to image repair processing based on a generative adversarial neural network GAN to obtain model surface complete two-dimensional images, and finally, the model surface complete two-dimensional images are subjected to re-rendering and model replacement to obtain a final real scene three-dimensional scene, so that the purpose of reprocessing the preliminary results of real scene three-dimensional modelling based on unmanned aerial vehicle oblique photography to repair three-dimensional model hollow regions can be achieved, and a cloned real scene three-dimensional scene of the target field region can be obtained, thereby meeting the high requirements of some specific projects on model quality and facilitating actual application and promotion.

[0094] As shown in Figure 2 the second aspect of the present embodiment provides a virtual system for implementing the real scene three-dimensional virtual reality scene construction method of the first aspect, comprising a data acquisition module, an initial modelling module, an image cutting module, a hollow identification module, an image repair module, an image rendering module and a model replacement module which are sequentially communicatively connected;

[0095] The data acquisition module is configured to acquire oblique photography data collected by an unmanned aerial vehicle oblique photography device on a target field region and unmanned aerial vehicle attitude measurement data or image control measurement data recorded synchronously with the oblique photography data.

[0096] The initial modeling module is used to construct an initial real-world 3D scene of the target site area using UAV oblique photography real-world 3D modeling software based on the oblique photography data and the UAV attitude measurement data or the image control measurement data. The initial real-world 3D scene contains initial 3D models of multiple site objects.

[0097] The image extraction module is used to extract the corresponding initial two-dimensional image of the model surface for each of the multiple on-site objects based on the corresponding initial three-dimensional model.

[0098] The cavity recognition module is used to perform cavity region recognition processing on the initial two-dimensional image of the model surface of each on-site object to obtain the cavity region recognition result of each on-site object.

[0099] The image restoration module is used to perform image restoration processing on the initial two-dimensional image of the model surface based on the generative adversarial neural network (GAN) for each of the on-site objects. If the identification result of the corresponding modeling hole region indicates that there is at least one modeling hole region in the initial two-dimensional image of the corresponding model surface, the module obtains the corresponding complete two-dimensional image of the model surface by using the initial two-dimensional image of the model surface directly. Otherwise, the module directly uses the initial two-dimensional image of the model surface as the corresponding complete two-dimensional image of the model surface.

[0100] The image rendering module is used to render the complete two-dimensional image of the corresponding model surface onto the surface of the corresponding initial three-dimensional model for each of the on-site objects, so as to obtain the corresponding final three-dimensional model.

[0101] The model replacement module is used to update the initial three-dimensional model of each on-site object to the corresponding final three-dimensional model in the initial real-world three-dimensional scene, so as to obtain the final real-world three-dimensional scene of the target on-site area.

[0102] The working process, working details and technical effects of the aforementioned system provided in the second aspect of this embodiment can be found in the real-scene three-dimensional virtual reality scene construction method described in the first aspect, and will not be repeated here.

[0103] like Figure 3 As shown, the third aspect of this embodiment provides a driving training simulator system that applies the real-scene three-dimensional virtual reality scene construction method described in the first aspect, including a driving component, a VR (Virtual Reality, VR) display, a motion control card, a four-degree-of-freedom motion platform, and a visual information processing device, wherein the driving component includes a steering wheel, an accelerator pedal, and a brake pedal.

[0104] The driving component is in communication connection with the visual information processing device, and is configured to generate a driving signal in response to the operation of the driving trainee, and transmit the driving signal to the visual information processing device.

[0105] The visual information processing device is in communication connection with the VR display and the motion control card respectively, and is configured to determine a driving simulation virtual image of the vehicle driven by the driving trainee and vehicle motion posture information in a final real three-dimensional scene according to the driving signal and the final real three-dimensional scene obtained by applying the real three-dimensional virtual reality scene construction method of the first aspect to the driving test area, and transmit the driving simulation virtual image to the VR display and the vehicle motion posture information to the motion control card.

[0106] The VR display is configured to output the driving simulation virtual image to the driving trainee.

[0107] The motion control card is in communication connection with the four-degree-of-freedom motion platform, and is configured to calculate a motor pulse amount according to the vehicle motion posture information, and then control the four-degree-of-freedom motion platform to perform driving simulation motion based on the motor pulse amount.

[0108] The working process, working details and technical effects of the aforementioned system provided by the third aspect of the embodiment can be referred to the real three-dimensional virtual reality scene construction method of the first aspect, which will not be described here.

[0109] As shown in Figure 4 The fourth aspect of the embodiment provides a computer device for executing the real three-dimensional virtual reality scene construction method of the first aspect, which comprises a memory, a processor and a transceiver in communication connection in sequence, wherein the memory is configured to store a computer program, the transceiver is configured to transceive messages, and the processor is configured to read the computer program and execute the real three-dimensional virtual reality scene construction method of the first aspect. Specifically, the memory can include, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a flash memory, a first input first output (FIFO) memory and / or a first input last output (FILO) memory, etc.; and the processor can be, but is not limited to, a microprocessor with a model number of STM32F105 series. In addition, the computer device can further include, but is not limited to, a power module, a display screen and other necessary components.

[0110] The working process, working details and technical effects of the aforementioned computer device provided by the fourth aspect of the embodiment can be referred to the live three-dimensional virtual reality scene construction method described in the first aspect, and will not be repeated here.

[0111] The fifth aspect of the embodiment provides a computer readable storage medium storing instructions of the live three-dimensional virtual reality scene construction method described in the first aspect, that is, the computer readable storage medium stores instructions, and when the instructions run on a computer, the live three-dimensional virtual reality scene construction method described in the first aspect is executed. Wherein, the computer readable storage medium refers to a carrier for storing data, which can include, but is not limited to, floppy disks, optical discs, hard disks, flash memories, USB flash disks and / or memory sticks and other computer readable storage media, and the computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable devices.

[0112] The working process, working details and technical effects of the aforementioned computer readable storage medium provided by the fifth aspect of the embodiment can be referred to the live three-dimensional virtual reality scene construction method described in the first aspect, and will not be repeated here.

[0113] The sixth aspect of the embodiment provides a computer program product containing instructions, which, when running on a computer, causes the computer to execute the live three-dimensional virtual reality scene construction method described in the first aspect. Wherein, the computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable devices.

[0114] Finally, it should be noted that: the above only describes the preferred embodiments of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for constructing a real three-dimensional virtual reality scene based on a generative adversarial neural network and oblique photography, characterized in that, The method comprises the following steps: obtaining oblique photography data collected by an unmanned aerial vehicle oblique photography device on a target site area and unmanned aerial vehicle attitude measurement data or image control measurement data recorded synchronously with the oblique photography data; constructing an initial real scene three-dimensional scene of the target site area by using an unmanned aerial vehicle oblique photography real scene three-dimensional modeling software according to the oblique photography data and the unmanned aerial vehicle attitude measurement data or the image control measurement data, wherein the initial real scene three-dimensional scene contains initial three-dimensional models of a plurality of site objects; for each site object in the plurality of site objects, obtaining a corresponding model surface initial two-dimensional image according to the corresponding initial three-dimensional model; respectively performing modeling hollow area identification processing on the model surface initial two-dimensional images of the site objects to obtain modeling hollow area identification results of the site objects; for each site object, if the corresponding modeling hollow area identification result indicates that there is at least one modeling hollow area in the corresponding model surface initial two-dimensional image, performing image restoration processing on the model surface initial two-dimensional image based on a generative adversarial neural network (GAN) to obtain a corresponding model surface complete two-dimensional image, otherwise directly taking the corresponding model surface initial two-dimensional image as the corresponding model surface complete two-dimensional image, and the specific steps include: for a certain site object in the plurality of site objects, if the corresponding modeling hollow area identification result indicates that there is at least one modeling hollow area in the corresponding model surface initial two-dimensional image, arranging the at least one modeling hollow area in order from small to large according to the area to obtain a modeling hollow area sequence; for the kth modeling hollow area in the modeling hollow area sequence, performing image restoration processing on the model surface repair two-dimensional image corresponding to the (k-1)th modeling hollow area in the modeling hollow area sequence based on the generative adversarial neural network (GAN) to obtain a corresponding model surface repair two-dimensional image, wherein k represents a positive integer, and the model surface initial two-dimensional image of the certain site object is taken as the model surface repair two-dimensional image corresponding to the zeroth modeling hollow area; and taking the model surface repair two-dimensional image corresponding to the last modeling hollow area in the modeling hollow area sequence as the model surface final two-dimensional image of the certain site object; for each site object, rendering the corresponding model surface complete two-dimensional image onto the surface of the corresponding initial three-dimensional model to obtain a corresponding final three-dimensional model; in the initial real scene three-dimensional scene, updating the initial three-dimensional models of the site objects to the corresponding final three-dimensional models to obtain a final real scene three-dimensional scene of the target site area.

2. The live three-dimensional virtual reality scene construction method of claim 1, wherein, for the kth modeling hollow area in the modeling hollow area sequence, performing image restoration processing on the model surface repair two-dimensional image corresponding to the (k-1)th modeling hollow area in the modeling hollow area sequence based on the generative adversarial neural network (GAN) to obtain a corresponding model surface repair two-dimensional image, comprising the following steps S521-S525: S521. An image generator in a complete image generation model based on a generative adversarial neural network (GAN) and pre-trained is applied to generate a new image, and then step S522 is performed; S522. An image discriminator in the complete image generation model is applied to determine whether the new image is a complete image, if yes, step S523 is performed, otherwise, the image generator is applied again to generate a new image, and then step S522 is performed; S523. A two-dimensional image is repaired according to the new image and a model surface corresponding to a k-th modeling hollow region, a color difference value of each pixel point in a non-modeling hollow region of the two images is calculated, and then step S524 is performed, wherein k represents a positive integer, and a model surface initial two-dimensional image of the certain field object is taken as a model surface repaired two-dimensional image corresponding to a zero-th modeling hollow region; S524. It is determined whether a standard deviation of the color difference values of the two images at the each pixel point reaches a preset standard deviation threshold, if yes, the new image is taken as a model surface repaired two-dimensional image corresponding to a k-th modeling hollow region in the sequence of modeling hollow regions, otherwise, step S525 is performed; S525. The color difference values of the two images at the each pixel point are taken as content loss penalty term data imported into the image generator, and the image generator is applied again to generate a new image, and then step S522 is performed.

3. The live three-dimensional virtual reality scene construction method of claim 2, wherein, The training process of the complete image generation model comprises: a plurality of object surface two-dimensional images are obtained; the plurality of object surface two-dimensional images are applied to train a generative adversarial neural network (GAN) comprising an image generator and an image discriminator, to obtain the complete image generation model.

4. The live three-dimensional virtual reality scene construction method of claim 1, wherein, The model surface initial two-dimensional images of the each field object are respectively subjected to modeling hollow region identification processing, to obtain the modeling hollow region identification results of the each field object, comprising: for the each field object, the corresponding model surface initial two-dimensional image is imported into a modeling hollow region identification model based on a YOLO target detection algorithm and pre-trained, and the corresponding modeling hollow region identification result is output.

5. The live three-dimensional virtual reality scene construction method of claim 1, wherein, The model surface initial two-dimensional images of the each field object are respectively subjected to modeling hollow region identification processing, to obtain the modeling hollow region identification results of the each field object, comprising: for the certain field object in the plurality of field objects, the corresponding model surface initial two-dimensional image is imported into a modeling hollow region identification model based on a YOLO target detection algorithm and pre-trained, and the corresponding modeling hollow region identification result is output; if the modeling hollow region identification result of the certain field object indicates that there is at least one modeling hollow region bounding box in the model surface initial two-dimensional image of the certain field object, at least one modeling hollow region image corresponding to the at least one modeling hollow region bounding box is cut out from the model surface initial two-dimensional image of the certain field object according to the at least one modeling hollow region bounding box; The at least one modeling hollow region image is sequentially subjected to image denoising processing, gray scale conversion processing and binary processing based on a preset gray scale threshold value, to obtain at least one binary image corresponding to the at least one modeling hollow region image, wherein the preset gray scale threshold value is preset according to a modeling hollow region gray scale value; For each binary image in the at least one binary image, a corresponding central connected domain is extracted based on a Canny algorithm, and the central connected domain is taken as a modeling hollow region in a corresponding modeling hollow region marking frame; All the modeling hollow regions are summarized to obtain a final modeling hollow region identification result of the certain field object.

6. A real scene three-dimensional virtual reality scene construction system based on a generative adversarial neural network and oblique photography, characterized in that, The data acquisition module, the initial modeling module, the image extraction module, the hollow identification module, the image repair module, the image rendering module and the model replacement module are sequentially connected in communication; The data acquisition module is configured to acquire oblique photography data collected by an unmanned aerial vehicle oblique photography device on a target field region and unmanned aerial vehicle attitude measurement data or image control measurement data recorded synchronously with the oblique photography data; The initial modeling module is configured to construct an initial real scene three-dimensional scene of the target field region by using an unmanned aerial vehicle oblique photography real scene three-dimensional modeling software according to the oblique photography data and the unmanned aerial vehicle attitude measurement data or the image control measurement data, wherein the initial real scene three-dimensional scene contains initial three-dimensional models of a plurality of field objects; The image extraction module is configured to extract a model surface initial two-dimensional image of each field object in the plurality of field objects according to a corresponding initial three-dimensional model; The hollow identification module is configured to perform modeling hollow region identification processing on the model surface initial two-dimensional image of each field object, to obtain a modeling hollow region identification result of the field object; and The image repair module is configured to perform image repair processing on the model surface initial two-dimensional image of each field object based on the modeling hollow region identification result of the field object, to obtain a model surface final two-dimensional image of the field object. The image repairing module is configured to, for each of the on-site objects, if the corresponding modeling hole region identification result indicates that there is at least one modeling hole region in the corresponding model surface initial two-dimensional image, perform image repairing processing on the model surface initial two-dimensional image based on a generative adversarial neural network (GAN) to obtain a corresponding model surface complete two-dimensional image, or directly use the corresponding model surface initial two-dimensional image as the corresponding model surface complete two-dimensional image. Specifically, for a certain on-site object in the plurality of on-site objects, if the corresponding modeling hole region identification result indicates that there is at least one modeling hole region in the corresponding model surface initial two-dimensional image, the at least one modeling hole region is arranged in order of area from small to large to obtain a modeling hole region sequence; for a kth modeling hole region in the modeling hole region sequence, a model surface repairing two-dimensional image corresponding to a (k-1)th modeling hole region in the modeling hole region sequence is repaired based on a generative adversarial neural network (GAN) to obtain a corresponding model surface repairing two-dimensional image, where k represents a positive integer, and the model surface initial two-dimensional image of the certain on-site object is used as a model surface repairing two-dimensional image corresponding to a zeroth modeling hole region; and a model surface repairing two-dimensional image corresponding to a last modeling hole region in the modeling hole region sequence is used as a model surface final two-dimensional image of the certain on-site object. The image rendering module is configured to, for each of the on-site objects, render the corresponding model surface complete two-dimensional image onto a surface of a corresponding initial three-dimensional model to obtain a corresponding final three-dimensional model. The model replacing module is configured to, in the initial real scene three-dimensional scene, update the initial three-dimensional model of each of the on-site objects to the corresponding final three-dimensional model to obtain a final real scene three-dimensional scene of the target on-site region.

7. A driving training simulator system characterized by, The driving component, the VR display, the motion control card, the four-degree-of-freedom motion platform, and the visual information processing device are included, wherein the driving component includes a steering wheel, a gas pedal, and a brake pedal. The driving component is in communication with the visual information processing device and is configured to generate a driving signal in response to an operation of a driving trainee and transmit the driving signal to the visual information processing device. The visual information processing device is in communication with the VR display and the motion control card respectively and is configured to determine a driving simulation virtual image of a vehicle driven by the driving trainee and vehicle motion posture information in a final real scene three-dimensional scene obtained by applying the real scene three-dimensional virtual reality scene construction method in any one of claims 1 to 5 according to the driving signal and the driving examination area, transmit the driving simulation virtual image to the VR display, and transmit the vehicle motion posture information to the motion control card. The VR display is configured to output the driving simulation virtual image to the driving trainee. The motion control card is in communication connection with the four-degree-of-freedom motion platform, is used for calculating motor pulse quantity according to the vehicle motion posture information, and then controls the four-degree-of-freedom motion platform to carry out driving simulation motion based on the motor pulse quantity.

8. A computer device, comprising: The method comprises the following steps: a memory, a processor and a transceiver are sequentially connected in communication, wherein the memory is used for storing a computer program, the transceiver is used for transmitting and receiving messages, and the processor is used for reading the computer program and executing the real three-dimensional virtual reality scene construction method in any one of claims 1-5.

9. A computer-readable storage medium, characterized in that The computer readable storage medium has instructions stored thereon, and when the instructions run on the computer, the real three-dimensional virtual reality scene construction method in any one of claims 1-5 is executed.

Images