Method, device and equipment for constructing city image based on simulated aerial photography, storage medium and program product

Abstract

The application discloses a city image construction method and device based on simulated aerial photography, equipment, a storage medium and a program product, and relates to the technical field of image processing. The city image construction method based on simulated aerial photography generates a virtual flight path and samples discrete virtual imaging viewpoints on the virtual flight path. By mapping the whole process of "planning-flying-shooting" in the physical world to the virtual environment, the method provides simulation for unmanned aerial vehicle teaching, high-risk areas and high-risk environments without being restricted by time and weather conditions, and improves the efficiency of task planning and verification of complex aerial photography tasks. By constructing a virtual imaging curved surface and mapping and calibrating time sequence viewpoint images, time sequence images are obtained, and a unified mapping projection reference is provided for all discrete time sequence viewpoint images.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a city image construction method and device based on simulated aerial photography, equipment, a storage medium and a program product. BACKGROUND

[0002] With the continuous development of unmanned aerial vehicle aerial photography technology, unmanned aerial vehicle aerial photography surveying and mapping instead of manual surveying and mapping has become a mainstream development trend in the fields of film and television production, city planning and the like. At present, schools pay more and more attention to the application teaching of unmanned aerial vehicle aerial photography.

[0003] However, the existing unmanned aerial vehicle aerial photography mainly involves that teachers lead students to perform field shooting, and the unmanned aerial vehicle is high in procurement cost and is easy to be damaged due to improper operation of students. Meanwhile, the field unmanned aerial vehicle teaching is greatly affected by the environment and weather. SUMMARY

[0004] The main purpose of the present application is to provide a city image construction method and device based on simulated aerial photography, equipment, a storage medium and a program product, which aims to solve the technical problems that the existing unmanned aerial vehicle aerial photography directly performs field shooting, is easy to be damaged and is greatly affected by the outside.

[0005] To achieve the above-mentioned purpose, the present application provides a city image construction method based on simulated aerial photography, which comprises the following steps:

[0006] generating a virtual flight path based on a simulated aerial photography task of a user, and performing discretization sampling on the virtual flight path to obtain a virtual imaging viewpoint;

[0007] performing global integration according to the virtual imaging viewpoint to obtain a virtual imaging curved surface covering a field of view range of the virtual flight path;

[0008] performing virtual image sampling in a three-dimensional city model based on the virtual imaging viewpoint and the virtual imaging curved surface to obtain a time sequence image corresponding to the virtual imaging viewpoint;

[0009] performing fusion based on the time sequence image corresponding to the virtual imaging viewpoint to obtain a virtual city image of the simulated aerial photography task.

[0010] In an embodiment, the step of performing global integration according to the virtual imaging viewpoint to obtain a virtual imaging curved surface covering a field of view range of the virtual flight path comprises the following steps:

[0011] fitting based on the virtual imaging viewpoint to determine a central trajectory axis of the virtual flight path;

[0012] constructing a complex curved surface parameterization model based on the central trajectory axis;

[0013] mapping the virtual imaging viewpoint to the complex surface parameterized model to obtain a viewpoint parameter coordinate and a normal direction of the virtual imaging viewpoint;

[0014] performing global integration based on the viewpoint parameter coordinate and the normal direction to obtain a virtual imaging surface covering a field of view range of the virtual flight path.

[0015] In an embodiment, the step of mapping the virtual imaging viewpoint to the complex surface parameterized model to obtain a viewpoint parameter coordinate and a normal direction of the virtual imaging viewpoint comprises:

[0016] obtaining a three-dimensional space coordinate and a line-of-sight direction vector of the virtual imaging viewpoint;

[0017] mapping the virtual imaging viewpoint to the complex surface parameterized model according to the three-dimensional space coordinate and the line-of-sight direction vector to obtain a viewpoint parameter coordinate and a normal direction of the virtual imaging viewpoint.

[0018] In an embodiment, the step of performing virtual image sampling based on the virtual imaging viewpoint and the virtual imaging surface in the three-dimensional city model to obtain a time sequence image corresponding to the virtual imaging viewpoint comprises:

[0019] performing virtual image sampling based on the virtual imaging viewpoint in the three-dimensional city model to obtain an initial viewpoint image sequence;

[0020] performing re-projection correction on the initial viewpoint image sequence based on the virtual imaging surface to obtain a time sequence image corresponding to the virtual imaging viewpoint.

[0021] In an embodiment, the step of performing fusion based on the time sequence image corresponding to the virtual imaging viewpoint to obtain the virtual city image of the simulated aerial photography task comprises:

[0022] performing fusion based on overlap region statistical information of the time sequence image corresponding to the virtual imaging viewpoint to obtain a panoramic virtual viewpoint image;

[0023] generating the virtual city image of the simulated aerial photography task based on the panoramic virtual viewpoint image.

[0024] In an embodiment, the step of performing discretization sampling on the virtual flight path to obtain a virtual imaging viewpoint comprises:

[0025] obtaining flight parameters of a UAV based on the simulated aerial photography task;

[0026] Discretely sample based on the generated video frame number and the flight parameter to determine a virtual imaging viewpoint on the virtual flight path.

[0027] In addition, to achieve the above object, the application further provides a city image construction device based on simulated aerial photography, which comprises:

[0028] A viewpoint generation module is configured to generate a virtual flight path based on a simulated aerial photography task of a user, and discretely sample on the virtual flight path to obtain a virtual imaging viewpoint.

[0029] A panorama management module is configured to globally integrate according to the virtual imaging viewpoint to obtain a virtual imaging curved surface covering a field of view range of the virtual flight path.

[0030] An image acquisition module is configured to perform virtual image sampling in a three-dimensional city model based on the virtual imaging viewpoint and the virtual imaging curved surface to obtain a time sequence image corresponding to the virtual imaging viewpoint.

[0031] An image generation module is configured to obtain a virtual city image of the simulated aerial photography task based on the time sequence image corresponding to the virtual imaging viewpoint.

[0032] In addition, to achieve the above object, the application further provides a city image construction device based on simulated aerial photography, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the city image construction method based on simulated aerial photography.

[0033] In addition, to achieve the above object, the application further provides a storage medium, which is a computer readable storage medium, and the storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the city image construction method based on simulated aerial photography.

[0034] In addition, to achieve the above object, the application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps of the city image construction method based on simulated aerial photography.

[0035] The one or more technical solutions provided by the application have at least the following technical effects:

[0036] The application generates a virtual flight path based on a user's simulated aerial photography task, and performs discretization sampling on the virtual flight path to obtain a virtual imaging viewpoint; performs global integration according to the virtual imaging viewpoint to obtain a virtual imaging surface covering the field of view range of the virtual flight path; performs virtual image sampling in a three-dimensional city model based on the virtual imaging viewpoint and the virtual imaging surface to obtain a time sequence image corresponding to the virtual imaging viewpoint; and performs fusion based on the time sequence image corresponding to the virtual imaging viewpoint to obtain a virtual city image of the simulated aerial photography task. Since the virtual flight path is generated and the discretized virtual imaging viewpoint is obtained by sampling on the virtual flight path, the whole process of "planning-flying-shooting" in the physical world is mapped to the virtual environment, so that the output image sequence can meet the preset requirements; at the same time, since the flight path of the simulated unmanned aerial vehicle is used for shooting, the simulation of unmanned aerial vehicle teaching, high-risk areas and high-risk environments is not restricted by time and weather conditions, and the efficiency of task planning and verification of complex aerial photography tasks is improved. The time sequence image is obtained by constructing the virtual imaging surface and mapping and calibrating the time sequence viewpoint image, which provides a unified mapping projection reference for all discrete time sequence viewpoint images. BRIEF DESCRIPTION OF DRAWINGS

[0037] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and serve to explain the principles of the present application together with the specification.

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, for those skilled in the art, other drawings can also be obtained based on these drawings without creative labor.

[0039] Figure 1 A flowchart provided for the first embodiment of the city image construction method based on simulated aerial photography of the present application;

[0040] Figure 2 A flowchart provided for the second embodiment of the city image construction method based on simulated aerial photography of the present application;

[0041] Figure 3 A flowchart provided for the third embodiment of the city image construction method based on simulated aerial photography of the present application;

[0042] Figure 4 A module structure diagram of the city image construction device based on simulated aerial photography of the embodiment of the present application;

[0043] Figure 5 A device structure diagram of the hardware running environment involved in the city image construction method based on simulated aerial photography in the embodiment of the present application.

[0044] The purposes, functional features and advantages of the present application will be further illustrated in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0045] It should be understood that the specific embodiments described herein are merely intended to explain the technical solutions of the present application, and are not intended to limit the present application.

[0046] In order to better understand the technical solutions of the present application, the following will be described in detail in conjunction with the drawings and specific embodiments.

[0047] The main solution of the embodiment of the present application is: generating a virtual flight path based on a user's simulated aerial photography task, and discretely sampling on the virtual flight path to obtain a virtual imaging viewpoint; performing global integration according to the virtual imaging viewpoint to obtain a virtual imaging surface covering the field of view of the virtual flight path; performing virtual image sampling in a three-dimensional city model based on the virtual imaging viewpoint and the virtual imaging surface to obtain a time series image corresponding to the virtual imaging viewpoint; and obtaining a virtual city image of the simulated aerial photography task based on the time series image corresponding to the virtual imaging viewpoint.

[0048] The present application provides a solution. Since the image is constructed by simulating aerial photography of a drone, the user can first understand the flight path control method of the drone and the generation of the simulated image, thereby reducing the risk of damage to the drone when used by student users. At the same time, since the image is generated by simulation, the influence of weather on teaching is reduced, and the scene adaptability of aerial photography of the drone is improved.

[0049] It should be noted that the execution subject of the present embodiment can be a computing service device with data processing, network communication and program running functions, such as a computer, a server, etc., or an electronic device, a virtual device, etc. capable of realizing the above functions. The present embodiment and the following embodiments will be described below taking a city image construction device based on simulated aerial photography (referred to as construction device) as an example.

[0050] Based on this, the present embodiment provides a city image construction method based on simulated aerial photography. Referring to Figure 1 , Figure 1 The present embodiment provides a flowchart of the city image construction method based on simulated aerial photography.

[0051] In the present embodiment, the city image construction method based on simulated aerial photography includes steps S10-S40:

[0052] Step S10, generating a virtual flight path based on a user's simulated aerial photography task, and discretely sampling on the virtual flight path to obtain a virtual imaging viewpoint.

[0053] It should be noted that in the process of UAV aerial photography, in order to obtain the target image of a specific area, the user usually controls the physical UAV and the camera carried on the physical UAV in a series of actions, and the combination of these control actions can be regarded as a UAV aerial photography task. In the embodiments of the present application, the simulated aerial photography task is applied to the virtual UAV model.

[0054] In some embodiments of the embodiments of the present application, the embodiments of the present application and the following embodiments take the construction of city images as an example to illustrate the scheme of the present application. In actual application, the virtual UAV model can be first placed at an initial position in the three-dimensional city model, and then the simulated aerial photography task is executed by the UAV model, so as to generate a flight path of the UAV model, which is the virtual flight path. Through the virtual flight path, the flight path, flight speed, motion turning, turning information and other parameters of the UAV can be represented.

[0055] It can be understood that the initial position can be any position selected by the user in the three-dimensional city model, and the embodiments of the present application do not limit this.

[0056] It should be noted that the virtual imaging viewpoint can be a virtual camera state obtained by discretely sampling the virtual flight path, which is an instantaneous, static and complete imaging parameter that can be used to represent the imaging parameter of the corresponding point of the physical UAV on the flight path. Specifically, the virtual imaging viewpoint can be a simulation of the imaging parameter of the corresponding position of the physical UAV in the physical world, which is equivalent to a digital snapshot of the physical UAV in the virtual world.

[0057] It can be understood that the complete imaging parameter can include camera intrinsic parameters, camera extrinsic parameters, imaging attitude angles, imaging relative times of the imaging unit carried on the physical UAV, and the embodiments of the present application do not limit this.

[0058] It should be noted that when discretely sampling the virtual flight path, it can be achieved by fixed spatial interval sampling or fixed time interval sampling, for example, a virtual imaging viewpoint is set every 10 meters or a virtual imaging viewpoint is set every 0.03 seconds, and the embodiments of the present application do not limit this.

[0059] In some embodiments of the present application, in the manner of using fixed time intervals, the time interval used can be set according to the video frame rate (i.e. the number of generated video frames) corresponding to the generated city image, for example, when the video frame rate of the generated city image is 30 frames, a virtual imaging viewpoint for sampling can be set on the path every 1 / 30 seconds; when the video frame rate of the generated city image is 60 frames, a virtual imaging viewpoint for sampling can be set on the virtual flight path every 1 / 60 seconds. That is, the step of discretely sampling on the virtual flight path to obtain a virtual imaging viewpoint includes: obtaining flight parameters of the unmanned aerial vehicle based on the simulated aerial photography task; discretely sampling based on the number of generated video frames and the flight parameters to determine the virtual imaging viewpoint on the virtual flight path.

[0060] It should be noted that the flight parameters of the unmanned aerial vehicle can include the flight direction and flight speed of the unmanned aerial vehicle on the flight path, and based on the flight parameters of the unmanned aerial vehicle and the number of video frames, the virtual imaging viewpoint on the virtual flight path can be determined.

[0061] In a specific implementation, the construction device of the present application can generate a virtual flight path corresponding to the physical unmanned aerial vehicle flight based on the simulated aerial photography task of the user, and discretely sample on the virtual flight path according to the flight parameters of the unmanned aerial vehicle and the number of video frames generated by the city image to determine the virtual imaging viewpoint on the virtual flight path. By mapping the whole process of "planning-flight-shooting" in the physical world to the virtual environment, the output image sequence can meet the preset requirements; at the same time, since the shooting is performed by simulating the flight path of the unmanned aerial vehicle, the simulation of the unmanned aerial vehicle teaching, high-risk areas and high-risk environments is not restricted by time and weather conditions, and the efficiency of task planning and verification of complex aerial photography tasks is improved.

[0062] Step S20, global integration is performed according to the virtual imaging viewpoint to obtain a virtual imaging surface covering the field of view range of the virtual flight path;

[0063] Step S30, virtual image sampling is performed in the three-dimensional city model based on the virtual imaging viewpoint and the virtual imaging surface to obtain a time sequence image corresponding to the virtual imaging viewpoint;

[0064] Step S40, fusion is performed based on the time sequence image corresponding to the virtual imaging viewpoint to obtain a virtual city image of the simulated aerial photography task.

[0065] It should be noted that the virtual imaging surface described above can be a continuous parametric surface model constructed by globally integrating the position information and the line-of-sight direction vector of all virtual imaging viewpoints. By constructing a continuous imaging virtual imaging surface that can cover the entire flight observation range, a unified framework is provided for image projection, correction and stitching of virtual images generated by each virtual imaging viewpoint, so that the initial viewpoint image sequences collected from different viewpoints can be accurately aligned and fused in the same reference system, thereby generating seamless panoramic virtual viewpoint images.

[0066] It can be understood that the line-of-sight direction vector of the virtual imaging viewpoint can represent the shooting direction of the unmanned aerial vehicle at the virtual imaging viewpoint position. When the virtual imaging viewpoints perform image rendering based on the three-dimensional city model, the initial viewpoint images corresponding to each virtual imaging viewpoint can be determined. Since each virtual imaging viewpoint can be regarded as a position at different time on the flight path of the unmanned aerial vehicle, it can be considered that there is a time sequence relationship between the initial viewpoint images of these virtual imaging viewpoints, and the corresponding time sequence labels are set based on the position / order of the virtual imaging viewpoints and the flight parameters of the unmanned aerial vehicle, to obtain time sequence viewpoint images. The images obtained by mapping the time sequence viewpoint images to the virtual imaging surface are time sequence images.

[0067] In a specific implementation, the construction device of the embodiment of the present application can globally integrate according to the position and viewpoint direction of the virtual imaging viewpoint to determine a virtual imaging surface covering the field of view of the entire virtual flight path. By mapping and calibrating the time sequence viewpoints collected and rendered by each virtual imaging viewpoint through the virtual imaging surface, the time sequence images corresponding to each virtual imaging viewpoint can be determined. Since the time sequence images are obtained by constructing a virtual imaging surface and mapping and calibrating the time sequence viewpoint images, a unified mapping projection reference is provided for all discrete time sequence viewpoint images, avoiding the defect that the traditional multi-viewpoint stitching error will be accumulated frame by frame. By fusing these time sequence images, virtual city images simulating aerial photography tasks can be obtained.

[0068] The embodiment of the application generates a virtual flight path based on a user's simulated aerial photography task, and performs discretization sampling on the virtual flight path to obtain a virtual imaging viewpoint; performs global integration according to the virtual imaging viewpoint to obtain a virtual imaging surface covering the field of view range of the virtual flight path; performs virtual image sampling in a three-dimensional city model based on the virtual imaging viewpoint and the virtual imaging surface to obtain a time sequence image corresponding to the virtual imaging viewpoint; and performs fusion based on the time sequence image corresponding to the virtual imaging viewpoint to obtain a virtual city image of the simulated aerial photography task. Since the virtual flight path is generated and the discretized virtual imaging viewpoint is obtained by sampling on the virtual flight path, the whole process of "planning-flying-shooting" in the physical world is mapped into a virtual environment, so that the output image sequence can meet the preset requirements; at the same time, since the flight path of the unmanned aerial vehicle is simulated for shooting, the simulation of the unmanned aerial vehicle teaching, high-risk areas and high-risk environments is not restricted by time and weather conditions, and the efficiency of task planning and verification of complex aerial photography tasks is improved. By constructing a virtual imaging surface and mapping and calibrating the time sequence viewpoint image to obtain a time sequence image, a unified mapping projection reference is provided for all discrete time sequence viewpoint images.

[0069] Based on the first embodiment of the application, in the second embodiment of the application, the same or similar contents as the above-mentioned first embodiment can be referred to the above introduction, and will not be described in detail. On this basis, please refer to Figure 2 , Figure 2 for the process schematic diagram provided by the second embodiment of the application for constructing a city image based on simulated aerial photography.

[0070] As Figure 2 indicated, in the embodiment of the application, the step of performing global integration according to the virtual imaging viewpoint to obtain a virtual imaging surface covering the field of view range of the virtual flight path comprises:

[0071] Step S21, fitting based on the virtual imaging viewpoint to determine the central trajectory axis of the virtual flight path;

[0072] Step S22, constructing a complex surface parameterization model based on the central trajectory axis.

[0073] It should be noted that the aforementioned central trajectory axis can be a reference spatial curve determined by path smoothing and fitting based on the virtual flight path, used to construct a parametric model of the complex surface. By using the continuously differentiable central trajectory axis as the reference spatial axis, the parametric model of the complex surface can be determined through the radial distance function. Specifically, the original virtual flight path consists of a series of points and may contain some high-frequency jitter. By smoothing and filtering or curve fitting the virtual flight path, flight control noise and minor fluctuations can be eliminated, thus obtaining the central trajectory axis. Based on this central trajectory axis, the parametric model of the complex surface can be constructed.

[0074] It should be explained that the above parametric model of the complex surface can be a parametric representation of the complex surface (i.e., the virtual imaging surface). Points on the parametric model of the complex surface can be represented as follows:

[0075] ;

[0076] in, The arc length on the central trajectory axis is... point The corresponding position vector, This represents the position vector of a point on the parametric model of the complex surface; Indicates at point A direction vector within the normal plane. and The angle between them The range of the value is [0, 2π). . The radius function is used to represent a point on the central trajectory axis. At that location, along the direction vector The direction, from point Points on the parametric model of the complex surface The corresponding distance.

[0077] It should be noted that for points on the central trajectory axis A Frenet frame can be defined, which can consist of three mutually orthogonal unit vector fields. Let Frénet's frame have a unit principal normal field, the direction of which is a point on the central trajectory axis. The point is located in the direction pointing towards the center of curvature, with a length equal to the axis of the central trajectory. The curvature at that point. For the Frenier frame, the unit tangent vector field is specifically the axis of the central trajectory at point [point missing]. The unit tangential vector at that location. Denotes the unit binormal vector field of the Frenet frame. , with , constitute a set of orthogonal unit bases in space.

[0078] In practical applications, the UAV aerial photography or simulated aerial photography can be regarded as a series of local images with perspective limitations collected at different times and different spatial positions (i.e., discrete virtual imaging viewpoints). The virtual imaging curve surface can map any point in the time sequence image rendered by the virtual imaging viewpoint to a unified complex surface, avoiding the ghosting and double imaging phenomena caused by the mismatch of the projection plane, the cumulative error of image alignment, and the like in the traditional splicing method.

[0079] In step S23, the virtual imaging viewpoint is mapped to the complex surface parameterization model to obtain the viewpoint parameter coordinates and the normal direction of the virtual imaging viewpoint.

[0080] In step S24, global integration is performed based on the viewpoint parameter coordinates and the normal direction to obtain a virtual imaging curve surface covering the field of view range of the virtual flight path.

[0081] It can be understood that, based on the conversion formula of the point on the complex surface parameterization model and any point on the central trajectory axis, the virtual imaging viewpoint can be mapped to the complex surface parameterization model.

[0082] It should be noted that, for each virtual imaging viewpoint, its three-dimensional space coordinates and line-of-sight direction vector can be determined. The three-dimensional space coordinates are the coordinate positions of the virtual imaging viewpoint in the three-dimensional city model, and the line-of-sight direction vector can be understood as the attitude angle of the UAV in the air, and specifically can represent the direction of the UAV in the air. According to the three-dimensional space coordinates and the line-of-sight direction vector of each virtual imaging viewpoint, these discrete virtual imaging viewpoints can be mapped to the complex surface parameterization model to obtain the corresponding viewpoint parameter coordinates and normal direction. That is, the step of mapping the virtual imaging viewpoint to the complex surface parameterization model to obtain the viewpoint parameter coordinates and the normal direction of the virtual imaging viewpoint includes: obtaining the three-dimensional space coordinates and the line-of-sight direction vector of the virtual imaging viewpoint; and mapping the virtual imaging viewpoint to the complex surface parameterization model according to the three-dimensional space coordinates and the line-of-sight direction vector to obtain the viewpoint parameter coordinates and the normal direction of the virtual imaging viewpoint.

[0083] In some embodiments of the present application, for a virtual imaging viewpoint The vertical projection point of the virtual imaging viewpoint on the central trajectory axis can be determined based on the three-dimensional space coordinates of the virtual imaging viewpoint, so that the distance of the virtual imaging viewpoint to the central trajectory axis is the shortest. When the vertical projection point is determined, the arc length corresponding to the vertical projection point can be determined, and the arc length parameter is the axial parameter corresponding to the virtual imaging viewpoint. Based on the three-dimensional space coordinates of the virtual imaging viewpoint, the vertical projection point corresponding to the virtual imaging viewpoint and the axial parameter, the azimuth angle around the axis corresponding to the virtual imaging viewpoint and the axial distance can be determined. The axial distance is the length of the line segment formed by connecting the virtual imaging viewpoint and the vertical projection point, and the azimuth angle around the axis can be determined based on the vertical projection point to establish a Frei frame. Specifically, the Frei frame can be established based on the vertical projection point, and the azimuth angle around the axis is the angle between the vector formed by the virtual imaging viewpoint and the vertical projection point and the tangent plane (a plane composed of a unit principal normal vector field and a unit secondary normal vector field). Through the above operation, any point can be converted into a set of parameters . indicates the arc length of the vertical projection point corresponding to the point, indicates the azimuth angle around the axis (which side of the central trajectory axis the point is on), indicates the axial distance (which indicates the distance of the point to the central trajectory axis), indicates the expected surface normal direction (opposite to the direction of the line-of-sight vector, which indicates the direction in which the desired complex surface should face).

[0084] In some embodiments of the embodiments of the present application, for the azimuth angle around the axis and the axial distance corresponding to the virtual imaging viewpoint, the corresponding point on the complex surface can be found. Specifically, through the above parameter set corresponding to each virtual imaging viewpoint , the radial distance function can be fitted, so that the function value of the radial distance function is as close as possible to the distance , and the actual normal direction calculated by the complex surface at the point is as close as possible to the expected surface normal direction , and then any point can be mapped to the virtual imaging surface based on the complex surface parameterization model.

[0085] The embodiment of the application determines the central trajectory axis of the virtual flight path by fitting based on the virtual imaging viewpoint; constructs a complex surface parameterization model based on the central trajectory axis; maps the virtual imaging viewpoint to the complex surface parameterization model to obtain the viewpoint parameter coordinates and the normal direction of the virtual imaging viewpoint; and performs global integration based on the viewpoint parameter coordinates and the normal direction to obtain the virtual imaging surface covering the field of view of the virtual flight path. By using discrete virtual imaging viewpoints to perform curve fitting, the central trajectory axis is determined, noise in the viewpoint data is reduced, and sampling error is reduced. By constructing a complex surface parameterization model, the problem of constructing a three-dimensional curved surface is converted into a problem of constructing a two-dimensional parameter domain determining a radial distance function , the calculation dimension and the calculation complexity are reduced.

[0086] Based on the first embodiment and / or the second embodiment of the application, in the third embodiment of the application, the same or similar contents as those in the above-mentioned first embodiment and / or the second embodiment can be referred to the above description, and the subsequent description will not be repeated. On this basis, please refer to Figure 3 , Figure 3 for the flowchart provided by the third embodiment of the application for constructing a city image based on simulated aerial photography.

[0087] As shown in Figure 3 , in the embodiment of the application, the step of performing virtual image sampling in the three-dimensional city model based on the virtual imaging viewpoint and the virtual imaging surface to obtain the time sequence image corresponding to the virtual imaging viewpoint includes:

[0088] Step S31: performing virtual image sampling in the three-dimensional city model based on the virtual imaging viewpoint to obtain an initial viewpoint image sequence;

[0089] Step S32: performing re-projection correction on the initial viewpoint image sequence based on the virtual imaging surface to obtain the time sequence image corresponding to the virtual imaging viewpoint.

[0090] It should be noted that for each virtual imaging viewpoint, virtual image sampling can be performed in the three-dimensional city model to obtain an initial viewpoint image sequence, and by performing re-projection correction of these initial viewpoint image sequences on the virtual imaging surface, the time sequence image corresponding to the virtual imaging viewpoint can be obtained.

[0091] In some embodiments of the application, the step of performing fusion based on the time sequence image corresponding to the virtual imaging viewpoint to obtain the virtual city image of the simulated aerial photography task includes: performing fusion based on the overlap area statistical information of the time sequence image corresponding to the virtual imaging viewpoint to obtain a panoramic virtual viewpoint image; and generating the virtual city image of the simulated aerial photography task based on the panoramic virtual viewpoint image.

[0092] It should be noted that for the time sequence images projected to the virtual imaging surface, there is an overlap of the fields of view of multiple adjacent virtual imaging viewpoints on the virtual imaging surface. According to the overlap information, the time sequence images corresponding to adjacent virtual imaging viewpoints can be fused to obtain a panoramic virtual viewpoint image. In the fusion, consistency enhancement, edge preservation, smooth transition, etc. can be used to improve the effect of the panoramic viewpoint image. Based on the finally generated panoramic virtual viewpoint image, the dynamic virtual city image can be generated based on the task parameters such as video generation frame rate, resolution, time length, etc. The specific way of image fusion and image generation is not limited in the embodiments of the present application, and can be set according to the needs in actual application.

[0093] The embodiments of the present application obtain an initial viewpoint image sequence by sampling virtual images based on virtual imaging viewpoints in a three-dimensional city model, correct the initial viewpoint image sequence based on a virtual imaging surface to obtain time sequence images corresponding to the virtual imaging viewpoints. The time sequence images corresponding to the virtual imaging viewpoints are fused based on the overlap region statistical information to obtain a panoramic virtual viewpoint image, and the virtual city image simulating aerial photography is generated based on the panoramic virtual viewpoint image. Since the initial viewpoint image sequence is corrected by re-projection, the deformation in the original image is eliminated; by fusing the time sequence images of different virtual imaging viewpoints, the consistency of the generated image is improved, and the quality of the generated image is improved.

[0094] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the city image construction method based on simulated aerial photography of the present application. More forms of simple transformation based on this technical concept are within the protection scope of the present application.

[0095] The present application also provides a city image construction device based on simulated aerial photography, which is described with reference to Figure 4 , Figure 4 FIG. 1 is a module structure schematic diagram of a city image construction device based on simulated aerial photography according to an embodiment of the present application. The city image construction device based on simulated aerial photography includes:

[0096] A viewpoint generation module 10 is configured to generate a virtual flight path based on a simulated aerial photography task of a user, and discretely sample the virtual flight path to obtain virtual imaging viewpoints.

[0097] A panoramic management module 20 is configured to globally integrate the virtual imaging viewpoints to obtain a virtual imaging surface covering the field of view of the virtual flight path.

[0098] The image acquisition module 30 is configured to perform virtual image sampling in the three-dimensional city model based on the virtual imaging viewpoint and the virtual imaging curved surface, to obtain time-series images corresponding to the virtual imaging viewpoint.

[0099] The image generation module 40 is configured to obtain virtual city images of the simulated aerial photography task based on the time-series images corresponding to the virtual imaging viewpoint.

[0100] The city image construction device based on simulated aerial photography provided in the present application adopts the city image construction method based on simulated aerial photography in the above embodiment, and can solve the technical problem that the existing unmanned aerial vehicle aerial photography directly performs field shooting, which is easy to cause damage and is greatly affected by the outside. Compared with the prior art, the city image construction device based on simulated aerial photography provided in the present application has the same beneficial effects as the city image construction method based on simulated aerial photography provided in the above embodiment, and other technical features in the city image construction device based on simulated aerial photography are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0101] The present application provides a city image construction device based on simulated aerial photography, which comprises at least one processor and a memory in communication connection with the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the city image construction method based on simulated aerial photography in the above embodiment one.

[0102] Reference will be made to the following description Figure 5 which shows a structure schematic diagram of the city image construction device based on simulated aerial photography suitable for being used to implement the embodiments of the present application. The city image construction device based on simulated aerial photography in the embodiments of the present application can include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistant), PADs (Portable Application Description), PMPs (Portable Media Player), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and the like, and fixed terminals such as digital TVs, desktop computers, and the like. Figure 5 The city image construction device based on simulated aerial photography shown is only an example, and should not bring any limitation to the functions and use range of the embodiments of the present application.

[0103] As Figure 5As shown, the aerial-simulated city image construction device can include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to programs stored in a Read Only Memory (ROM) 1002 or programs loaded from a storage device 1003 into a Random Access Memory (RAM) 1004. In the RAM 1004, various programs and data required for the aerial-simulated city image construction device to operate are also stored. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, etc.; the storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the aerial-simulated city image construction device to communicate with other devices wirelessly or by wire to exchange data. Although the aerial-simulated city image construction device with various systems is shown in the figure, it should be understood that all the shown systems are not required to be implemented or possessed. More or fewer systems can be alternatively implemented or possessed.

[0104] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through a communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the methods of the embodiments of the present disclosure are performed.

[0105] The city image construction device based on simulated aerial photography provided by the application adopts the city image construction method based on simulated aerial photography in the above embodiment, and can solve the technical problem that the existing unmanned aerial vehicle aerial photography directly performs field shooting, which is easy to cause damage and is greatly affected by the outside. Compared with the prior art, the beneficial effects of the city image construction device based on simulated aerial photography provided by the application are the same as those of the city image construction method based on simulated aerial photography provided by the above embodiment, and other technical features in the city image construction device based on simulated aerial photography are the same as those disclosed in the above embodiment method, and will not be repeated here.

[0106] It should be understood that parts of the present application can be realized by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0107] The above is merely specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0108] The present application provides a computer readable storage medium having computer readable program instructions (i.e. computer programs) stored thereon, the computer readable program instructions being used to execute the city image construction method based on simulated aerial photography in the above embodiment.

[0109] The computer readable storage medium provided in the application may be, for example, a U disk, but is not limited to an electric, magnetic, optical, electromagnetic, infrared, or semiconductor system, system, or device, or any combination of the above. More specific examples of the computer readable storage medium may include, but are not limited to, an electric connection with one or more conductive wires, a portable computer disk, a hard disk, a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM) or a flash memory, an optical fiber, a portable compact disk read only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the embodiment, the computer readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system, or device. The program code contained on the computer readable storage medium can be transmitted by any suitable medium, including but not limited to an electric wire, an optical cable, an RF (Radio Frequency, RF), and the like, or any suitable combination of the above.

[0110] The computer readable storage medium described above may be contained in a device for constructing city images based on simulated aerial photography, or may exist separately without being assembled into the device for constructing city images based on simulated aerial photography.

[0111] The computer readable storage medium described above carries one or more programs, which, when executed by the device for constructing city images based on simulated aerial photography, cause the device for constructing city images based on simulated aerial photography to:

[0112] generate a virtual flight path based on the simulated aerial photography task of the user, and discretely sample on the virtual flight path to obtain virtual imaging viewpoints;

[0113] globally integrate according to the virtual imaging viewpoints to obtain a virtual imaging surface covering the field of view range of the virtual flight path;

[0114] perform virtual image sampling in a three-dimensional city model based on the virtual imaging viewpoints and the virtual imaging surface to obtain time sequence images corresponding to the virtual imaging viewpoints;

[0115] fuse the time sequence images corresponding to the virtual imaging viewpoints to obtain virtual city images of the simulated aerial photography task.

[0116] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0117] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0118] The modules involved in the embodiments of the present application can be implemented in software or hardware. In some cases, the names of the modules do not limit the modules themselves.

[0119] The readable storage medium provided by the present application is a computer readable storage medium, which stores computer readable program instructions (i.e. computer programs) for executing the above-mentioned city image construction method based on simulated aerial photography. The technical problems that the existing unmanned aerial vehicle aerial photography directly performs field shooting, which is easy to cause damage and is greatly affected by the outside world, can be solved. Compared with the prior art, the computer readable storage medium provided by the present application has the same beneficial effects as the city image construction method based on simulated aerial photography provided by the above-mentioned embodiments. Details are not repeated here.

[0120] The application also provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of the method for constructing city images based on simulated aerial photography as described above.

[0121] The computer program product provided by the application can solve the technical problem that the existing unmanned aerial vehicle aerial photography directly carries out field shooting, which is easy to cause damage and is greatly affected by the outside. Compared with the prior art, the beneficial effects of the computer program product provided by the application are the same as those of the method for constructing city images based on simulated aerial photography provided by the above-mentioned embodiments, and are not described here.

[0122] The above only describes some embodiments of the application, and does not limit the protection scope of the application, and any equivalent structural transformation made by using the content of the application specification and drawings, or direct / indirect application in other related technical fields is included in the protection scope of the application.

Claims

1. A method for constructing urban images based on simulated aerial photography, characterized in that, The method includes: A virtual flight path is generated based on the user's simulated aerial photography mission, and discretized sampling is performed on the virtual flight path to obtain a virtual imaging viewpoint; Based on the virtual imaging viewpoint, a virtual imaging surface covering the field of view of the virtual flight path is obtained through global integration. Virtual image sampling is performed in a 3D city model based on the virtual imaging viewpoint and the virtual imaging surface to obtain a time series image corresponding to the virtual imaging viewpoint; The virtual city image of the simulated aerial photography mission is obtained by fusing the time-series images corresponding to the virtual imaging viewpoint. The step of globally integrating the virtual imaging viewpoints to obtain a virtual imaging surface covering the field of view of the virtual flight path includes: The center trajectory axis of the virtual flight path is determined by fitting the virtual imaging viewpoint. A parametric model of the complex surface is constructed based on the central trajectory axis; The virtual imaging viewpoint is mapped to the complex surface parameterization model to obtain the viewpoint parameter coordinates and normal direction of the virtual imaging viewpoint. Based on the viewpoint parameter coordinates and the normal direction, a virtual imaging surface covering the field of view of the virtual flight path is obtained through global integration.

2. The urban image construction method based on simulated aerial photography as described in claim 1, characterized in that, The step of mapping the virtual imaging viewpoint to the parametric model of the complex surface to obtain the viewpoint parameter coordinates and normal direction of the virtual imaging viewpoint includes: Obtain the three-dimensional spatial coordinates and the line-of-sight vector of the virtual imaging viewpoint; The virtual imaging viewpoint is mapped to the complex surface parameterized model based on the three-dimensional spatial coordinates and the line-of-sight vector to obtain the viewpoint parameter coordinates and normal direction of the virtual imaging viewpoint.

3. The urban image construction method based on simulated aerial photography as described in claim 1, characterized in that, The step of performing virtual image sampling in a 3D city model based on the virtual imaging viewpoint and the virtual imaging surface to obtain a time-series image corresponding to the virtual imaging viewpoint includes: Based on the virtual imaging viewpoint, virtual image sampling is performed in a 3D city model to obtain an initial viewpoint image sequence; Based on the virtual imaging surface, the initial viewpoint image sequence is reprojected and corrected to obtain the time-series image corresponding to the virtual imaging viewpoint.

4. The urban image construction method based on simulated aerial photography as described in claim 1, characterized in that, The step of fusing the time-series images corresponding to the virtual imaging viewpoint to obtain the virtual city image of the simulated aerial photography mission includes: The panoramic virtual viewpoint image is obtained by fusing statistical information of overlapping regions of time-series images corresponding to the virtual imaging viewpoint. The virtual city imagery for the simulated aerial photography mission is generated based on the panoramic virtual viewpoint image.

5. The urban image construction method based on simulated aerial photography as described in claim 1, characterized in that, The step of discretizing sampling along the virtual flight path to obtain a virtual imaging viewpoint includes: The flight parameters of the UAV are obtained based on the simulated aerial photography mission; Based on the number of generated video frames and the flight parameters, discretized sampling is performed to determine the virtual imaging viewpoint on the virtual flight path.

6. A device for constructing urban images based on simulated aerial photography, characterized in that, The urban image construction device based on simulated aerial photography includes: The viewpoint generation module is used to generate a virtual flight path based on the user's simulated aerial photography mission, and to perform discretization sampling on the virtual flight path to obtain a virtual imaging viewpoint; The panoramic management module is used to perform global integration based on the virtual imaging viewpoint to obtain a virtual imaging surface that covers the field of view of the virtual flight path; The image acquisition module is used to perform virtual image sampling in a three-dimensional city model based on the virtual imaging viewpoint and the virtual imaging surface to obtain a time series image corresponding to the virtual imaging viewpoint; The image generation module is used to fuse time-series images corresponding to the virtual imaging viewpoint to obtain virtual city images of the simulated aerial photography mission. The panoramic management module is further configured to: fit the virtual imaging viewpoint to determine the central trajectory axis of the virtual flight path; construct a complex surface parameterized model based on the central trajectory axis; map the virtual imaging viewpoint to the complex surface parameterized model to obtain the viewpoint parameter coordinates and normal direction of the virtual imaging viewpoint; and globally integrate the viewpoint parameter coordinates and normal direction to obtain a virtual imaging surface covering the field of view of the virtual flight path.

7. A device for constructing urban images based on simulated aerial photography, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the urban image construction method based on simulated aerial photography as described in any one of claims 1 to 5.

8. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the urban image construction method based on simulated aerial photography as described in any one of claims 1 to 5.

9. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the urban image construction method based on simulated aerial photography as described in any one of claims 1 to 5.

Images