Aircraft simulation training method and system based on virtual-real combination

Abstract

The invention provides an aircraft simulation training method and system based on virtual-real combination, and relates to the technical field of aircraft simulation training. Accurate fusion of the actual optoelectronic device and the virtual scene is realized. Firstly, introduction of geographic information data provides a real scene basis for training; secondly, distortion correction of the two-dimensional picture ensures the accuracy of live-action imaging; and finally, by creating a virtual camera which is the same as the aircraft in position and has the same field angle, the system can always keep an output picture in an actual video range, and strictly follows the first view angle of a driver. According to the virtual-real combination mode based on position synchronization and visual angle matching, the real imaging characteristics of actual optoelectronic equipment are reserved, accurate superposition of the virtual targets is achieved, the real visual effect and the accurate space relation of training images are kept all the time, and therefore the vivid training experience is provided while the training cost is reduced.

Description

Technical Field

[0001] The present application relates to the technical field of aircraft simulation training, and in particular to an aircraft simulation training method and system based on the combination of virtual and real. Background Art

[0002] In military training, aircraft pilots need to master the operation skills of optoelectronic equipment. In actual combat environments, pilots not only need to control the flight status of the aircraft, but also need to accurately identify and track ground targets through onboard optoelectronic equipment. This complex collaborative operation places extremely high demands on the pilot's professional skills.

[0003] Currently, there are two main ways to train aircraft pilots: one is actual flight training, in which the pilot drives a real aircraft and uses actual optoelectronic equipment for target observation training; the other is simulator training, in which the pilot operates in a ground simulator and the system simulates flight scenes and targets through computer graphics technology.

[0004] However, related technologies have certain problems to a greater or lesser extent. For example, although actual flight training can provide a real operating experience, the training cost is high, and it is difficult to conduct continuous training due to external conditions such as weather and airspace. Although simulator training can be carried out at any time, due to the use of a completely virtual training environment, the observation images generated by it are quite different from the imaging effects of actual optoelectronic equipment, which affects the training effect. Therefore, the low training cost and high authenticity of related technologies cannot coexist at present. Summary of the invention

[0005] The present application provides an aircraft simulation training method and system based on the combination of virtual and real, which is used to reduce the training cost while ensuring the authenticity of the training images.

[0006] In the first aspect, the present application provides an aircraft simulation training method based on the combination of virtual and real, including: obtaining geographic information data, and using the geographic information data as a layer to reconstruct a three-dimensional scene of the aircraft simulation training scene area; establishing a mapping relationship between a two-dimensional image obtained by the aircraft's optoelectronic equipment and the three-dimensional scene; performing distortion correction on the two-dimensional image; generating a virtual target in the three-dimensional scene; creating a virtual camera with the same position as the aircraft and the same field of view as the projection direction of the two-dimensional image, and collecting a fused image of the corrected two-dimensional image and the virtual target; and performing aircraft simulation training according to the fused image.

[0007] By adopting the above technical solutions, the precise integration of actual optoelectronic equipment and virtual scenes is achieved. First, the introduction of geographic information data provides a real scene basis for training; second, the distortion correction of the two-dimensional picture ensures the accuracy of real-scene imaging; finally, by creating a virtual camera with the same position and field of view as the aircraft, the system can always keep the output picture within the actual video range and strictly follow the first-person perspective of the pilot. This virtual-real combination based on position synchronization and perspective matching not only retains the real imaging characteristics of the actual optoelectronic equipment, but also realizes the precise superposition of virtual targets, so that the training picture always maintains the real visual effect and accurate spatial relationship, thereby reducing the training cost while providing a realistic training experience.

[0008] In conjunction with some embodiments of the first aspect, in some embodiments, the step of establishing a mapping relationship between a two-dimensional image acquired by an optoelectronic device of an aircraft and a three-dimensional scene specifically includes: In the formula, is the coordinate of the three-dimensional scene; M lla2ecef The matrix for converting the longitude and latitude coordinate system to the three-dimensional scene; M ENU_WITHOUT_TRANS The coordinate transformation matrix that converts the three-dimensional scene corresponding to the longitude and latitude of the aircraft into the northeast sky coordinate system at the longitude and latitude without translation operation; M RP is the rotation matrix of the aircraft relative to its own coordinate system; M TC is the rotation matrix of the optoelectronic device relative to its own coordinate system; d E The straight-line distance between the three-dimensional coordinates corresponding to the current pixel and the origin of the optoelectronic device; is the affine coordinate identifier of the back-projected position of the pixel in the viewing frustum; A three-dimensional vector consisting of the xyz components of the affine coordinates of the back-projected position of the pixel in the viewing frustum; T clip_h_w A scalar consisting of the w component of the radial coordinates of the back-projected position of the pixel in the viewing frustum; is the back-projected position of the pixel in the local coordinate system of the viewing frustum; r width is the resolution of the optoelectronic device width; r height The resolution of the optoelectronic device height; fovH is the horizontal field of view of the optoelectronic device; T xThe horizontal coordinates of the pixel coordinates of the required pixel point; T y The vertical coordinate of the pixel coordinate of the required pixel point.

[0009] In combination with some embodiments of the first aspect, in some embodiments, the step of correcting the distortion of a two-dimensional image specifically includes: adjusting the position and posture of a calibration object, collecting a calibration image containing the calibration object at each preset position and posture to obtain multiple calibration images; the calibration object has a preset grid size and number of rows and columns; detecting the corner points of the grid in the calibration image, and obtaining the pixel coordinate values ​​of the corner points; calculating the physical coordinate values ​​of the corner points according to the actual size parameters of the calibration object, the position relationship in the world coordinate system, and the pixel coordinate values ​​of the corner points; calculating the intrinsic parameters and extrinsic parameters of the optoelectronic device based on the pixel coordinate values ​​and the physical coordinate values, under the condition that the distortion coefficient of the optoelectronic device is zero; while considering the distortion of the optoelectronic device lens, optimizing the intrinsic parameters and extrinsic parameters by using the likelihood estimation method, so that the error between the pixel coordinate value of the corner point and the theoretical pixel coordinate value obtained by projecting the physical coordinate value of the corner point based on the intrinsic parameters and the extrinsic parameters is minimized; solving the radial distortion coefficient by using the least squares method; and correcting the distortion of the two-dimensional image based on the optimized intrinsic parameters, the optimized extrinsic parameters, and the radial distortion coefficient.

[0010] By adopting the above technical solution, first of all, sufficient calibration data samples are obtained by collecting images of the calibration object at multiple angles and multiple positions. During the calibration process, the system first calculates the basic parameters of the optoelectronic device under ideal distortion-free conditions, then optimizes the internal and external parameters through the likelihood estimation method, and finally uses the least squares method to solve the radial distortion coefficient, effectively overcoming the inevitable optical errors of optoelectronic equipment in the production process, improving the distortion problem in the edge area of ​​the picture, and providing a high-quality real scene foundation for the subsequent virtual-real fusion.

[0011] In combination with some embodiments of the first aspect, in some embodiments, before the step of performing distortion correction on the two-dimensional image, the method also includes: dividing the two-dimensional image into image data of three field of view angles: large field of view angle, medium field of view angle and small field of view angle, the observation angles of the large field of view angle, medium field of view angle and small field of view angle decreasing successively; and performing distortion correction on the image data of the three field of view angles respectively.

[0012] By adopting the above technical solution, distortion correction is performed for images with different field of view angles. The system divides the two-dimensional image into three fields of view: large, medium, and small, according to the observation angle, and establishes a correction parameter model for each field of view. This classification processing method overcomes the problem that unified parameters are difficult to adapt to the switching of multiple field of view angles, ensures the accuracy of distortion correction under different observation ranges, and improves the practicality of the system.

[0013] In combination with some embodiments of the first aspect, in some embodiments, the steps of obtaining geographic information data and using the geographic information data as a layer to reconstruct a three-dimensional scene of an aircraft simulation training scene area specifically include: obtaining geographic information system base map data of the aircraft simulation training scene area, the geographic information system base map data including satellite image data, geographic information elevation data and vector geographic data; constructing a layer based on the satellite image data, geographic information elevation data and vector geographic data, and reconstructing the three-dimensional scene of the aircraft simulation training scene area.

[0014] By adopting the above technical solutions, high-precision reconstruction of training scenes is achieved. The system integrates satellite images, elevation and vector geographic data, and constructs three-dimensional scenes in a layered modeling manner. This scene reconstruction method based on multi-source data not only ensures the authenticity and integrity of the scene, but also provides geographic information support, providing a real environment foundation for the addition and interaction of virtual targets.

[0015] In combination with some embodiments of the first aspect, in some embodiments, the steps of creating a virtual camera with the same position as the aircraft and the same field of view as the projection direction of the two-dimensional image, and collecting a fused image of the corrected two-dimensional image and the virtual target specifically include: creating a virtual camera so that the position of the virtual camera is the same as the position of the aircraft and the field of view of the virtual camera is the same as the projection direction of the two-dimensional image; collecting the fused frame image of the corrected two-dimensional image and the virtual target through the virtual camera; superimposing avionics data on the fused frame image to generate a fused frame image with the superimposed avionics data; and outputting the fused frame image with the superimposed avionics data to the display system through the HDMI interface.

[0016] By adopting the above technical solution, by creating a virtual camera that is consistent with the position of the aircraft and synchronizing its field of view with the actual video projection direction, the system ensures that the fused image is always limited to the effective video range. This dual matching mechanism of spatial position and field of view enables the pilot to always observe the training scene from the first-person perspective, ensuring the continuity and consistency of the observation perspective. At the same time, the real-time superposition of avionics data further enhances the information integrity of the training image, allowing pilots to obtain the necessary flight parameters while maintaining a real observation experience. This fusion method based on precise position matching and perspective synchronization not only ensures the natural transition of the virtual and real fusion images, but also restores the visual experience during actual flight to the greatest extent, improving the realism and practicality of the training.

[0017] In combination with some embodiments of the first aspect, in some embodiments, after the step of generating a virtual target in a three-dimensional scene, the method further includes: obtaining position information of a corrected two-dimensional image and the virtual target; superimposing the virtual target on the corrected two-dimensional image based on the position information to generate a two-dimensional fused image superimposed with the virtual target.

[0018] By adopting the above technical solution, the system first obtains the distortion-corrected two-dimensional real scene image and the precise position information of the virtual target, and then directly superimposes the virtual target on the real scene image based on this information. This direct superposition method enables the pilot to clearly observe the virtual target in the actual scene.

[0019] In the second aspect, the present application provides an aircraft simulation training system based on the combination of virtual and real, the aircraft simulation training system based on the combination of virtual and real includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, the one or more processors call the computer instructions to enable the aircraft simulation training system based on the combination of virtual and real to execute the method described in the first aspect and any possible implementation method of the first aspect.

[0020] In a third aspect, the present application provides a computer program product comprising instructions, which, when run on an aircraft simulation training system based on a combination of virtual and real, enables the aircraft simulation training system based on a combination of virtual and real to execute the method described in the first aspect and any possible implementation method of the first aspect.

[0021] In a fourth aspect, the present application provides a computer-readable storage medium comprising instructions, which, when executed on an aircraft simulation training system based on a combination of virtual and real, enables the aircraft simulation training system based on a combination of virtual and real to execute the method described in the first aspect and any possible implementation method of the first aspect.

[0022] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: 1. The precise integration of actual optoelectronic equipment and virtual scenes is achieved. First, the introduction of geographic information data provides a real scene basis for training; second, the distortion correction of the two-dimensional picture ensures the accuracy of real-scene imaging; finally, by creating a virtual camera with the same position and field of view as the aircraft, the system can always keep the output picture within the actual video range and strictly follow the first-person perspective of the pilot. This virtual-real combination based on position synchronization and perspective matching not only retains the real imaging characteristics of the actual optoelectronic equipment, but also realizes the precise superposition of virtual targets, so that the training picture always maintains the real visual effect and accurate spatial relationship, thereby reducing the training cost while providing a realistic training experience.

[0023] 2. First, by collecting images of the calibration object at multiple angles and positions, sufficient calibration data samples were obtained. During the calibration process, the system first calculated the basic parameters of the optoelectronic equipment under ideal distortion-free conditions, then optimized the internal and external parameters through the likelihood estimation method, and finally used the least squares method to solve the radial distortion coefficient, effectively overcoming the inevitable optical errors of optoelectronic equipment in the production process, improving the distortion problem in the edge area of ​​the picture, and providing a high-quality real scene foundation for the subsequent virtual-real fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a flowchart of an aircraft simulation training method based on the combination of virtual and real in an embodiment of the present application; Figure 2 This is a specific flowchart of step S103 in the embodiment of the present application; Figure 3 is a schematic diagram of a problem in an exemplary application scenario in an embodiment of the present application; Figure 4 is another problem schematic diagram in an exemplary application scenario in an embodiment of the present application; Figure 5 This is a schematic diagram of a scenario in step S104 in an embodiment of the present application; Figure 6 This is a schematic diagram of a scenario of step S105 in an embodiment of the present application; Figure 7 This is an effect diagram of the fusion screen superimposed with avionics data in the embodiment of the present application; Figure 8 is a virtual target in the embodiment of the present application; Fig. 9 This is a real scene in the embodiment of this application; Fig.10 It is an effect diagram combining real scenes and virtual targets in an embodiment of the present application.

[0025] Fig.11 It is a schematic diagram of an exemplary hardware structure of an aircraft simulation training system based on the combination of virtual and real in an embodiment of the present application. DETAILED DESCRIPTION

[0026] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification and appended claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include plural expressions, unless there is a clear indication to the contrary in the context. It should also be understood that the term "and / or" used in the present application refers to and includes any or all possible combinations of one or more listed items.

[0027] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0028] See also Figure 1 , Figure 1 It is a flowchart of an aircraft simulation training method based on the combination of virtual and real in an embodiment of the present application; S101, acquiring geographic information data, and using the geographic information data as a layer to perform three-dimensional scene reconstruction of an aircraft simulation training scene area; Among them, geographic information data refers to digital information that describes the surface features of the earth. For example, in some embodiments, it includes satellite image data, elevation data, and vector geographic data; layers represent data structures that organize different types of geographic information in a hierarchical manner; three-dimensional scene reconstruction refers to the use of geographic information data to construct a virtual three-dimensional environment with real geographic features; and training scene areas represent specific geographical areas for flight training.

[0029] This step is performed before the simulation training begins to build a basic training environment. In some embodiments, the system first obtains multi-source geographic data of the training area from the geographic information system database, then processes these data in layers according to different geographic elements, and finally integrates these layer data into a complete three-dimensional virtual scene through three-dimensional modeling technology for subsequent training.

[0030] In some embodiments, step S101 specifically includes: S1011, obtaining geographic information system base map data of the aircraft simulation training scene area, where the geographic information system base map data includes satellite image data, geographic information elevation data, and vector geographic data; Geographic information system base map data refers to comprehensive digital information describing the characteristics of the earth's surface; satellite image data refers to surface image information taken from the air; geographic information elevation data refers to numerical data describing the changes in terrain; vector geographic data refers to geographic element data expressed in geometric forms such as points, lines, and surfaces; S1012: construct layers based on satellite image data, geographic information elevation data and vector geographic data, and perform three-dimensional scene reconstruction of the aircraft simulation training scene area.

[0031] In some embodiments, the system first processes different types of geographic data according to their characteristics and creates independent layers. The terrain layer uses elevation data to generate surface undulations; the image layer uses satellite images as surface textures to cover the terrain; the building layer performs three-dimensional modeling based on vector data; the vegetation layer arranges the vegetation model by identifying image features; and elements such as roads and water systems also construct corresponding layers. Finally, all layers are combined according to the correct spatial relationship to form a complete three-dimensional scene model.

[0032] In some specific embodiments, a terrain model is generated using a triangular mesh algorithm; building outlines are extracted and stretched for modeling; surface classification is performed based on image features; road and river networks are constructed; and vegetation and landscape elements are added, which are not limited here.

[0033] It can be seen that the high-precision reconstruction of the training scene has been achieved. The system integrates satellite images, elevation and vector geographic data, and uses a layered modeling approach to construct a three-dimensional scene. This scene reconstruction method based on multi-source data not only ensures the authenticity and integrity of the scene, but also provides geographic information support, providing a real environment foundation for the addition and interaction of virtual targets.

[0034] S102, establishing a mapping relationship between a two-dimensional image acquired by an optoelectronic device of the aircraft and a three-dimensional scene; Among them, the optoelectronic device refers to a sensor device for acquiring images, and in some embodiments, it is an optoelectronic turret; the two-dimensional picture represents the real-time video image collected by the optoelectronic device; the mapping relationship refers to the mathematical correspondence between the two-dimensional plane coordinate system and the three-dimensional space coordinate system.

[0035] It should be noted that a three-dimensional scene refers to a virtual environment with three-dimensional spatial characteristics, and a two-dimensional picture refers to a planar projection image of the three-dimensional real world at a specific time and perspective.

[0036] This step is performed after acquiring real-time video images and constructing a three-dimensional scene. In some embodiments, the system establishes a coordinate transformation matrix to establish a one-to-one correspondence between each pixel point in the two-dimensional image and the corresponding position in the three-dimensional scene, thereby achieving accurate conversion from the two-dimensional image space to the three-dimensional scene space, laying the foundation for subsequent virtual-real fusion.

[0037] In some specific embodiments, it is possible to: 1) establish a two-dimensional image plane coordinate system; 2) establish a three-dimensional world coordinate system; 3) calculate a coordinate system transformation matrix; 4) establish pixel point correspondence; 5) verify mapping accuracy, which is not limited here.

[0038] In some specific embodiments, step S102 specifically includes: In the formula, is the coordinate of the three-dimensional scene; M lla2ecef The matrix for converting the longitude and latitude coordinate system to the three-dimensional scene; M ENU_WITHOUT_TRANS The coordinate transformation matrix that converts the three-dimensional scene corresponding to the longitude and latitude of the aircraft into the northeast sky coordinate system at the longitude and latitude without translation operation; M RP is the rotation matrix of the aircraft relative to its own coordinate system; M TC is the rotation matrix of the optoelectronic device relative to its own coordinate system; d E The straight-line distance between the three-dimensional coordinates corresponding to the current pixel and the origin of the optoelectronic device; is the affine coordinate identifier of the back-projected position of the pixel in the viewing frustum; A three-dimensional vector consisting of the xyz components of the affine coordinates of the back-projected position of the pixel in the viewing frustum; T clip_h_w A scalar consisting of the w component of the radial coordinates of the back-projected position of the pixel in the viewing frustum; is the back-projected position of the pixel in the local coordinate system of the viewing frustum; r width is the resolution of the optoelectronic device width; r height The resolution of the optoelectronic device height; fovH is the horizontal field of view of the optoelectronic device; T x The horizontal coordinates of the pixel coordinates of the required pixel point; T y The vertical coordinate of the pixel coordinate of the required pixel point.

[0039] S103, performing distortion correction on the two-dimensional image; In actual use, because the aircraft's optoelectronic equipment will inevitably have optical errors and optical distortion problems during the production and manufacturing process, there will be errors if it is calculated completely according to the theory. In particular, the errors may be greater at the edges of the video screen, so it is necessary to correct the distortion of the video screen.

[0040] See also Figure 2 , Figure 2 This is a specific flowchart of step S103 in the embodiment of the present application; In some embodiments, step S103 specifically includes: S1031, adjusting the position and posture of the calibration object, and acquiring a calibration image containing the calibration object at each preset position and posture to obtain multiple calibration images; the calibration object has a preset grid size and number of rows and columns; Among them, the calibration object refers to a calibration tool with known geometric features, usually a black and white checkerboard pattern; the preset position and posture represent different pre-planned spatial position and angle combinations; the calibration image refers to the captured image containing the calibration object; the grid size represents the actual size of each square on the calibration board; the number of rows and columns represents the horizontal and vertical number of grids on the calibration board.

[0041] This step is performed first when starting the camera calibration process. In some embodiments, the system collects multiple sets of images containing the complete calibration plate by placing the calibration plate at different distances, angles, and heights.

[0042] S1032, detecting corner points of the grid in the calibration image, and obtaining pixel coordinate values ​​of the corner points; Among them, the pixel coordinate value represents the two-dimensional coordinate of the corner point on the image plane; detection refers to the process of identifying and locating the corner point through image processing algorithms.

[0043] In some embodiments, the system preprocesses each calibration image, including image enhancement and noise reduction, and then uses a special corner detection algorithm to accurately locate the position of each grid intersection, and finally converts the position information of these corner points into numerical values ​​in a pixel coordinate system.

[0044] S1033, calculating the physical coordinate values ​​of the corner points according to the actual size parameters of the calibration object, the positional relationship in the world coordinate system, and the pixel coordinate values ​​of the corner points; Among them, the actual size parameter refers to the physical size data of the measured object; the world coordinate system represents the reference coordinate system of the three-dimensional space; and the physical coordinate value refers to the actual position of the corner point in the world coordinate system.

[0045] In some embodiments, the system first establishes a world coordinate system, sets a corner point of the calibration plate as the origin, and then calculates the three-dimensional coordinates of each corner point in the world coordinate system based on the actual size and grid spacing of the calibration plate, and establishes a corresponding relationship between the pixel coordinate system and the world coordinate system.

[0046] In some specific embodiments, a standard world coordinate system is established; the calibration plate plane equation is calculated; the corner point spacing is determined; a coordinate transformation matrix is ​​established; and the physical coordinates of all corner points are calculated, which is not limited here.

[0047] In other specific embodiments, the origin of the coordinate system is set; the actual physical dimensions are measured; the relative position relationship is calculated; a coordinate mapping table is established; and the coordinate accuracy is verified, which is not limited here.

[0048] S1034, calculating the intrinsic parameters and extrinsic parameters of the optoelectronic device based on the pixel coordinate values ​​and the physical coordinate values, under the condition that the distortion coefficient of the optoelectronic device is zero; Among them, intrinsic parameters refer to parameters that describe the internal geometry and optical characteristics of the camera; extrinsic parameters represent the position and posture of the camera in the world coordinate system; and distortion coefficients refer to parameters that describe the degree of lens distortion.

[0049] In some embodiments, the system first assumes that the camera is distortion-free and establishes a pinhole camera model. Then, by solving the projection equation, it calculates the camera's internal parameters such as focal length and principal point coordinates, as well as external parameters such as the camera's rotation matrix and translation vector relative to the world coordinate system.

[0050] In some specific embodiments, a projection equation group is established; a homography matrix is ​​constructed; internal and external parameters are decomposed; parameter estimation is initialized; and a reprojection error is calculated, which is not limited here.

[0051] In some other specific embodiments, a camera model is established; a constraint equation is constructed; a system of linear equations is solved; an initial solution is calculated; and parameter accuracy is evaluated, which is not limited here.

[0052] S1035. Taking into account the lens distortion of the optoelectronic device, optimize the intrinsic parameters and the extrinsic parameters using a likelihood estimation method so that the error between the pixel coordinate value of the corner point and the theoretical pixel coordinate value obtained by projecting the physical coordinate value of the corner point based on the intrinsic parameters and the extrinsic parameters is minimized. In some embodiments, the system takes into account the distortion of the actual lens, adopts the maximum likelihood estimation method, and continuously adjusts the internal and external parameters through iterative optimization until the projection error is minimized, thereby obtaining more accurate camera parameters.

[0053] In some specific embodiments, first, multiple sets of images of the calibration plate are obtained, and the coordinate information of the corner points in each image is extracted. According to the actual size of the calibration plate, the physical coordinate values ​​of the corner points in the world coordinate system can be obtained, and the pixel coordinate values ​​corresponding to these corner points in the image can also be obtained.

[0054] Secondly, according to the camera imaging principle, the initial estimation of the internal and external parameters is established. Based on the obtained internal and external parameters, the projection model is used to project the physical coordinates of the corner points in the world coordinate system to the image plane to obtain the theoretical pixel coordinate values. The projection model here needs to consider the internal parameters (including the camera focal length, the position of the principal point, etc.) and the external parameters (including the position and posture information of the camera in the world coordinate system).

[0055] Next, the error between the actual pixel coordinate value and the theoretical pixel coordinate value of each corner point is calculated. Assuming that these errors follow a Gaussian distribution, a likelihood function is constructed. The likelihood function represents the probability of observing the actual pixel coordinate value under the current parameters. The logarithm of the likelihood function is converted into a minimization problem, and the goal is to find the parameter combination that minimizes the total error.

[0056] Then, an iterative optimization algorithm is used to solve the optimal parameters. In each iteration, the projection error under the current parameters is first calculated, and then the gradient of the error with respect to each parameter is calculated. Based on the gradient information, the parameter value is updated to further reduce the error. This process is repeated until the convergence condition is met, such as the error change is less than the preset threshold or the maximum number of iterations is reached.

[0057] Finally, the optimized parameters are evaluated and verified. The final reprojection error is calculated to ensure that the error is within an acceptable range. At the same time, the stability and generalization performance of the parameters can be tested through cross-validation methods to ensure the reliability of the optimization results.

[0058] S1036, using the least square method to solve the radial distortion coefficient; The radial distortion coefficient represents a parameter describing the degree of radial deformation of the lens.

[0059] In some specific embodiments, firstly, based on the camera imaging model, a relationship equation between the ideal coordinates and the actual distorted coordinates is established: xd=x(1+k1r 2 +k2r 4 ) yd=y(1+k1r 2 +k2r 4 ) In the formula, xd and yd are the actual distortion coordinates; x and y are the ideal distortion-free coordinates; k1 and k2 are the second-order and fourth-order radial distortion coefficients respectively; r 2 =x 2 +y 2 is the square of the normalized distance from the point to the center of the image.

[0060] Then, construct the error equation for each observation point: Δx=xd′-x(1+k1r 2 +k2r 4 ) Δy=yd′-y(1+k1r 2 +k2r 4 ) Where Δx and Δy are the errors between the actual observed values ​​and the theoretical values; xd′ and yd′ are the actually observed distorted coordinates.

[0061] Next, the error equations for all observation points are combined into a matrix form: AX=L Where A is the coefficient matrix; X is the parameter vector to be solved; L is the vector consisting of the difference between the observed value and the ideal value.

[0062] Finally, the system of equations is solved using the least squares method: X=(A T A) -1 A T L Where A is the coefficient matrix, A T is the transpose of matrix A; (A T A) -1 is the matrix A T The inverse of A; X is the final radial distortion coefficient vector.

[0063] S1037: Perform distortion correction on the two-dimensional image based on the optimized intrinsic parameters, the optimized extrinsic parameters and the radial distortion coefficient.

[0064] In some embodiments, the system uses the optimized intrinsic parameters, extrinsic parameters and radial distortion coefficients to establish an image correction model, performs pixel remapping on the input original image, eliminates the distortion effect, and obtains an image with geometric distortion corrected.

[0065] In some specific embodiments, camera parameters are loaded; a correction lookup table is generated; bilinear interpolation is performed; and image edges are processed, which are not limited here.

[0066] It can be seen that, first of all, sufficient calibration data samples are obtained by collecting images of the calibration object at multiple angles and multiple positions. During the calibration process, the system first calculates the basic parameters of the optoelectronic equipment under ideal distortion-free conditions, then optimizes the internal and external parameters through the likelihood estimation method, and finally uses the least squares method to solve the radial distortion coefficient, effectively overcoming the inevitable optical errors of optoelectronic equipment in the production process, improving the distortion problem in the edge area of ​​the picture, and providing a high-quality real scene foundation for the subsequent virtual-real fusion.

[0067] In actual use, aircraft optoelectronic equipment usually has a multi-viewing angle switching function, which can adjust the observation range according to different mission requirements. However, the distortion characteristics of the optical system are different under different view angles. The existing technology uses unified intrinsic parameters, extrinsic parameters and radial distortion coefficients for image correction, which cannot adapt to the multi-viewing angle switching scenario, resulting in unsatisfactory correction effect.

[0068] See also Figure 3 and Figure 4 , Figure 3 is a schematic diagram of a problem in an exemplary application scenario in an embodiment of the present application; Figure 4is another problem schematic diagram in an exemplary application scenario in an embodiment of the present application; Figure 3 In the figure, the picture on the left is a theoretical two-dimensional picture, and the picture on the right is an actual two-dimensional picture under a long focal length. It can be seen that the image under a long focal length lens presents a "barrel distortion" feature, with the center area of ​​the picture being relatively clear, while the edge area is deformed and bulges outward. The target in the center area maintains a relatively real shape, but as the distance from the center point increases, the deformation of the object's outline becomes more obvious, especially the distortion in the four corners of the picture is the most obvious.

[0069] Figure 4 In the figure, the picture on the left is a theoretical two-dimensional picture, and the picture on the right is an actual two-dimensional picture under a short focal length. It can be seen that the image under a short focal length lens exhibits a "pincushion distortion" feature, and the edge area of ​​the picture shrinks toward the center of the picture. This distortion makes the straight line appear concave when it is far away from the center of the image, and the proportion of objects in the edge area of ​​the image is squeezed and deformed, while the degree of deformation in the center area is relatively small.

[0070] In some embodiments, before steps S1031 to S1037, the method further includes: The two-dimensional image is divided into image data of three viewing angles: a large viewing angle, a medium viewing angle, and a small viewing angle, wherein the observation angles of the large viewing angle, the medium viewing angle, and the small viewing angle decrease in sequence; Among them, the two-dimensional picture refers to the original image data collected by the optoelectronic device; the large field of view angle represents the observation angle with the widest coverage; the medium field of view angle refers to the observation angle between the large and small field of view angles; the small field of view angle represents the observation angle with the smallest coverage; this step is performed before starting the camera calibration. In some embodiments, the system first adjusts the optoelectronic device to different field of view angle modes, and collects images at large, medium, and small field of view angles respectively. Due to the differences in the optical properties and distortion characteristics of the lens at different field of view angles, a separate calibration process is required for each field of view angle. The system processes the images of the three field of view angles in order from large to small to ensure that accurate distortion correction effects can be obtained in each field of view angle mode.

[0071] S1031 to S1037 are respectively performed on the image data of the three viewing angles.

[0072] It can be seen that the distortion correction is performed separately for images with different field of view angles. The system divides the two-dimensional image into three fields of view: large, medium, and small according to the observation angle, and establishes a correction parameter model for each field of view. This classification processing method overcomes the problem that unified parameters are difficult to adapt to the switching of multiple field of view angles, ensures the accuracy of distortion correction under different observation ranges, and improves the practicality of the system.

[0073] See also Figure 5 , Figure 5This is a schematic diagram of a scenario in step S104 in an embodiment of the present application; S104, generating a virtual target in a three-dimensional scene; Among them, the virtual target refers to a digital target model generated in a three-dimensional scene for training; This step is performed after the three-dimensional scene is constructed. In some embodiments, the system generates virtual targets with specific attributes and behaviors at specific locations according to training requirements. These targets can be static or dynamic targets with predefined motion trajectories. The system ensures the visual realism and rationality of the targets through real-time rendering.

[0074] In some specific embodiments, the target type and parameters are selected; the target model is instantiated at a specified location; the target material and texture are set; the target motion trajectory is configured; and the target interactive behavior is added, which is not limited here.

[0075] In other specific embodiments, a prefabricated target model is loaded; the initial position of the target is calculated; target attribute parameters are set; a target action script is defined; and target state management is established, which is not limited here.

[0076] In some embodiments, after step S104, the method further includes: S1041, obtaining the corrected two-dimensional image and the position information of the virtual target; Among them, the position information represents the coordinates and posture data of the target in space; acquisition refers to the process of collecting and receiving relevant data; in some embodiments, the system first obtains the corrected real-scene image data from the image processing module, including image resolution, field of view and other information. At the same time, the three-dimensional position coordinates, orientation angle, motion state and other information of the virtual target are obtained from the scene management module. The system needs to ensure the time synchronization of the two types of data and establish a unified spatial reference coordinate system to provide an accurate spatial registration basis for subsequent fusion processing.

[0077] S1042: superimpose the virtual target on the corrected two-dimensional image based on the position information to generate a two-dimensional fused image superimposed with the virtual target.

[0078] In some embodiments, the system calculates the projection position and size of the virtual target in the two-dimensional image based on the three-dimensional position information of the virtual target, and superimposes the processed virtual target image on the real scene image in the correct position and order.

[0079] It can be seen that the system first obtains the distortion-corrected two-dimensional real scene image and the precise position information of the virtual target, and then directly superimposes the virtual target on the real scene image based on this information. This direct superposition method enables the pilot to clearly observe the virtual target in the actual scene.

[0080] See also Figure 6 , Figure 6 This is a schematic diagram of a scenario of step S105 in an embodiment of the present application; S105, creating a virtual camera that is at the same position as the aircraft and has the same field of view as the projection direction of the two-dimensional image, and collecting a fused image of the corrected two-dimensional image and the virtual target; Among them, the virtual camera refers to the viewpoint created in the three-dimensional scene for observing the scene; the fused image represents the final display image after combining the real scene image and the virtual target; the projection direction of the two-dimensional image represents the observation direction of the actual optoelectronic device.

[0081] This step is performed after distortion correction and virtual target generation. In some embodiments, the system creates a virtual camera that is exactly the same as the actual position of the aircraft, and sets its field of view to be the same as the actual optoelectronic device. Through this precise position and perspective matching, it ensures that the virtual target can be accurately superimposed on the real scene, producing a coherent and natural fusion effect.

[0082] In some specific embodiments, the real-time position data of the aircraft is obtained; the position of the virtual camera is synchronized; the field of view angle is calculated and set; the camera parameters are adjusted; and the fused image is collected, which is not limited here.

[0083] In some other specific embodiments, a camera coordinate system is established; attitude angle data is synchronized; a projection matrix is ​​set; rendering parameters are optimized; and a fused image is output, which is not limited here.

[0084] In some embodiments, step S105 specifically includes: S1051, creating a virtual camera, making the position of the virtual camera the same as the position of the aircraft, and the field of view of the virtual camera the same as the projection direction of the two-dimensional image; S1052, collecting the corrected two-dimensional image and the fused frame image of the virtual target through a virtual camera; Among them, the fused frame refers to an image synthesized by the real scene image and the virtual target; In some embodiments, the system first uses the corrected actual two-dimensional image as the background, and then renders the virtual target in the three-dimensional scene from the perspective of the virtual camera. Through the image fusion algorithm, the virtual target is accurately superimposed on the actual image, and factors such as lighting conditions and occlusion relationships need to be considered to ensure the authenticity and consistency of the fusion effect. The system also needs to ensure real-time performance so that the fusion process can be updated in real time as the aircraft moves.

[0085] S1053, superimposing the avionics data on the fused frame to generate a fused frame superimposed with the avionics data; Among them, avionics data refers to various status parameters and navigation information of the aircraft.

[0086] In some embodiments, the system first collects and processes real-time avionics data, including flight altitude, speed, attitude angle, and other information. Then, the display layout of the avionics data is designed, and various types of data are added to the appropriate positions of the fused frame in the form of text, icons, rulers, etc. The system needs to ensure that the display of avionics data is clear, the position is reasonable, does not affect the main observation target, and can be updated in real time according to the flight status.

[0087] See also Figure 7 , Figure 7 This is an effect diagram of the fusion screen superimposed with avionics data in the embodiment of the present application; S1054, outputting the fused frame image superimposed with the avionics data to the display system via the HDMI interface.

[0088] In some embodiments, the system first encodes and packages the generated picture data according to the HDMI protocol requirements, and then transmits the digital video signal to the display device through the HDMI interface, and the stability and synchronization of the signal need to be ensured.

[0089] It can be seen that by creating a virtual camera that is consistent with the position of the aircraft and synchronizing its field of view with the actual video projection direction, the system ensures that the fused image is always limited to the effective video range. This dual matching mechanism of spatial position and field of view enables the pilot to always observe the training scene from the first-person perspective, ensuring the continuity and consistency of the observation perspective. At the same time, the real-time superposition of avionics data further enhances the information integrity of the training image, allowing pilots to obtain the necessary flight parameters while maintaining a real observation experience. This fusion method based on precise position matching and perspective synchronization not only ensures the natural transition of the virtual and real fusion images, but also restores the visual experience during actual flight to the greatest extent, improving the realism and practicality of the training.

[0090] S106. Perform aircraft simulation training according to the fused images.

[0091] It can be seen that the precise integration of actual optoelectronic equipment and virtual scenes has been achieved. First, the introduction of geographic information data provides a real scene basis for training; second, the distortion correction of the two-dimensional picture ensures the accuracy of real-scene imaging; finally, by creating a virtual camera with the same position and field of view as the aircraft, the system can always keep the output picture within the actual video range and strictly follow the first-person perspective of the pilot. This virtual-real combination based on position synchronization and perspective matching not only retains the real imaging characteristics of the actual optoelectronic equipment, but also realizes the precise superposition of virtual targets, so that the training picture always maintains the real visual effect and accurate spatial relationship, thereby reducing the training cost while providing a realistic training experience.

[0092] See also Figures 8 to 10 , Figure 8is a virtual target in the embodiment of the present application; Fig. 9 This is a real scene in the embodiment of this application; Fig.10 It is an effect diagram combining real scenes and virtual targets in an embodiment of the present application.

[0093] Figure 8 The virtual targets of aircraft pre-set in the system are displayed.

[0094] Fig. 9 The real scene images in the actual flight environment are presented, which are collected in real time by the camera system. These images include the real lighting conditions and environmental background.

[0095] Fig.10 The fusion effect obtained by using the virtual-real aircraft simulation training method proposed in this paper is demonstrated. This method not only retains the real imaging characteristics of the actual optoelectronic equipment, but also realizes the precise superposition of virtual targets, so that the training screen always maintains the real visual effect and accurate spatial relationship, thus providing a realistic training experience while reducing the training cost.

[0096] The following introduces an exemplary aircraft simulation training system 1100 based on the combination of virtual and real provided in an embodiment of the present application. Fig.11 It is a schematic diagram of an exemplary hardware structure of an aircraft simulation training system 1100 based on the combination of virtual and real provided in an embodiment of the present application.

[0097] In some embodiments, the aircraft simulation training system 1100 based on the combination of virtual and real is a computer device or the aircraft simulation training system 1100 based on the combination of virtual and real includes a computer device. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The network interface of the computer device is used to communicate with other external terminals or servers through a network connection. In some embodiments, the network interface can be a wired network interface, and in some embodiments, the network interface can also be a wireless network interface. When the computer program is executed by the processor, the method in the embodiment of the present application is implemented.

[0098] Those skilled in the art will understand that Fig.11The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0099] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0100] As used in the above embodiments, the term "when..." may be interpreted to mean "if..." or "after..." or "in response to determining..." or "in response to detecting...", depending on the context. Similarly, the phrases "upon determining..." or "if (the stated condition or event) is detected" may be interpreted to mean "if determining..." or "in response to determining..." or "upon detecting (the stated condition or event)" or "in response to detecting (the stated condition or event)", depending on the context.

[0101] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media integration. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk), etc.

[0102] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

Claims

1. A method for aircraft simulation training based on the combination of virtual and real, characterized in that: include: Acquire geographic information data, and use the geographic information data as a layer to perform three-dimensional scene reconstruction of the aircraft simulation training scene area; Establishing a mapping relationship between a two-dimensional image acquired by an optoelectronic device of the aircraft and the three-dimensional scene; Performing distortion correction on the two-dimensional image; Generating a virtual target in the three-dimensional scene; Creating a virtual camera that is at the same position as the aircraft and has the same field of view as the projection direction of the two-dimensional image, and collecting the corrected two-dimensional image and the fused image of the virtual target; Perform aircraft simulation training according to the fused images.

2. The method according to claim 1, characterized in that The step of establishing a mapping relationship between the two-dimensional picture acquired by the optoelectronic device of the aircraft and the three-dimensional scene specifically includes: Where, T E are the coordinates of the three-dimensional scene; M lla2ecef Convert the matrix of the three-dimensional scene into a latitude and longitude coordinate system; M ENU_WITHOUT_TRANS A coordinate transformation matrix that converts the three-dimensional scene corresponding to the longitude and latitude of the aircraft into the northeast sky coordinate system at the longitude and latitude without translation operation; M RP is the rotation matrix of the aircraft relative to its own coordinate system; M TC is the rotation matrix of the optoelectronic device relative to its own coordinate system; d E The straight-line distance between the three-dimensional coordinates corresponding to the current pixel and the origin of the optoelectronic device; is the affine coordinate identifier of the back-projected position of the pixel in the viewing frustum; A three-dimensional vector consisting of the xyz components of the affine coordinates of the back-projected position of the pixel in the viewing frustum; T clip_h_w A scalar consisting of the w component of the radial coordinates of the back-projected position of the pixel in the viewing frustum; is the back-projected position of the pixel in the local coordinate system of the viewing frustum; r width is the resolution of the width of the optoelectronic device; r height is the resolution of the height of the optoelectronic device; fovH is the horizontal field of view of the optoelectronic device; T x The horizontal coordinates of the pixel coordinates of the required pixel point; T y The vertical coordinate of the pixel coordinate of the required pixel point.

3. The method according to claim 1, characterized in that The step of performing distortion correction on the two-dimensional image specifically includes: Adjusting the position and posture of the calibration object, capturing a calibration image containing the calibration object at each preset position and posture to obtain a plurality of calibration images; the calibration object has a preset grid size and number of rows and columns; Detecting corner points of the grid in the calibration image and obtaining pixel coordinate values ​​of the corner points; Calculating the physical coordinate value of the corner point according to the actual size parameter of the calibration object, the position relationship in the world coordinate system, and the pixel coordinate value of the corner point; Calculating the intrinsic parameters and extrinsic parameters of the optoelectronic device based on the pixel coordinate value and the physical coordinate value, under the condition that the distortion coefficient of the optoelectronic device is zero; Taking into account the lens distortion of the optoelectronic device, optimizing the intrinsic parameters and the extrinsic parameters by using a likelihood estimation method so that the error between the pixel coordinate value of the corner point and the theoretical pixel coordinate value obtained by projecting the physical coordinate value of the corner point based on the intrinsic parameters and the extrinsic parameters is minimized; The least square method is used to solve the radial distortion coefficient; The two-dimensional image is subjected to distortion correction based on the optimized intrinsic parameters, the optimized extrinsic parameters and the radial distortion coefficient.

4. The method according to claim 3, characterized in that Before the step of performing distortion correction on the two-dimensional image, the method further includes: The two-dimensional image is divided into image data of three viewing angles: a large viewing angle, a medium viewing angle, and a small viewing angle, wherein the observation angles of the large viewing angle, the medium viewing angle, and the small viewing angle decrease in sequence; Distortion correction is performed on the image data of three viewing angles respectively.

5. The method according to claim 1, characterized in that The step of acquiring geographic information data and using the geographic information data as a layer to perform three-dimensional scene reconstruction of the aircraft simulation training scene area specifically includes: Acquire geographic information system base map data of the aircraft simulation training scene area, wherein the geographic information system base map data includes satellite image data, geographic information elevation data and vector geographic data; A layer is constructed based on the satellite image data, the geographic information elevation data and the vector geographic data to perform three-dimensional scene reconstruction on the aircraft simulation training scene area.

6. The method according to claim 1, characterized in that The step of creating a virtual camera with the same position as the aircraft and the same field of view as the projection direction of the two-dimensional image, and collecting the corrected two-dimensional image and the fused image of the virtual target specifically includes: Creating a virtual camera, so that the position of the virtual camera is the same as the position of the aircraft, and the field of view of the virtual camera is the same as the projection direction of the two-dimensional image; Capturing a fused frame of the corrected two-dimensional image and the virtual target by using the virtual camera; superimposing the avionics data on the fused frame to generate a fused frame superimposed with the avionics data; The fused frame image of the superimposed avionics data is output to a display system via an HDMI interface.

7. The method according to claim 1, characterized in that After the step of generating a virtual target in the three-dimensional scene, the method further comprises: Acquiring the corrected two-dimensional image and position information of the virtual target; The virtual target is superimposed on the corrected two-dimensional picture based on the position information to generate a two-dimensional fused picture superimposed with the virtual target.

8. An aircraft simulation training system based on the combination of virtual and real, characterized in that: The aircraft simulation training system based on the combination of virtual and real includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the aircraft simulation training system based on the combination of virtual and real to execute the method as described in any one of claims 1-7.

9. A computer program product comprising instructions, characterized in that When the computer program product runs on an aircraft simulation training system based on a combination of virtual and real, the aircraft simulation training system based on a combination of virtual and real executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium comprising instructions, characterized in that: When the instructions are executed on an aircraft simulation training system based on a combination of virtual and real, the aircraft simulation training system based on a combination of virtual and real executes the method according to any one of claims 1 to 7.

Images