An aircraft flight test panoramic visual perception simulation method, system, electronic terminal and storage medium

Abstract

The application discloses a kind of photodetector technical field's aircraft test flight panoramic visual perception simulation method, system, electronic terminal and storage medium, method includes: generating aircraft test flight multi-view image, the aircraft test flight multi-view image is projected to display;According to the mapping relationship between real camera and the display, the mapping relationship between the display and virtual camera, the mapping relationship between the real camera and the virtual camera is established;Control the virtual camera according to aircraft test flight real-time video continuously projects the real-time multi-view image of aircraft test flight to the display, and control the real-time acquisition display image of real camera;Obtain the aircraft test flight panoramic image in the real camera visual angle of virtual scene perception simulation.Can solve the technical problems that the accuracy of test required experimental sample is reduced due to the inconsistency between test flight image and actually collected test flight image in the prior art.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of photoelectric detection, and relates to a method and system for simulating panoramic visual perception during aircraft test flight, an electronic terminal and a storage medium. BACKGROUND

[0002] Aircraft test flight is an important link in the process of aircraft design, testing and finalization, and plays a key role in ensuring its performance and safety. A large amount of video data needs to be collected during the test flight, so cameras need to be installed on each part of the aircraft to collect data, and a panoramic video collection system for aircraft test flight is constructed to record and alarm abnormalities in time. The data collected by the cameras at each part need to have high stability and high reliability to ensure the complete recording of the test flight process. Therefore, ground testing of the video collection equipment becomes an effective means to verify its reliability. However, the experimental samples required for testing are usually obtained through simulation testing, site testing and road testing. The scene of site testing is single and cannot approach the real scene. Road testing is costly and complex to operate. Simulation testing simulates various driving scenes and road conditions through a computer to conduct a large number of tests and data analysis. However, the existing simulation testing has the problem that the data generated by the computer simulation does not match the actual working conditions, which cannot approach the real scene, resulting in a decrease in the accuracy of the experimental samples required for testing, and cannot meet the needs of actual testing. SUMMARY

[0003] The purpose of the present application is to overcome the deficiencies in the prior art and provide a method and system for simulating panoramic visual perception during aircraft test flight, an electronic terminal and a storage medium, which can solve the technical problem of a decrease in the accuracy of experimental samples required for testing due to the inconsistency between the test flight images generated by simulation and the test flight images actually collected in the prior art.

[0004] To solve the above technical problems, the present application is implemented by using the following technical solutions:

[0005] In a first aspect, the present application provides a method for simulating panoramic visual perception during aircraft test flight, comprising:

[0006] controlling a virtual camera in a virtual scene to generate a multi-view image of the test flight of the aircraft through a panoramic multi-view virtual imaging model of the aircraft, and projecting the multi-view image of the test flight of the aircraft to a display;

[0007] controlling a real camera to collect an image of the display, calibrating the transformation external parameters of the real camera and the display based on the image of the display, and establishing a mapping relationship between the real camera and the display;

[0008] based on the image of the display and the multi-view image of the test flight of the aircraft displayed by the display, establishing a mapping relationship between the display and the virtual camera;

[0009] According to the mapping relationship between the real camera and the display, the mapping relationship between the display and the virtual camera, a mapping relationship between the real camera and the virtual camera is established;

[0010] The virtual camera is controlled to project real-time multi-view images of the aircraft test flight to the display according to the real-time video of the aircraft test flight, and the real camera is controlled to collect display images in real time;

[0011] According to the mapping relationship between the real camera and the virtual camera, and the display images collected by the real camera in real time, an aircraft test flight panoramic image in a virtual scene is obtained, which is perceived in a simulated real camera view angle.

[0012] The distribution of the virtual camera in the virtual scene is consistent with the distribution of the real camera in the real scene.

[0013] Further, before controlling the real camera to collect display images:

[0014] The multi-view images of the aircraft test flight displayed by the display are processed by the Zhang Zhengyou camera calibration method to calibrate the internal parameters of the real camera.

[0015] Further, it further comprises:

[0016] Based on the calibrated internal parameters of the real camera, the multi-view images of the aircraft test flight displayed by the display are processed by the image de-distortion function of opencv to complete image de-distortion, and non-distorted multi-view images of the aircraft test flight are obtained. The non-distorted multi-view images of the aircraft test flight are projected to the display.

[0017] In a second aspect, the present application provides a method for testing aircraft test flight anomalies, the method comprising:

[0018] Obtaining an aircraft test flight panoramic image;

[0019] Inputting the aircraft test flight panoramic image into a pre-trained adversarial network model to obtain a generated image, calculating a pixel-level difference value of the generated image and the aircraft test flight panoramic image, if the pixel-level difference value is greater than a preset difference value threshold, indicating that the aircraft test flight is abnormal; otherwise, indicating that the aircraft test flight is normal.

[0020] The aircraft test flight panoramic image is obtained by the aircraft test flight panoramic visual perception simulation method of any one of the first aspect.

[0021] Further, the adversarial network model comprises a generator and a discriminator, and the training process of the adversarial network model comprises:

[0022] inputting the training sample into a pre-constructed adversarial network model, generating an image by a generator of the adversarial network model;

[0023] discriminating, by a discriminator, a similarity between the image generated by the generator and the input training sample;

[0024] updating, by a gradient back propagation algorithm, parameters of the generator iteratively until the similarity reaches a maximum.

[0025] Further, calculating the pixel-level difference value between the generated image and the virtual perspective camera image comprises:

[0026] calculating the pixel-level difference value between the generated image and the virtual perspective camera image by a two-norm loss function:

[0027] ,

[0028] wherein, is the pixel-level difference value; is the virtual perspective camera image, is the generated image.

[0029] In a third aspect, the present application provides a simulation system of the flight test panoramic visual perception simulation method of the first aspect, comprising:

[0030] an image generation module configured to control a virtual camera in a virtual scene to generate a flight test multi-perspective image by a flight test panoramic multi-perspective virtual imaging model, and project the flight test multi-perspective image to a display;

[0031] a mapping module configured to control a real camera to capture a display image, calibrate a transformation external parameter of the real camera and the display based on the display image, establish a mapping relationship between the real camera and the display, and establish a mapping relationship between the display and the virtual camera based on the display image and the flight test multi-perspective image displayed by the display, and establish a mapping relationship between the real camera and the virtual camera according to the mapping relationship between the real camera and the display, the mapping relationship between the display and the virtual camera;

[0032] a simulation module configured to control the virtual camera to project a real-time multi-perspective image of a flight test to the display continuously according to a real-time video of the flight test, and control the real camera to capture a display image in real time, and acquire a flight test panoramic image in a virtual scene perceived by a real camera according to the mapping relationship between the real camera and the virtual camera and the display image captured by the real camera in real time.

[0033] The distribution positions of the virtual cameras in the virtual scene are consistent with the distribution positions of the real cameras in the real scene.

[0034] In a fourth aspect, the present application provides an electronic terminal, comprising a processor, a memory connected to the processor, and a computer program stored in the memory.

[0035] The memory, wherein the computer program is stored in the memory, when the computer program is executed by the processor, executes the steps of the method according to the first aspect or the second aspect.

[0036] In a fifth aspect, the present application provides a computer readable storage medium, wherein a computer program is stored on the computer readable storage medium, and the computer program is executed by a processor to implement the steps of the method according to the first aspect or the second aspect.

[0037] Compared with the prior art, the present application has the following beneficial effects:

[0038] The multi-view image of the airplane is generated by the pre-constructed panoramic multi-view virtual imaging model of the airplane, and is projected to the display, and the real camera is used to collect the image of the display, and then the image transformation from the real camera to the virtual camera is established, so that the visual perception of the panoramic space of the airplane in the virtual scene is realized, and the data acquisition in the airplane test flight process is simulated, and the stability and reliability of the video acquisition equipment during ground testing are improved.

[0039] The pixel-level difference value of the generated image and the virtual perspective camera image is calculated by the two-norm loss function, the pixel-level difference value is compared with the preset difference value threshold, the acquired virtual perspective camera image is judged, and the accuracy of the experimental sample required for testing is improved. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 A flowchart of the airplane test flight panoramic visual perception simulation method provided by the embodiment one of the present application;

[0041] Figure 2 A control logic diagram of the airplane test flight panoramic visual perception simulation method provided by the embodiment one of the present application;

[0042] Figure 3 An airplane and virtual camera position schematic diagram of the airplane panoramic multi-view virtual imaging model of the airplane test flight panoramic visual perception simulation system provided by the embodiment three of the present application;

[0043] Figure 4 A mapping relationship diagram of the airplane test flight panoramic visual perception simulation system provided by the embodiment three of the present application. DETAILED DESCRIPTION

[0044] The technical solutions of the present application will be described in detail below with reference to the drawings and specific embodiments. It should be understood that the embodiments and specific features in the embodiments are detailed descriptions of the technical solutions of the present application, and are not limitations of the technical solutions of the present application. In the case of no conflict, the technical features in the embodiments and the embodiments can be combined with each other.

[0045] The term "and / or", only describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent three cases of A alone, A and B together, and B alone. In addition, the character " / " generally represents that the associated objects before and after it are in an "or" relationship. Embodiments

[0046] The present application provides a flight test panoramic visual perception simulation method, as shown in the accompanying drawings Figure 1 and Figure 2 , comprising:

[0047] controlling the virtual camera in the virtual scene to generate a flight test multi-view image through an aircraft panoramic multi-view virtual imaging model, and projecting the flight test multi-view image to a display;

[0048] processing the flight test multi-view image displayed by the display through Zhang Zhengyou camera calibration method to calibrate the internal parameters of the real camera;

[0049] based on the calibrated internal parameters of the real camera, processing the flight test multi-view image displayed by the display through the image de-distortion function of opencv to complete image de-distortion, and obtaining a non-distorted flight test multi-view image, and projecting the non-distorted flight test multi-view image to the display.

[0050] Regarding the Zhang Zhengyou camera calibration method, Zhang Zhengyou camera calibration method is a widely used camera calibration technology, which was proposed by Dr. Zhang Zhengyou in his paper "Flexible Camera Calibration By Viewing a Plane From Unknown Orientations" in 1999. This method is widely used because of its simple operation and high precision. In practical applications, Zhang Zhengyou calibration method is usually combined with computer vision library such as OpenCV, and the corresponding code is written to automatically complete the calibration process and obtain the camera parameters; that is, the present application processes the flight test multi-view image displayed by the display through Zhang Zhengyou camera calibration method, and processes the flight test multi-view image displayed by the display through the image de-distortion function of opencv, both of which belong to the prior art, and will not be described here.

[0051] Control a real camera to acquire images from a display screen, calibrate the transformation extrinsic parameters of the real camera and the display screen based on the display screen images, and establish a mapping relationship between the real camera and the display screen:

[0052] Based on the images captured by the real camera and the multi-view images of the aircraft test flight displayed on the monitor, solve for the transformation matrices of the real camera and the monitor:

[0053] ,

[0054] Among them, the To capture images using a real camera, For the display image, This is the transformation matrix, i.e., the transformation extrinsic parameters;

[0055] Based on the images displayed on the monitor and the multi-view images of the aircraft test flight displayed on the monitor, a mapping relationship between the monitor and the virtual camera is established, represented as follows: ;

[0056] Based on the mapping relationship between the actual camera and the display The mapping relationship between the display and the virtual camera Establish a mapping relationship between the real camera and the virtual camera. ;

[0057] The virtual camera is controlled to continuously project real-time multi-view images of the aircraft test flight onto the display based on the real-time video of the aircraft test flight, and the real camera is controlled to capture images of the display in real time.

[0058] Based on the mapping relationship between the real camera and the virtual camera, and the real camera's real-time captured images on the display, a panoramic image of the aircraft test flight from the perspective of the real camera in the virtual scene is obtained.

[0059] The virtual cameras are located in the same positions in the virtual scene as the real cameras are located in the real scene.

[0060] At this point, virtual view camera images of the aircraft panoramic multi-view virtual imaging model can be acquired using real cameras, completing data acquisition and processing. The obtained virtual view camera images and the configuration parameters of each virtual camera, such as internal and external parameters, are then used for aerial target identification and measurement.

[0061] The multi-view image of the airplane is generated through the pre-constructed panoramic multi-view virtual imaging model of the airplane, and is projected to a display, a real camera is used to collect the image of the display, and then the image transformation from the real camera to the virtual camera is established, so that the visual perception of the panoramic space of the airplane in the virtual scene is realized, and the data collection in the airplane test flight process is simulated, and the stability and reliability of the video collection equipment in the ground test are improved. Embodiments

[0062] The application provides an airplane test flight anomaly testing method, which comprises the following steps:

[0063] An airplane test flight panoramic image is acquired.

[0064] The airplane test flight panoramic image is input into a pre-trained adversarial network model to obtain a generated image.

[0065] The adversarial network model comprises a generator and a discriminator, and the training process of the adversarial network model comprises the following steps.

[0066] A training sample is input into the pre-constructed adversarial network model, and an image is generated through the generator of the adversarial network model.

[0067] The similarity between the image generated by the generator and the input training sample is judged by using the discriminator.

[0068] The parameters of the generator are iteratively updated through a gradient back propagation algorithm until the similarity reaches a maximum.

[0069] The pixel-level difference value between the generated image and a virtual perspective camera image is calculated through a two-norm loss function.

[0070] ,

[0071] wherein, is the pixel-level difference value, is the virtual perspective camera image, is the generated image,

[0072] If the pixel-level difference value is greater than a preset difference value threshold, it indicates that the airplane test flight is abnormal, otherwise, it indicates that the airplane test flight is normal, and the detection of the airplane test flight condition is completed.

[0073] The airplane test flight panoramic image is acquired by using the airplane test flight panoramic visual perception simulation method in Embodiment 1, and has corresponding beneficial effects. Embodiments

[0074] The embodiment provides a plane test flight panoramic visual perception simulation system, which can be used for implementing the plane test flight panoramic visual perception simulation method in the embodiment one and has the beneficial effects of the plane test flight panoramic visual perception simulation method in the embodiment one, as shown in the accompanying drawings Figure 3 and the accompanying drawings Figure 4 As shown in the accompanying drawings, the plane test flight panoramic visual perception simulation system comprises a plane panoramic multi-view virtual imaging model, a display and a real camera, the plane panoramic multi-view virtual imaging model comprises virtual cameras, and the distribution positions of the virtual cameras in a virtual scene are consistent with the distribution positions of the real camera in a real scene.

[0075] The plane panoramic multi-view virtual imaging model is a system for generating panoramic images inside and outside a plane by using computer graphics and image processing technology, and the model can be used for multiple purposes, including flight simulation training, virtual travel, safety demonstration, entertainment and visualization of plane design and manufacturing, and is a mature technology, which will not be described herein.

[0076] The system further comprises an image generation module configured to control the virtual cameras in the virtual scene to generate plane test flight multi-view images by using the plane panoramic multi-view virtual imaging model and project the plane test flight multi-view images to the display.

[0077] The system further comprises a mapping module configured to control the real camera to capture display images, calibrate transformation external parameters of the real camera and the display based on the display images, establish a mapping relationship between the real camera and the display, establish a mapping relationship between the display and the virtual cameras based on the display images and the plane test flight multi-view images displayed by the display, and establish a mapping relationship between the real camera and the virtual cameras according to the mapping relationship between the real camera and the display and the mapping relationship between the display and the virtual cameras.

[0078] The system further comprises a simulation module configured to control the virtual cameras to continuously project real-time multi-view images of the plane test flight to the display according to real-time video of the plane test flight and control the real camera to capture display images in real time, and acquire plane test flight panoramic images in a virtual scene in a real camera perspective of the perception simulation according to the mapping relationship between the real camera and the virtual cameras and the display images captured by the real camera in real time.

[0079] The plane multi-view images are generated by using the pre-constructed plane panoramic multi-view virtual imaging model and projected to the display, the display images of the display are captured by using the real camera, and then the image transformation from the real camera to the virtual camera is established, so that the visual perception of the plane panoramic space in the virtual scene is realized, the data acquisition in the plane test flight process is simulated, and the stability and reliability of the video acquisition equipment during the ground test are improved.

[0080] The pixel-level difference value of the generated image and the virtual perspective camera image is calculated by a two-norm loss function, the pixel-level difference value is compared with a preset difference value threshold, and the obtained virtual perspective camera image is judged, thereby improving the accuracy of the experimental sample required for testing. Embodiments

[0081] The embodiment four further provides an electronic terminal, comprising a processor and a memory connected with the processor, wherein the memory stores a computer program, and the processor is used for operating according to the instruction to execute the steps of the method in the embodiment one or the embodiment two. Embodiments

[0082] The embodiment five further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the method in the embodiment one or the embodiment two, and has the corresponding function modules and beneficial effects of the executed method.

[0083] The above only describes the preferred embodiments of the present application, and it should be pointed out that for ordinary skilled in the art, without departing from the technical principles of the present application, a number of improvements and modifications can be made, and these improvements and modifications should be considered as the protection scope of the present application.

Claims

1. A panoramic visual perception simulation method for aircraft flight testing, characterized in that, include: The virtual camera in the virtual scene is controlled to generate multi-view images of the aircraft test flight through the aircraft panoramic multi-view virtual imaging model, and the multi-view images of the aircraft test flight are projected onto the display. Control the real camera to acquire images from the display, calibrate the transformation extrinsic parameters of the real camera and the display based on the display images, and establish a mapping relationship between the real camera and the display; Based on the images displayed on the monitor and the multi-view images of the aircraft test flight displayed on the monitor, a mapping relationship is established between the monitor and the virtual camera; Based on the mapping relationship between the real camera and the display, and the mapping relationship between the display and the virtual camera, establish the mapping relationship between the real camera and the virtual camera; The virtual camera is controlled to continuously project real-time multi-view images of the aircraft test flight onto the display based on the real-time video of the aircraft test flight, and the real camera is controlled to capture images of the display in real time. Based on the mapping relationship between the real camera and the virtual camera, and the real camera's real-time captured images on the display, a panoramic image of the aircraft test flight from the perspective of the real camera in the virtual scene is obtained. The virtual cameras are located in the same positions in the virtual scene as the real cameras are located in the real scene.

2. The panoramic visual perception simulation method for aircraft flight testing according to claim 1, characterized in that, Before controlling a real camera to capture images on a monitor: The intrinsic parameters of the real camera are calibrated by processing the multi-view images of the aircraft test flight displayed on the monitor using the Zhang Zhengyou camera calibration method.

3. The panoramic visual perception simulation method for aircraft flight testing according to claim 2, characterized in that, Also includes: Based on the calibrated real camera intrinsic parameters, the multi-view images of the aircraft test flight displayed on the monitor are processed by the image distortion correction function of OpenCV to complete the image distortion correction and obtain distortion-free multi-view images of the aircraft test flight. The distortion-free multi-view images of the aircraft test flight are then projected onto the monitor.

4. A method for testing anomalies during aircraft flight testing, characterized in that, The method includes: Acquire panoramic images of the aircraft during test flights; The panoramic image of the aircraft test flight is input into a pre-trained adversarial network model to obtain a generated image. The pixel-level difference between the generated image and the panoramic image of the aircraft test flight is calculated. If the pixel-level difference is greater than a preset difference threshold, it indicates that the aircraft test flight is abnormal; otherwise, it indicates that the aircraft test flight is normal. The panoramic image of the aircraft test flight is obtained using the panoramic visual perception simulation method for aircraft test flight as described in any one of claims 1 to 3.

5. The aircraft flight test anomaly testing method according to claim 4, characterized in that, The adversarial network model includes a generator and a discriminator, and the training process of the adversarial network model includes: The training samples are input into a pre-built adversarial network model, and the generator of the adversarial network model generates images. The discriminator is used to determine the similarity between the image generated by the generator and the corresponding input training sample; The generator parameters are iteratively updated using the gradient backpropagation algorithm until the similarity reaches its maximum.

6. The aircraft flight test anomaly testing method according to claim 4, characterized in that, Calculating the pixel-level difference between the generated image and the virtual viewpoint camera image includes: The pixel-level difference between the generated image and the virtual viewpoint camera image is calculated using the L2 loss function: ， in, These are pixel-level differences. Images from a virtual camera perspective. To generate an image.

7. A panoramic visual perception simulation system for aircraft flight testing, characterized in that, include: The image generation module is used to control the virtual camera in the virtual scene to generate multi-view images of the aircraft test flight through the aircraft panoramic multi-view virtual imaging model, and project the multi-view images of the aircraft test flight onto the display. The mapping module is used to control the real camera to acquire images from the display, calibrate the transformation extrinsic parameters of the real camera and the display based on the display images, and establish the mapping relationship between the real camera and the display. Based on the images displayed on the monitor and the multi-view images of the aircraft test flight displayed on the monitor, a mapping relationship is established between the monitor and the virtual camera; based on the mapping relationship between the real camera and the monitor, and the mapping relationship between the monitor and the virtual camera, a mapping relationship is established between the real camera and the virtual camera; The simulation module is used to control the virtual camera to continuously project real-time multi-view images of the aircraft test flight onto the display based on the real-time video of the aircraft test flight, and to control the real camera to acquire images of the display in real time. Based on the mapping relationship between the real camera and the virtual camera, and the real camera's real-time captured images on the display, a panoramic image of the aircraft test flight from the perspective of the real camera in the virtual scene is obtained. The virtual cameras are located in the same positions in the virtual scene as the real cameras are located in the real scene.

8. An electronic terminal, characterized in that, The method includes a processor and a memory connected to the processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the steps of the method as described in any one of claims 1-3 or 4-6 are performed.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method as described in any one of claims 1-3 or 4-6.

Images