Panoramic visual perception simulation method and system for aircraft test flight, electronic terminal and storage medium

Abstract

The invention discloses an aircraft test flight panoramic visual perception simulation method and system, an electronic terminal and a storage medium in the technical field of photoelectric detection, and the method comprises the steps: generating an aircraft test flight multi-view image, and projecting the aircraft test flight multi-view image to a display; establishing a mapping relation between a real camera and a virtual camera according to the mapping relation between the real camera and the display and the mapping relation between the display and the virtual camera; controlling the virtual camera to continuously project real-time multi-view images of aircraft test flight to the display according to the real-time video of aircraft test flight, and controlling the real camera to collect display images in real time; and acquiring an aircraft test flight panoramic image under a real camera visual angle sensed and simulated in the virtual scene. The technical problem that in the prior art, the accuracy of an experimental sample needed by testing is reduced due to the fact that a test flight image does not accord with an actually-collected test flight image can be solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of photoelectric detection, and relates to a method, system, electronic terminal and storage medium for simulating panoramic visual perception of an aircraft test flight. Background Art

[0002] Aircraft flight test is an important part of the aircraft design, testing, and finalization process, and plays a key role in ensuring its performance and safety. A large amount of video data needs to be collected during the flight test. Therefore, it is necessary to install cameras in various parts of the aircraft for data collection and construct a panoramic video acquisition system for aircraft flight test so as to detect abnormalities in time for recording and alarming. The data collected by the cameras in each part must have high stability and high reliability to ensure the full record of the flight test process. Therefore, ground testing of video acquisition equipment has become an effective means to verify its reliability. However, the experimental samples required for testing are usually obtained through simulation testing, field testing, and road testing. Among them, the scene of the field test is single and cannot approach the real scene; the cost of road testing is high and the operation is complicated; simulation testing simulates various driving scenes and road conditions through computers to conduct a large number of tests and data analysis. However, the existing simulation tests have the problem that the data generated by computer simulation does not match the actual working conditions and cannot approach the real scene, resulting in a decrease in the accuracy of the experimental samples required for testing and cannot meet the needs of actual testing. Summary of the invention

[0003] The purpose of the present invention is to overcome the deficiencies in the prior art and to provide a method, system, electronic terminal and storage medium for simulating panoramic visual perception of aircraft test flight, which can solve the technical problem in the prior art that the accuracy of experimental samples required for the test is reduced due to the discrepancy between the test flight images generated by simulation and the test flight images actually collected.

[0004] In order to solve the above technical problems, the present invention is implemented by adopting the following technical solutions: In a first aspect, the present invention provides a method for simulating panoramic visual perception of an aircraft test flight, comprising: Controlling a virtual camera in a virtual scene to generate a multi-perspective image of an aircraft test flight through a panoramic multi-perspective virtual imaging model of the aircraft, and projecting the multi-perspective image of the aircraft test flight to a display; Controlling a real camera to capture a display image, calibrating transformation extrinsic parameters of the real camera and the display based on the display image, and establishing a mapping relationship between the real camera and the display; Establishing a mapping relationship between the display and the virtual camera based on the display image and the aircraft test flight multi-view image displayed on the display; Establishing a mapping relationship between the real camera and the virtual camera according to a mapping relationship between the real camera and the display and a mapping relationship between the display and the virtual camera; Control the virtual camera to continuously project real-time multi-view images of the aircraft test flight onto the display according to the real-time video of the aircraft test flight, and control the real camera to collect display images in real time; According to the mapping relationship between the real camera and the virtual camera, and the display image captured by the real camera in real time, a panoramic image of the aircraft test flight under the perspective of the real camera perceived and simulated in the virtual scene is obtained; The distribution position of the virtual cameras in the virtual scene is consistent with the distribution position of the real cameras in the real scene.

[0005] Furthermore, before controlling the real camera to capture the display image: The multi-view images of the aircraft test flight displayed on the display are processed by Zhang Zhengyou's camera calibration method to calibrate the intrinsic parameters of the real camera.

[0006] Furthermore, it also includes: Based on the calibrated real camera intrinsic parameters, the aircraft test flight multi-view image displayed on the display is processed by the image dedistortion function of OpenCV to complete the image dedistortion, obtain the distortion-free aircraft test flight multi-view image, and project the distortion-free aircraft test flight multi-view image to the display.

[0007] In a second aspect, the present invention provides an aircraft flight test abnormality testing method, the method comprising: Acquire panoramic images of aircraft test flights; Input the aircraft test flight panoramic image into a pre-trained adversarial network model to obtain a generated image, and calculate the pixel-level difference value between the generated image and the aircraft test flight panoramic image. If the pixel-level difference value is greater than a preset difference value threshold, it indicates that the aircraft test flight is abnormal; otherwise, it indicates that the aircraft test flight is normal; Wherein, the aircraft test flight panoramic image is obtained by using the aircraft test flight panoramic visual perception simulation method described in any one of the first aspects.

[0008] Furthermore, the adversarial network model includes a generator and a discriminator, and the training process of the adversarial network model includes: Inputting the training sample into a pre-built adversarial network model, and generating an image through a generator of the adversarial network model; Using a discriminator to determine the similarity between the image generated by the generator and the corresponding input training sample; The parameters of the generator are iteratively updated through the gradient back-propagation algorithm until the similarity reaches the maximum.

[0009] Further, calculating the pixel-level difference value between the generated image and the virtual view camera image includes: The pixel-level difference value between the generated image and the virtual view camera image is calculated by the two-norm loss function: , in, is the pixel level difference value; is the virtual perspective camera image, To generate images.

[0010] In a third aspect, the present invention provides a simulation system for the aircraft test flight panoramic visual perception simulation method according to the first aspect, comprising: An image generation module is used to control a virtual camera in a virtual scene to generate a multi-perspective image of an aircraft test flight through a panoramic multi-perspective virtual imaging model of the aircraft, and project the multi-perspective image of the aircraft test flight to a display; A mapping module is used to control a real camera to capture a display image, calibrate the transformation external parameters of the real camera and the display based on the display image, and establish a mapping relationship between the real camera and the display; establish a mapping relationship between the display and the virtual camera based on the display image and the aircraft test flight multi-view image displayed on the display; establish a mapping relationship between the real camera and the virtual camera according to the mapping relationship between the real camera and the display and the mapping relationship between the display and the virtual camera; The simulation module is used to control the virtual camera to continuously project real-time multi-view images of the aircraft test flight to the display according to the real-time video of the aircraft test flight, and control the real camera to collect the display image in real time; according to the mapping relationship between the real camera and the virtual camera, and the display image collected by the real camera in real time, obtain the aircraft test flight panoramic image under the perspective of the real camera perceived and simulated in the virtual scene; The distribution position of the virtual cameras in the virtual scene is consistent with the distribution position of the real cameras in the real scene.

[0011] In a fourth aspect, the present invention provides an electronic terminal, comprising a processor and a A memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the steps of the method described in the first aspect or the second aspect are performed.

[0012] In a fifth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect or the second aspect above.

[0013] Compared with the prior art, the present invention has the following beneficial effects: The aircraft multi-view images are generated through the pre-built aircraft panoramic multi-view virtual imaging model and projected onto the display. The display images are captured by a real camera, and then the image transformation from the real camera to the virtual camera is established to achieve visual perception of the aircraft panoramic space in the virtual scene. This is used to simulate data collection during the aircraft test flight, and improve the stability and reliability of the video acquisition equipment during ground testing. The pixel-level difference value between the generated image and the virtual view camera image is calculated by the two-norm loss function, and the pixel-level difference value is compared with a preset difference value threshold to judge the acquired virtual view camera image, thereby improving the accuracy of the experimental samples required for the test. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 A schematic diagram of a flow chart of a method for simulating panoramic visual perception of an aircraft test flight provided by a first embodiment of the present invention; Figure 2 A control logic diagram of a panoramic visual perception simulation method for aircraft test flight provided by Embodiment 1 of the present invention; Figure 3 A schematic diagram of the positions of an aircraft and a virtual camera of an aircraft panoramic multi-view virtual imaging model of an aircraft test flight panoramic visual perception simulation system provided by a third embodiment of the present invention; Figure 4 A mapping relationship diagram of a panoramic visual perception simulation system for an aircraft test flight provided in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0015] The technical solution of the present invention is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations on the technical solution of the present invention. The embodiments of the present invention and the technical features in the embodiments may be combined with each other unless there is a conflict.

[0016] The term "and / or" is only a description of the association relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " generally indicates that the related objects are in an "or" relationship. Example

[0017] The present invention provides a method for simulating panoramic visual perception of an aircraft test flight, as shown in the attached Figure 1 and attached Figure 2 As shown, including: Controlling a virtual camera in a virtual scene to generate a multi-perspective image of an aircraft test flight through a panoramic multi-perspective virtual imaging model of the aircraft, and projecting the multi-perspective image of the aircraft test flight to a display; Processing the aircraft test flight multi-view images displayed on the display by Zhang Zhengyou's camera calibration method to calibrate the intrinsic parameters of the real camera; Based on the calibrated real camera intrinsic parameters, the aircraft test flight multi-view image displayed on the display is processed by the image dedistortion function of OpenCV to complete the image dedistortion, obtain the distortion-free aircraft test flight multi-view image, and project the distortion-free aircraft test flight multi-view image to the display.

[0018] Regarding the Zhang Zhengyou camera calibration method, the Zhang Zhengyou camera calibration method is a technology widely used in camera calibration. It was proposed by Dr. Zhang Zhengyou in his 1999 paper "Flexible Camera Calibration By Viewing a Plane From Unknown Orientations". This method is widely used because of its simple operation and high precision. In practical applications, the Zhang Zhengyou calibration method is usually used in combination with a computer vision library such as OpenCV. The calibration process is automatically completed by writing corresponding code, and the camera parameters are obtained; that is, this application uses the Zhang Zhengyou camera calibration method to process the aircraft test flight multi-view images displayed on the display, and uses the image dedistortion function of OpenCV to process the aircraft test flight multi-view images displayed on the display, both of which belong to the prior art and will not be repeated here.

[0019] Controlling a real camera to capture a display image, calibrating transformation external parameters of the real camera and the display based on the display image, and establishing a mapping relationship between the real camera and the display: Based on the display image captured by the real camera and the aircraft test flight multi-view image displayed on the display, the transformation matrix of the real camera and the display is solved: , Among them, the Collect images for real cameras, For display images, is the transformation matrix, i.e., the external parameters of the transformation; Based on the display image and the aircraft test flight multi-view image displayed on the display, a mapping relationship between the display and the virtual camera is established, which is expressed as ; According to the mapping relationship between the real camera and the display , the mapping relationship between the display and the virtual camera , establish a mapping relationship between the real camera and the virtual camera ; Control the virtual camera to continuously project real-time multi-view images of the aircraft test flight onto the display according to the real-time video of the aircraft test flight, and control the real camera to collect display images in real time; According to the mapping relationship between the real camera and the virtual camera, and the display image collected by the real camera in real time, a panoramic image of the aircraft test flight under the perspective of the real camera perceived and simulated in the virtual scene is obtained.

[0020] The distribution position of the virtual cameras in the virtual scene is consistent with the distribution position of the real cameras in the real scene.

[0021] At this point, the virtual perspective camera images of the aircraft's panoramic multi-perspective virtual imaging model can be collected through the real camera, and data collection and processing are completed. The obtained virtual perspective camera images and the configuration parameters of each virtual camera, such as internal and external parameters, are used for aerial target recognition and measurement.

[0022] A pre-built aircraft panoramic multi-view virtual imaging model is used to generate multi-view images of the aircraft and project them onto a display. A real camera is used to capture the display image, and then an image transformation from a real camera to a virtual camera is established to achieve visual perception of the aircraft's panoramic space in a virtual scene. This is used to simulate data collection during aircraft test flights, improving the stability and reliability of video acquisition equipment during ground testing. Example

[0023] The present invention provides an aircraft flight test abnormality testing method, comprising: Acquire panoramic images of aircraft test flights; Inputting the aircraft test flight panoramic image into the pre-trained adversarial network model to obtain a generated image, The adversarial network model includes a generator and a discriminator, and the training process of the adversarial network model includes: Input the training sample into a pre-built adversarial network model, and generate an image through a generator of the adversarial network model; Using a discriminator to determine the similarity between the image generated by the generator and the corresponding input training sample; Iteratively update the parameters of the generator through the gradient back-propagation algorithm until the similarity reaches a maximum; The pixel-level difference value between the generated image and the virtual view camera image is calculated by the two-norm loss function: , in, is the pixel-level difference value, is the virtual perspective camera image, To generate the image, If the pixel-level difference value is greater than a preset difference value threshold, it indicates that the aircraft test flight is abnormal; otherwise, it indicates that the aircraft test flight is normal and the detection of the aircraft test flight condition is completed; Among them, the aircraft test flight panoramic image is obtained by using the aircraft test flight panoramic visual perception simulation method described in Example 1, which has corresponding beneficial effects. Example

[0024] This embodiment provides an aircraft test flight panoramic visual perception simulation system, which can be used to execute the aircraft test flight panoramic visual perception simulation method described in Example 1, and has the beneficial effects of the aircraft test flight panoramic visual perception simulation method described in Example 1, as shown in the attached figure. Figure 3 and attached Figure 4 As shown, it includes: an aircraft panoramic multi-view virtual imaging model, a display and a real camera, wherein the aircraft panoramic multi-view virtual imaging model includes a virtual camera, wherein the distribution position of the virtual camera in the virtual scene is consistent with the distribution position of the real camera in the real scene; The aircraft panoramic multi-view virtual imaging model is a system that uses computer graphics and image processing technology to generate panoramic images of the inside and outside of the aircraft. This model can be used for a variety of purposes, including flight simulation training, virtual tourism, safety demonstration, entertainment, and visualization of aircraft design and manufacturing. It is an existing mature technology and will not be described in detail here. It also includes: an image generation module, which is used to control the virtual camera in the virtual scene to generate a multi-perspective image of the aircraft test flight through the aircraft panoramic multi-perspective virtual imaging model, and project the multi-perspective image of the aircraft test flight to a display; A mapping module is used to control a real camera to capture a display image, calibrate the transformation external parameters of the real camera and the display based on the display image, and establish a mapping relationship between the real camera and the display; establish a mapping relationship between the display and the virtual camera based on the display image and the aircraft test flight multi-view image displayed on the display; establish a mapping relationship between the real camera and the virtual camera according to the mapping relationship between the real camera and the display and the mapping relationship between the display and the virtual camera; The simulation module is used to control the virtual camera to continuously project real-time multi-view images of the aircraft test flight to the display according to the real-time video of the aircraft test flight, and control the real camera to collect the display image in real time; according to the mapping relationship between the real camera and the virtual camera, and the display image collected by the real camera in real time, obtain the aircraft test flight panoramic image under the perspective of the real camera perceived and simulated in the virtual scene; The aircraft multi-view images are generated through the pre-built aircraft panoramic multi-view virtual imaging model and projected to the display. The display images of the display are captured by a real camera, and then the image transformation from the real camera to the virtual camera is established to realize the visual perception of the aircraft panoramic space in the virtual scene. It is used to simulate the data collection during the aircraft test flight, and improve the stability and reliability of the video acquisition equipment during ground testing. The pixel-level difference value between the generated image and the virtual view camera image is calculated by the two-norm loss function, and the pixel-level difference value is compared with a preset difference value threshold to judge the acquired virtual view camera image, thereby improving the accuracy of the experimental samples required for the test. Example

[0025] Embodiment 4 of the present invention further provides an electronic terminal, comprising a processor and a memory connected to the processor, wherein a computer program is stored in the memory, and the processor is configured to operate according to the instructions to execute the steps of the method described in Embodiment 1 or Embodiment 2. Example

[0026] Embodiment 5 of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in Embodiment 1 or Embodiment 2 are implemented, and the computer program has the corresponding functional modules and beneficial effects of the execution method.

[0027] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for simulating panoramic visual perception of an aircraft test flight, characterized in that: include: Controlling a virtual camera in a virtual scene to generate a multi-perspective image of an aircraft test flight through a panoramic multi-perspective virtual imaging model of the aircraft, and projecting the multi-perspective image of the aircraft test flight to a display; Controlling a real camera to capture a display image, calibrating transformation extrinsic parameters of the real camera and the display based on the display image, and establishing a mapping relationship between the real camera and the display; Establishing a mapping relationship between the display and the virtual camera based on the display image and the aircraft test flight multi-view image displayed on the display; Establishing a mapping relationship between the real camera and the virtual camera according to a mapping relationship between the real camera and the display and a mapping relationship between the display and the virtual camera; Control the virtual camera to continuously project real-time multi-view images of the aircraft test flight onto the display according to the real-time video of the aircraft test flight, and control the real camera to collect display images in real time; According to the mapping relationship between the real camera and the virtual camera, and the display image captured by the real camera in real time, a panoramic image of the aircraft test flight under the perspective of the real camera perceived and simulated in the virtual scene is obtained; The distribution position of the virtual cameras in the virtual scene is consistent with the distribution position of the real cameras in the real scene.

2. The aircraft test flight panoramic visual perception simulation method according to claim 1, characterized in that: Before controlling the real camera to capture the display image: The multi-view images of the aircraft test flight displayed on the display are processed by Zhang Zhengyou's camera calibration method to calibrate the intrinsic parameters of the real camera.

3. The aircraft test flight panoramic visual perception simulation method according to claim 2, characterized in that: Also includes: Based on the calibrated real camera intrinsic parameters, the aircraft test flight multi-view image displayed on the display is processed by the image dedistortion function of OpenCV to complete the image dedistortion, obtain the distortion-free aircraft test flight multi-view image, and project the distortion-free aircraft test flight multi-view image to the display.

4. An aircraft flight test abnormality test method, characterized in that: The method comprises: Acquire panoramic images of aircraft test flights; Input the aircraft test flight panoramic image into a pre-trained adversarial network model to obtain a generated image, and calculate the pixel-level difference value between the generated image and the aircraft test flight panoramic image. If the pixel-level difference value is greater than a preset difference value threshold, it indicates that the aircraft test flight is abnormal; otherwise, it indicates that the aircraft test flight is normal; Wherein, the aircraft test flight panoramic image is obtained by using the aircraft test flight panoramic visual perception simulation method according to any one of claims 1 to 3.

5. The aircraft flight test abnormality testing method according to claim 4, characterized in that: The adversarial network model includes a generator and a discriminator, and the training process of the adversarial network model includes: Inputting the training sample into a pre-built adversarial network model, and generating an image through a generator of the adversarial network model; Using a discriminator to determine the similarity between the image generated by the generator and the corresponding input training sample; The parameters of the generator are iteratively updated through the gradient back-propagation algorithm until the similarity reaches the maximum.

6. The aircraft flight test abnormality testing method according to claim 4, characterized in that: Calculating the pixel-level difference value between the generated image and the virtual view camera image includes: The pixel-level difference value between the generated image and the virtual view camera image is calculated by the two-norm loss function: ， in, is the pixel level difference value; is the virtual perspective camera image, To generate images.

7. An aircraft test flight panoramic visual perception simulation system, characterized in that: include: An image generation module is used to control a virtual camera in a virtual scene to generate a multi-perspective image of an aircraft test flight through a panoramic multi-perspective virtual imaging model of the aircraft, and project the multi-perspective image of the aircraft test flight to a display; A mapping module, used to control a real camera to capture a display image, calibrate transformation external parameters of the real camera and the display based on the display image, and establish a mapping relationship between the real camera and the display; Based on the display image and the aircraft test flight multi-view images displayed on the display, a mapping relationship between the display and the virtual camera is established; based on the mapping relationship between the real camera and the display and the mapping relationship between the display and the virtual camera, a mapping relationship between the real camera and the virtual camera is established; A simulation module, used to control the virtual camera to continuously project real-time multi-view images of the aircraft test flight onto the display according to the real-time video of the aircraft test flight, and to control the real camera to collect display images in real time; According to the mapping relationship between the real camera and the virtual camera, and the display image captured by the real camera in real time, a panoramic image of the aircraft test flight under the perspective of the real camera perceived and simulated in the virtual scene is obtained; The distribution position of the virtual cameras in the virtual scene is consistent with the distribution position of the real cameras in the real scene.

8. An electronic terminal, characterized in that: The method comprises a processor and a memory connected to the processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the steps of the method according to any one of claims 1 to 3 or claims 4 to 6 are performed.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 3 or claims 4 to 6 are implemented.

Images