A reliability testing method for intelligent assisted driving based on virtual camera

Abstract

The present invention discloses a reliability test method for intelligent assisted driving based on a virtual camera, comprising the following steps: building a full-motion simulation test platform; setting virtual camera parameters and embedding an airport model; sending visual cabin external image data to the intelligent assisted driving software after time synchronization, and the intelligent assisted driving software provides a door-to-door automatic flight control operation procedure; the runway feature image and data information calculated in real time by the intelligent assisted driving system are marked and displayed on the PFD and / or a schematic page is displayed on the ND; the pilot completes the test subject through the full-motion simulation test platform. This application provides full-motion simulation test platform test items through different test conditions and environments, including scene taxiing test, visual take-off and landing process flight test and door-to-door automatic flight function test, and uses simulator data to test the functional reliability of the assisted driving software, thereby reducing the cost of the assisted driving system and improving the accuracy of assisted driving.

Description

Technical Field

[0001] The present invention belongs to the technical field of control system testing, and in particular relates to a virtual camera-based intelligent assisted driving reliability testing method. Background Art

[0002] Intelligent assisted driving system refers to a series of aircraft driving assistance systems based on advanced sensors, artificial intelligence and computer vision technology, which aims to improve driving safety and convenience. Intelligent assisted driving system can help pilots better perceive the flight environment and monitor the status of other aircraft in the air, thereby ensuring flight safety.

[0003] At present, the intelligent assisted driving system is in the initial design stage and the industry is developing slowly. In order to verify the accuracy, integrity, reliability, timeliness, economy, verifiability and safety of the assisted driving principle system, a full-motion simulation test platform for the intelligent assisted driving system is urgently needed to test the intelligent assisted driving system. Summary of the invention

[0004] The purpose of the present invention is to provide a reliability testing method for intelligent assisted driving based on a virtual camera. By adding a virtual camera model in a visual system, different test conditions and environments are provided to provide a full-motion simulation test platform. Test items include scene taxiing test, visual take-off and landing process flight test and door-to-door automatic flight function test. The reliability of the assisted driving software function is tested using simulator data, which solves the problems of the existing assisted driving system being high in cost, insufficient in accuracy and unable to truly simulate the driving environment.

[0005] In order to solve the above technical problems, the present invention is achieved through the following technical solutions:

[0006] The present invention is a virtual camera-based intelligent assisted driving reliability testing method, comprising the following steps:

[0007] Step S1: Build a full-motion simulation test platform and add a virtual camera model to the visual system;

[0008] Step S2: Setting the internal and external parameters of the virtual camera and embedding the airport model;

[0009] Step S3: The image data outside the visual cabin is sent to the intelligent assisted driving software after time synchronization;

[0010] Step S4: The intelligent assisted driving software provides a door-to-door automatic flight control operation procedure and cooperates to complete the design and implementation of the corresponding control logic;

[0011] Step S5: the runway characteristic image and data information calculated in real time by the intelligent assisted driving system are marked and displayed on the PFD and / or a simplified diagram page is displayed on the ND;

[0012] Step S6: The pilot completes the test subjects through the full-motion simulation test platform, including the surface taxiing subject, the visual take-off and landing test flight subject, and the door-to-door automatic flight function test subject.

[0013] As a preferred technical solution, in step S1, the full-motion simulation test platform includes a visual system, a full-motion simulator, a full-motion simulation test platform and an intelligent assisted driving computer; the full-motion simulator is used to provide data to the full-motion simulation test platform;

[0014] The visual system supports the embedding of multiple airport model scenes and visual cabin external image data; the visual system contains multiple airport database scenes for completing the simulation of the aircraft take-off and landing operating environment; the visual cabin external image data is sent to the assisted driving software in the intelligent assisted driving computer after time synchronization;

[0015] The intelligent assisted driving computer is used to achieve data cross-linking with the full-motion simulation test platform; the data cross-linking includes airborne data transmission and cross-linking testing of the intelligent assisted driving software.

[0016] As a preferred technical solution, the full-motion simulation test platform is used to assist the intelligent assisted driving software to complete the following test contents:

[0017] Content 1: Surface taxiing test, testing the surface taxiing line detection and obstacle detection functions and performance of the auxiliary driving system;

[0018] Content 2: Visual take-off and landing process flight test, testing the navigation and guidance functions and performance of the intelligent assisted driving system during take-off, approach and landing;

[0019] Content 3: Gate-to-gate automatic flight function test, testing the gate-to-gate automatic flight function and performance of the intelligent assisted driving system during the entire simulated route flight.

[0020] As a preferred technical solution, the full-motion simulation test platform is used to realize the primary flight display PFD and navigation display ND display interface and the cockpit airborne terminal data interface.

[0021] As a preferred technical solution, in step S2, the virtual camera simulation computer is installed with NetVDNAPP, a visual system desktop program and an Obs studio program; the virtual camera simulation computer is used to realize virtual camera simulation, extravehicular visual image acquisition and transmission, and data interaction and cross-linking testing between the full-motion simulator and the intelligent assisted driving computer.

[0022] As a preferred technical solution, the NetVDN APP is used for data interaction and cross-linking testing between the full-motion simulator and the intelligent assisted driving software, specifically including: providing a test interface, the test interface can collect data from each simulation system of the full-motion simulator, provide relative position data of the virtual camera, inertial navigation, and GPS airborne equipment in the body coordinate system, and transmit it to the intelligent assisted driving high-performance platform via Ethernet at an agreed frequency; at the same time, the test interface receives control data of the intelligent assisted driving system, cross-links the intelligent assisted driving system with the flight simulation equipment, realizes data interaction, and completes the verification test; the NetVDN APP realizes the configuration of the virtual camera external parameters, and after the configuration is completed, it is synchronized with the data of the full-motion flight simulator in real time, simulates the real camera external parameter parameters, and sends the virtual camera internal and external parameters to the intelligent assisted driving system and to the intelligent assisted driving computer at an agreed frequency.

[0023] As a preferred technical solution, in step S4, the intelligent assisted driving computer control instruction execution process is as follows:

[0024] Step S41: loading initial conditions;

[0025] Step S42: The aircraft is trimmed;

[0026] Step S43: Freeze;

[0027] Step S44: the full motion simulator is started;

[0028] Step S45: loading a virtual camera computer;

[0029] Step S46: transmitting to the intelligent assisted driving high-performance computing platform;

[0030] Step S47: the intelligent assisted driving high-performance computing platform issues a target instruction;

[0031] Step S48: the intelligent assisted driving system overrides the full-motion simulator through the digital pilot;

[0032] Step S49: The test ends.

[0033] As a preferred technical solution, in step S5, the intelligent assisted driving system uses a quaternion q to control the aircraft orientation, and gives an Euler rotation (X, Y, Z), that is, rotating X, Y, Z around the x-axis, y-axis, and z-axis respectively, to obtain the following formula:

[0034] ;

[0035] The corresponding quaternion .

[0036] As a preferred technical solution, in step S5, the driving force of the full-motion simulation test platform during the test , sliding resistance , Slope resistance , wind resistance , according to the balance of forces during the virtual aircraft test, the basic mathematical model of aircraft speed is obtained:

[0037] ;

[0038] ;

[0039] In the formula, is the speed of the aircraft, is the aircraft weight, is the rolling resistance, is the slope, is the air resistance coefficient, is the frontal area of ​​the aircraft, is the aircraft speed, is the steering angle, is the turning radius of the aircraft, is the steering wheel angle, is the coefficient of turning radius and turning angle.

[0040] The present invention has the following beneficial effects:

[0041] (1) The present invention mainly relates to a flight simulation test platform, which provides an intelligent assisted driving system for testing on a full-motion simulation test platform, provides different test conditions and environments, and provides full-motion simulation test platform test items, including field taxiing test, visual take-off and landing process flight test, and door-to-door automatic flight function test. The reliability of the assisted driving software function is tested using simulator data, which reduces the cost of the assisted driving system and improves the accuracy of assisted driving.

[0042] (2) The present invention improves the accuracy of assisted driving by adding a virtual camera model to the vision system for data interaction and performing image processing, setting and calibration in the virtual camera, and further evaluates the rationality of the algorithm and software and hardware design of the intelligent assisted driving system and its functional applicability in a near-real operating environment.

[0043] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0045] Figure 1 This is a flow chart of a reliability testing method for intelligent assisted driving based on a virtual camera of the present invention;

[0046] Figure 2 This is a schematic diagram of the structure of an intelligent assisted driving reliability test system based on a virtual camera;

[0047] Figure 3 It is a flowchart for the execution of control instructions for the intelligent assisted driving computer;

[0048] Figure 4 This is a schematic diagram of the effect of a field of view of 100×20;

[0049] Figure 5 This is a schematic diagram of the effect of a field of view of 60×40;

[0050] Figure 6 Virtual camera image for the calibration process;

[0051] Figure 7 This is the calibration process image in MatLab;

[0052] Figure 8 Displays the logic diagram for the intelligent assisted driving system. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0055] In order to make the purpose, technical solution and advantages of this application clearer, the following Figure 2 The implementation methods of the present application are described in further detail.

[0056] Before introducing the embodiments of the present application, the full-motion simulation test platform is first described.

[0057] The full-motion simulation test platform meets the following requirements:

[0058] (1) The test platform is provided with data by a full-motion simulator.

[0059] (2) Vision system: The Collins EP-8100 vision system is the preferred vision system. The vision system adds a virtual camera model, sets the internal and external parameters of the virtual camera, has an adjustable focal length, and supports parameter calibration of the virtual camera. The vision system supports scene embedding of airport models such as other aircraft and vehicles. The vision system includes databases of Beijing Capital, Beijing Daxing, Shanghai Hongqiao, Shanghai Pudong and other airports to support the simulation of operating environments such as aircraft takeoff and landing. The image data outside the vision cabin is sent to the intelligent assisted driving software after time synchronization.

[0060] (3) Data cross-linking: The test platform provides the relative position data of virtual cameras, inertial navigation, GPS and other airborne equipment in the body coordinate system and the relevant interface data format, transmission protocol, frequency and other information required for test verification, and also provides the airborne flight data agreed upon by both parties after time synchronization; the full-motion simulation test platform provides the simplest operation procedure of door-to-door automatic flight control, and cooperates to complete the design and implementation of the corresponding control logic. The control logic mainly replaces the pilot's control of the automatic flight control panel and the control of the turning hand wheel, throttle lever, brake and other operations during taxiing; the test platform supports the cross-linking integration test verification of the intelligent assisted driving system, which meets the requirements of integrating the position solution information of the intelligent assisted driving system into the automatic flight system and replacing the original navigation guidance signal.

[0061] (4) PFD and ND display: Based on the PFD and ND display design of current civil aviation passenger aircraft (such as A320, B737, C919), the corresponding display interface and cockpit airborne data interface are developed using commonly used display interface development languages ​​such as WPF, C#, QT or VAPS XT; the PFD overlays and displays the image information of the extravehicular environment from the pilot's perspective, and the images and data information such as the runway characteristics calculated in real time by the intelligent assisted driving system are marked and displayed on the PFD; a new window (simple diagram page) is added to the ND to display information related to the taxiing phase of the scene, display key scene data information such as the online airport moving map, the position of the aircraft itself, and the obstacle detection result information calculated in real time by the intelligent assisted driving system; the planned taxiing path is displayed on the airport moving map, and displayed from a bird's-eye view.

[0062] (5) The test requires the pilot to cooperate in completing the test subjects, including surface taxiing subjects, visual takeoff and landing test flight subjects, and door-to-door automatic flight function test subjects.

[0063] In order to make the purpose, technical solutions and advantages of this application more clear, the following Figure 1 -Attached Figure 8It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0064] Embodiment 1

[0065] See also Figure 1 As shown, the present invention is a method for testing the reliability of intelligent assisted driving based on a virtual camera, comprising the following steps:

[0066] Step S1: Build a full-motion simulation test platform and add a virtual camera model to the visual system;

[0067] Step S2: Setting the internal and external parameters of the virtual camera and embedding the airport model;

[0068] Step S3: The image data outside the visual cabin is sent to the intelligent assisted driving software after time synchronization;

[0069] Step S4: The intelligent assisted driving software provides a door-to-door automatic flight control operation procedure and cooperates to complete the design and implementation of the corresponding control logic;

[0070] Step S5: the runway characteristic image and data information calculated in real time by the intelligent assisted driving system are marked and displayed on the PFD and / or a simplified diagram page is displayed on the ND;

[0071] Step S6: The pilot completes the test subjects through the full-motion simulation test platform, including the surface taxiing subject, the visual take-off and landing test flight subject, and the door-to-door automatic flight function test subject.

[0072] In step S1, the full-motion simulation test platform includes a visual system, a full-motion simulator, a full-motion simulation test platform and an intelligent auxiliary driving computer; the full-motion simulator is used to provide data to the full-motion simulation test platform;

[0073] The visual system supports the embedding of multiple airport model scenes and image data outside the visual cabin; the visual system contains multiple airport database scenes to complete the simulation of the aircraft take-off and landing operating environment; the image data outside the visual cabin is sent to the assisted driving software in the intelligent assisted driving computer after time synchronization;

[0074] The intelligent assisted driving computer is used to achieve data cross-linking with the full-motion simulation test platform; data cross-linking includes onboard data transmission and cross-linking testing of the intelligent assisted driving software.

[0075] The full-motion simulation test platform is used to assist intelligent assisted driving software in completing the following test contents:

[0076] Content 1: Surface taxiing test, testing the surface taxiing line detection and obstacle detection functions and performance of the auxiliary driving system;

[0077] Content 2: Visual take-off and landing process flight test, testing the navigation and guidance functions and performance of the intelligent assisted driving system during take-off, approach and landing;

[0078] Content 3: Gate-to-gate automatic flight function test, testing the gate-to-gate automatic flight function and performance of the intelligent assisted driving system during the entire simulated route flight.

[0079] The full-motion simulation test platform is used to realize the primary flight display PFD and navigation display ND display interface and the cockpit airborne data interface.

[0080] In step S2, the virtual camera simulation computer is installed with NetVDN APP, the visual system desktop program and the Obsstudio program; the virtual camera simulation computer is used to realize virtual camera simulation, extravehicular visual image acquisition and transmission, full-motion simulator and intelligent assisted driving computer data interaction and cross-linking test.

[0081] NetVDN APP is used for data interaction and cross-linking testing between the full-motion simulator and the intelligent assisted driving software, including: providing a test interface, which can collect data from each simulation system of the full-motion simulator, provide relative position data of the virtual camera, inertial navigation, and GPS airborne equipment in the body coordinate system, and transmit it to the intelligent assisted driving high-performance platform via Ethernet at an agreed frequency; at the same time, the test interface receives the control data of the intelligent assisted driving system, cross-links the intelligent assisted driving system with the flight simulation equipment, realizes data interaction, and completes the verification test; NetVDN APP realizes the configuration of the virtual camera external parameters, and after the configuration is completed, it is synchronized with the data of the full-motion flight simulator in real time, simulates the real camera external parameter parameters, and sends the virtual camera internal and external parameters to the intelligent assisted driving system and to the intelligent assisted driving computer at an agreed frequency.

[0082] In step S4, the intelligent assisted driving computer control instruction execution process is as follows:

[0083] Step S41: loading initial conditions;

[0084] Step S42: The aircraft is trimmed;

[0085] Step S43: Freeze;

[0086] Step S44: the full motion simulator is started;

[0087] Step S45: loading a virtual camera computer;

[0088] Step S46: transmitting to the intelligent assisted driving high-performance computing platform;

[0089] Step S47: the intelligent assisted driving high-performance computing platform issues a target instruction;

[0090] Step S48: the intelligent assisted driving system overrides the full-motion simulator through the digital pilot;

[0091] Step S49: The test ends.

[0092] In step S5, the intelligent assisted driving system uses a quaternion q to control the direction of the aircraft and gives an Euler rotation (X, Y, Z), that is, rotating X, Y, Z around the x-axis, y-axis, and z-axis respectively, and the formula is as follows:

[0093] ;

[0094] The corresponding quaternion .

[0095] In step S5, the driving force of the full-motion simulation test platform during the test , sliding resistance , Slope resistance , wind resistance , according to the balance of forces during the virtual aircraft test, the basic mathematical model of aircraft speed is obtained:

[0096] ;

[0097] ;

[0098] In the formula, is the speed of the aircraft, is the aircraft weight, is the rolling resistance, is the slope, is the air resistance coefficient, is the frontal area of ​​the aircraft, is the aircraft speed, is the steering angle, is the turning radius of the aircraft, is the steering wheel angle, is the coefficient of turning radius and turning angle.

[0099] Embodiment 2

[0100] See also Figure 2As shown, the present invention is a virtual camera-based intelligent assisted driving reliability test system, which can be used to execute the method content of embodiment 1 and subsequent embodiment 3 of the present invention. This embodiment provides a full-motion simulation test platform for testing the intelligent assisted driving system. The full-motion simulation test platform visual system realizes the addition of a virtual camera model, and the transmission of the out-of-cabin visual image data after time synchronization; realizes the data cross-linking between the full-motion simulation test platform and the intelligent assisted driving software, including airborne data transmission and intelligent assisted driving software cross-linking test; realizes the display interface of the primary flight display (PFD: Primary FlightDisplay) and the navigation display (ND: Navigation Display) and the cockpit airborne terminal data interface.

[0101] The full-flight simulator is a device that has passed the D-level certification and can provide simulated flight training in the full flight phase. The full-flight simulation test platform is used to ensure that the D-level delivery status of the full-flight simulator remains unchanged to ensure the accuracy of the simulation data of the full-flight simulator in the full flight phase. The full-flight simulation test platform is independent of the full-flight simulator as a whole, and data can be exchanged with the full-flight simulator by connecting the switch device with a network cable. The modification test environment can be set up without complex operations such as electrical modification, reducing the waiting time before the test preparation.

[0102] The test platform can help intelligent assisted driving software support the following test contents:

[0103] 1) Field taxi test, to test the functions and performance of the auxiliary driving system such as field taxi line detection and obstacle detection;

[0104] 2) Visual take-off and landing process flight test, testing the navigation and guidance functions and performance of the intelligent assisted driving system during take-off, approach and landing;

[0105] 3) Gate-to-gate automatic flight function test, testing the gate-to-gate automatic flight function and performance of the intelligent assisted driving system during the entire simulated route flight process.

[0106] The overall system consists of a full-motion simulator, a full-motion simulation test platform and an intelligent assisted driving computer.

[0107] The equipment information of the full-motion simulation test platform is as follows:

[0108] 1) Virtual camera simulation computer includes:

[0109] Graphics image server, independent graphics card, Windows system, installed visual system desktop software, preferably the corresponding desktop version of the visual system software, terrain library and airport model consistent with the full motion simulator, Obs studio, NetVDN APP.

[0110] 2) Network switch model, preferably GS724T-400PRS.

[0111] 3) Wireless router.

[0112] 4) The intelligent assisted driving display system includes: Windows system and intelligent assisted driving display PAD.

[0113] The details are shown in Table 1 below.

[0114] Table 1 shows the system names and their technical indicators

[0115] 1 Full-motion simulator test virtual camera computer Software modification plan for the full-motion simulator platform visual system (visual system corresponding to the desktop software and Obsstudio program), simulator platform cross-linking test and gate-to-gate automatic flight control logic design and modification plan (NetVDN program and digital pilot design), simulator platform display plan (PFD and ND display and data interface) content 2 Intelligent assisted driving display Full-motion simulator display interactive flight software, based on civil airliner (such as A320, B737, C919) PFD and ND style display interactive software 3 Full-motion simulation test platform test items Full-motion simulator platform intelligent assisted driving system training data set, that is, the full-motion simulator assisted driving system all test training data sets planned according to the test outline

[0116] The virtual camera simulation computer is installed with NetVDN APP, the visual system desktop program and Obs studio program. The virtual camera simulation computer mainly realizes virtual camera simulation, extravehicular visual image acquisition and transmission, data interaction between the full-motion simulator and the intelligent assisted driving computer, and cross-linking test.

[0117] NetVDN APP is used to realize data interaction and cross-linking test between full-motion simulator and intelligent assisted driving software. NetVDN APP can provide the required relevant interface data format, transmission protocol, frequency and other information.

[0118] The main functions of NetVDN APP are: providing a test interface, which can collect data from various simulation systems of the full-motion simulator (such as airborne sensor data simulated by the full-motion flight simulator, such as IRS, GPS, RA, etc.), provide relative position data of airborne equipment such as virtual cameras, inertial navigation, GPS, etc. in the body coordinate system, and transmit it to the intelligent assisted driving high-performance platform through Ethernet at an agreed frequency (such as 20HZ); at the same time, the test interface can receive control data of the intelligent assisted driving system, realize the cross-linking of the intelligent assisted driving system and the flight simulation equipment, realize data interaction, and complete the verification test; NetVDN APP realizes the configuration of the virtual camera external parameters, and after the configuration is completed, it is synchronized with the data of the full-motion flight simulator in real time, simulates the real camera external parameter parameters, and sends the virtual camera internal and external parameters to the intelligent assisted driving system (Note: the virtual camera internal parameters are configured through the corresponding desktop version software of the vision system) and sent to the intelligent assisted driving computer at an agreed frequency (such as 20Hz).

[0119] The content of the data that NetVDN APP subscribes to and sends from the full-motion simulator can be agreed upon according to the needs of the intelligent assisted driving computer. The data may include: aircraft position (longitude, latitude, high speed), aircraft attitude (pitch angle, roll angle, yaw angle), inertial navigation data (inertial navigation data-XYZ three-axis acceleration angular velocity, inertial navigation data-combined inertial navigation north-east speed, inertial navigation data-combined inertial navigation longitude and latitude altitude), GNSS data (satellite latitude, satellite longitude, satellite altitude, satellite status), GPS-UTC time, sampling time, magnetic heading, some information in the scenery database (position and order of the four corner points of the runway), vertical deviation and horizontal deviation calculated by automatic flight, timestamp (main control computer unified clock), and flight phase.

[0120] NetVDN APP supports the simplest operation procedure of automatic flight control from door to door through digital pilots, and cooperates to complete the design and implementation of corresponding control logic. The digital pilot simulates receiving the target instructions sent by the computer of the intelligent assisted driving system, and converts the target instructions into control instructions for the simulator systems such as engines, flight control, automatic flight, brakes, landing gear systems, etc. The full-motion simulator responds to the control instructions and controls the corresponding systems to implement the test. During the test, the full-motion simulator decides whether to end the test according to the time and target values ​​(such as altitude, speed, attitude, etc.) set by the digital pilot. This function is implemented through the virtual camera simulation computer NetVDN APP, and it can be decoupled from the hardware as needed during the test. The digital pilot supports the cross-linked integration test verification of the intelligent assisted driving system, which meets the requirements of integrating the position solution information of the intelligent assisted driving system into the automatic flight system and replacing the original navigation guidance signal. The control instructions sent by the intelligent assisted driving system to other systems of the full-motion simulator include the following: (the data used for control can be added as needed, and the cockpit control list is shown in the appendix: Cockpit control list table) vertical deviation, horizontal deviation runway and center offset (angle, position).

[0121] Door-to-door related trigger signals, such as autopilot (AP) and autothrottle (AT).

[0122] The intelligent assisted driving system uses a digital pilot to control the full-motion simulator to test the control execution process such as Figure 3 described.

[0123] The desktop version of the visual system software simulates the virtual camera to realize related functions. During the test, the desktop version of the visual system software and the simulator vision simultaneously receive instructions sent by the full-motion simulator main control computer, respond and display images in real time, ensuring that the full-motion simulator vision and virtual camera responses are consistent.

[0124] The visual system corresponds to the desktop version and installs the same scenery and airport database as the full-motion simulator to ensure consistent display. The airport database provides hub airport databases including Beijing Capital, Beijing Daxing, Shanghai Hongqiao, Shanghai Pudong, etc., to meet the testing of airport take-off and landing scenarios and operation simulation environments.

[0125] The desktop version of the visual system supports the setting of virtual camera internal and external parameters. The virtual camera internal parameters are configured through the desktop version of the visual system, including (fovh (horizontal field of view angle), fovv (vertical field of view angle), h (camera orientation), p (camera pitch angle), r (camera roll angle), zoom, near, far); the virtual camera external parameters include the position x, y, z of the virtual camera relative to the aircraft eye point, and the longitude and latitude data of the virtual camera in the world coordinate system. The longitude and latitude data of the virtual camera external parameters are obtained by calculating the offset relative to the aircraft eye point through the real-time aircraft eye point position data obtained in the NetVDN APP. The desktop version of the visual system can set the image resolution. It supports the function of adding obstacles such as vehicles, aircraft, pedestrians, etc. in the desktop version of the visual system.

[0126] The field of view can be adjusted as needed, and the virtual camera has an adjustable focal length function. The display effects of virtual camera scenes with different field of view angles are shown in Figures 4 and 5.

[0127] See also Figure 6 As shown, the virtual camera supports internal calibration and provides corresponding camera calibration results. The customization method is as follows:

[0128] 1) Build a 10×10 chessboard 6m×6m calibration board model and import it into the corresponding desktop software of the vision system.

[0129] 2) Fix the calibration plate, control the position and angle of the virtual camera eye point, and obtain the video image by taking screenshots.

[0130] 3) Use MATLAB's camera calibration toolbox to calibrate the virtual camera.

[0131] 4) MATLAB calibration, such as Figure 7 shown.

[0132] 5) Two methods to confirm the calibration results:

[0133] i. Provide the calibration results as input to the intelligent assisted driving computer, calculate the distance to a certain position through the algorithm, calculate the actual distance through the actual position coordinates, compare the two, and verify the calibration results.

[0134] ii. Compare the X, Y, and Z distances of the calibration object from the eye point recorded in the vision computer with the X, Y, and Z distances during the calibration process in MatLab to verify the calibration results.

[0135] The virtual camera simulation computer uses Obsstudio to add timestamps to the visual image data generated by the corresponding desktop version of the visual system and converts the image data into a video stream format (RAW, YUV, RGB) acceptable to the intelligent assisted driving high-performance platform, and transmits it to the intelligent assisted driving system computer via USB or Ethernet at the agreed frequency.

[0136] Embodiment 3

[0137] The embodiment of the present invention is a reliability testing method for intelligent assisted driving based on a virtual camera. The corresponding method content is the content embodied after the technical solution of Example 1 is combined with a specific platform, tools and algorithms. In addition, the embodiment of the present invention is explained from the theoretical side of the basic method, and the related preferred solutions in Example 1 are not excessively explained in the embodiment of the present invention.

[0138] Embodiment 4

[0139] This application realizes the development of PFD, ND display interface and data interface of intelligent assisted driving display system. The intelligent assisted driving display system software is based on the PFD and ND display design of civil aircraft (such as C919), developed using WPF or GL studio, and installed and run on the display system computer (tablet computer). The tablet computer is installed on the side bracket of the pilot in the cockpit. This solution will not modify the instrument equipment in the full-motion simulator and will not affect the normal display of the instruments in the full-motion simulator.

[0140] The functions of the intelligent assisted driving display system include PFD and ND display. The PFD of the intelligent assisted driving display system is based on the PFD display design of civil aviation passenger aircraft (such as C919), and the image information of the extra-cabin environment from the pilot's perspective is superimposed and displayed. The images and data information such as runway characteristics calculated in real time by the intelligent assisted driving system are marked and displayed on the PFD and / or a simple diagram page is displayed on the ND; the ND of the intelligent assisted driving display system is based on the ND display design of civil aviation passenger aircraft (such as C919). A new window is added to display relevant information of the taxiing stage of the scene, key scene data information such as online maps and the position of the main aircraft, and obstacle detection result information calculated in real time by the intelligent assisted driving system, etc., and the planned taxiing path is displayed on the airport map, with a bird's-eye view.

[0141] PFD and ND display data are derived from the full-motion simulator and the intelligent assisted driving computer. The data sent by the intelligent assisted driving computer include:

[0142] Image data (processed by the intelligent assisted driving system), vertical deviation, horizontal deviation during approach, aircraft to runway threshold, offset from takeoff runway center, ground taxiing guidance tracks (number), ground obstacle size, ground obstacle location, ground obstacle type, and number of ground obstacles.

[0143] The intelligent assisted driving display system is designed for two stages, namely the take-off and landing stage and the taxiing stage. The display system realizes the content display of different stages by subscribing to the flight stages of the full-motion simulator (taxiing, take-off, landing, cruising, and climbing). The basic display logic of the intelligent assisted driving system is as follows: Figure 8 shown.

[0144] The PFD display style is changed according to different flight models. This application takes the civil aviation passenger aircraft C919 as an example. C919 includes functions such as FMA, ILS, ALT, ATT, DEV, HDG, LCOR, SPD, VS, etc. This application parses the ARINC data of the full-motion simulator instrument, binds the parsed variables to the PFD controls, and drives the controls to display the effects. The flight and ARINC data displayed in the PFD are obtained in real time from the full-motion simulator via Ethernet.

[0145] Another part of the functional data of PFD comes from the intelligent assisted driving computer. The data calculated by the intelligent assisted driving computer (such as vertical deviation, horizontal deviation, distance from the aircraft to the runway entrance during approach, takeoff run center offset, ground taxiing guidance track (number), ground obstacle size, ground obstacle position, ground obstacle type, and ground obstacle number) are obtained from the intelligent assisted driving system through HDMI or UDP and displayed as prompt information content in the PFD; the video stream processed by the intelligent assisted driving system is obtained from the intelligent assisted driving system through HDMI or UDP and superimposed in the PFD. The video display frequency is greater than or equal to 10Hz.

[0146] The PFD will overlay the camera images and annotation result information sent by the intelligent assisted driving system during the taxiing and take-off and landing phases. The PFD will not overlay the intelligent assisted driving system data during other phases. The PFD will display the taxiing, take-off and landing phases.

[0147] This application is based on the current C919 ND display design and displays information related to the taxiing stage. A new window is added to display key scene data information such as online airport maps, the position of the aircraft, and obstacle detection result information calculated in real time by the intelligent assisted driving system. The style of the simple diagram page is adjusted according to different situations. This application only provides a schematic diagram.

[0148] ND displays a 90-degree bird's-eye view using a combination of airport maps and data. The aircraft's location data (longitude, latitude, altitude) and basic information about the aircraft (flight number, model, time) are derived from the full-motion simulator. Taxi guidance trajectories, obstacle information, etc. are theoretically derived from the intelligent auxiliary computer and can be adjusted as needed based on actual conditions. ND obtains this type of data from the intelligent auxiliary driving computer via HDMI or UDP. The data update frequency is greater than or equal to 10Hz.

[0149] ND display supports the following functions:

[0150] 1) The airport mobile map is a two-dimensional plane map or a high-definition satellite map of the airport;

[0151] 2) It can display the basic information of the aircraft being driven, and display it in the upper left corner, such as flight number, model, time, etc.;

[0152] 3) The aircraft position can be displayed on the ND / side display and indicated by an arrow symbol, and the position is located in the center of the screen in real time;

[0153] 4) Taxi guidance perspective: It can display the taxi guidance track, which starts from the current position of the aircraft, passes through key nodes, and points to the taxi end point in real time. The display color is green and provides zoom function;

[0154] 5) When passing a key node, the upper left corner will be enlarged and the corresponding intersection will be indicated by a yellow arrow. When passing a non-key node, the display will be hidden. When missing a key node, an alarm will be displayed, with a red background and white text in the upper right corner to indicate a missed intersection alarm;

[0155] 6) When an obstacle is detected on the sliding track, the track display color from the obstacle to the end point turns red, and a prompt is given that there is an obstacle ahead, and basic information is provided in white text on a red background in the upper right corner;

[0156] 7) The lower right corner provides the distance from the current position to the end point;

[0157] 8) Provides two modes of display: north direction display and heading direction display, which can be switched;

[0158] 9) Ability to mark and display surrounding aircraft and other objects and display conflicts between the currently planned trajectory and other flight trajectories.

[0159] It is worth noting that in the above system embodiment, the various units included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0160] In addition, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiments can be completed by instructing related hardware through a program, and the corresponding program can be stored in a computer-readable storage medium.

[0161] The preferred embodiments of the present invention disclosed above are only used to help illustrate the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A virtual camera-based intelligent assisted driving reliability testing method, characterized in that: The steps include: Step S1: Build a full-motion simulation test platform and add a virtual camera model to the visual system; Step S2: Setting the internal and external parameters of the virtual camera and embedding the airport model; Step S3: The image data outside the visual cabin can be sent to the intelligent assisted driving software after time synchronization; Step S4: The intelligent assisted driving software provides a door-to-door automatic flight control operation procedure and cooperates to complete the design and implementation of the corresponding control logic; Step S5: the runway feature image and data information calculated in real time by the intelligent assisted driving system are marked and displayed on the PFD and / or a simplified diagram page is displayed on the ND; Step S6: The pilot completes the test subjects through the full-motion simulation test platform, including the surface taxiing subject, the visual take-off and landing test flight subject, and the door-to-door automatic flight function test subject; In step S1, the full-motion simulation test platform includes a visual system, a full-motion simulator, a full-motion simulation test platform and an intelligent assisted driving computer; the full-motion simulator is used to provide data to the full-motion simulation test platform; The visual system supports the embedding of multiple airport model scenes and visual cabin external image data; the visual system contains multiple airport database scenes for completing the simulation of the aircraft take-off and landing operating environment; the visual cabin external image data is sent to the assisted driving software in the intelligent assisted driving computer after time synchronization; The intelligent assisted driving computer is used to realize data cross-linking with the full-motion simulation test platform; the data cross-linking includes airborne data transmission and cross-linking testing of the intelligent assisted driving software; In step S4, the intelligent assisted driving computer control instruction execution process is as follows: Step S41: loading initial conditions; Step S42: The aircraft is trimmed; Step S43: Freeze; Step S44: the full motion simulator is started; Step S45: loading a virtual camera computer; Step S46: transmitting to the intelligent assisted driving high-performance computing platform; Step S47: the intelligent assisted driving high-performance computing platform issues a target instruction; Step S48: the intelligent assisted driving system overrides the full-motion simulator through the digital pilot; Step S49: The test ends.

2. According to claim 1, a virtual camera-based intelligent assisted driving reliability testing method is characterized in that: The full-motion simulation test platform is used to assist the intelligent assisted driving software to complete the following test contents: Content 1: Surface taxiing test, testing the surface taxiing line detection and obstacle detection functions and performance of the auxiliary driving system; Content 2: Visual take-off and landing process flight test, testing the navigation and guidance functions and performance of the intelligent assisted driving system during take-off, approach and landing; Content 3: Gate-to-gate automatic flight function test, testing the gate-to-gate automatic flight function and performance of the intelligent assisted driving system during the entire simulated route flight process.

3. The method for testing the reliability of intelligent assisted driving based on a virtual camera according to claim 1, characterized in that: The full-motion simulation test platform is used to realize the primary flight display PFD and navigation display ND display interface and the cockpit airborne terminal data interface.

4. The method for testing the reliability of intelligent assisted driving based on a virtual camera according to claim 1, characterized in that: In step S2, the virtual camera simulation computer is installed with NetVDN APP, the visual system desktop program and the Obsstudio program; the virtual camera simulation computer is used to realize virtual camera simulation, extravehicular visual image acquisition and transmission, full-motion simulator and intelligent assisted driving computer data interaction and cross-linking test.

5. The method for testing the reliability of intelligent assisted driving based on a virtual camera according to claim 4, characterized in that: The NetVDN APP is used for data interaction and cross-linking testing between the full-motion simulator and the intelligent assisted driving software, specifically including: providing a test interface, the test interface can collect data from each simulation system of the full-motion simulator, provide relative position data of the virtual camera, inertial navigation, and GPS airborne equipment in the body coordinate system, and transmit it to the intelligent assisted driving high-performance platform via Ethernet at an agreed frequency; at the same time, the test interface receives control data of the intelligent assisted driving system, cross-links the intelligent assisted driving system with the flight simulation equipment, realizes data interaction, and completes the verification test; the NetVDN APP realizes the configuration of the virtual camera external parameters, and after the configuration is completed, it can be synchronized with the data of the full-motion flight simulator in real time, simulate the real camera external parameter parameters, and send the virtual camera internal and external parameters to the intelligent assisted driving system and to the intelligent assisted driving computer at an agreed frequency.

6. The method for testing the reliability of intelligent assisted driving based on a virtual camera according to claim 1, characterized in that: In step S5, the intelligent assisted driving system uses a quaternion q to control the direction of the aircraft, and gives an Euler rotation (X, Y, Z), that is, rotating X, Y, Z around the x-axis, y-axis and z-axis respectively, and the formula is as follows: ， The corresponding quaternion .

7. The method for testing the reliability of intelligent assisted driving based on a virtual camera according to claim 1, characterized in that: In step S5, the driving force of the full-motion simulation test platform during the test , sliding resistance , Slope resistance , wind resistance , according to the balance relationship of forces during the virtual aircraft test, the basic mathematical model of aircraft speed is obtained: ； ； In the formula, is the speed of the aircraft, is the aircraft weight, is the rolling resistance, is the slope, is the air resistance coefficient, A is the frontal area of ​​the aircraft, is the aircraft speed, is the steering angle, R is the turning radius of the aircraft, is the steering wheel angle, is the coefficient of turning radius and turning angle.

Images