A method, device and medium for processing flight simulator data

Abstract

The present invention provides a method, device and medium for processing flight simulator data, and relates to the technical field of flight simulators. The method comprises: acquiring a first operation action of a main driver user and a second operation action of a co-driver user collected by an action collection module; if the first operation action and the second operation action both meet preset recognition conditions, identifying the first operation action and the second operation action to obtain a first execution instruction corresponding to the first operation action and a second execution instruction corresponding to the second operation action; acquiring a first operation time corresponding to the first operation action and a second operation time corresponding to the second operation action; if the first operation time is different from the second operation time, storing the first execution instruction and the second execution instruction in a preset storage position in a data processing and recording module; the present invention can improve the accuracy of identifying execution instructions corresponding to user operation actions.

Description

Technical Field

[0001] The present invention relates to the technical field of flight simulators, and in particular to a method, device and medium for processing flight simulator data. Background Art

[0002] Flight simulators are the main equipment for flight training. The training data after flight simulator training will be recorded in the data recording system to assist flight instructors in commenting on the students' performance. However, the current data recording systems generally use video to record, which has huge requirements on the hardware memory. The data recording system is often unable to save new data due to full memory, or crashes or delays due to equipment overheating. Moreover, the video of the current data recording system is shot from the rear of the cockpit and cannot clearly cover the specific operations of the students, resulting in low accuracy in identifying the specific operations of the students. Summary of the invention

[0003] The present invention aims to provide a method, device and medium for processing flight simulator data to improve the accuracy of identifying the specific operations of trainees.

[0004] According to a first aspect of the present application, a method for processing flight simulator data is provided, the method being applied to a flight simulator data processing system, the processing system comprising: a data acquisition module and a data processing and recording module; wherein the data acquisition module and the data processing and recording module are communicatively connected; the data acquisition module comprises an action acquisition module and a parameter acquisition module, the data acquisition module is used to identify the operation actions of the main driver and the co-driver, and the parameter acquisition module is used to acquire each execution instruction;

[0005] The method comprises the following steps:

[0006] The first operation action of the main driver user and the second operation action of the co-driver user collected by the action collection module are obtained.

[0007] If both the first operation action and the second operation action meet the preset recognition condition, the first operation action and the second operation action are recognized to obtain a first execution instruction corresponding to the first operation action and a second execution instruction corresponding to the second operation action.

[0008] A first operation time corresponding to the first operation action and a second operation time corresponding to the second operation action are obtained.

[0009] If the first operation time is different from the second operation time, the first execution instruction and the second execution instruction are stored in a preset storage location in the data processing and recording module.

[0010] If the first operation time is the same as the second operation time and the first execution instruction is different from the second execution instruction, the first execution instruction and the second execution instruction are stored in a preset storage location in the data processing and recording module.

[0011] If the first operation time is the same as the second operation time and the first execution instruction is the same as the second execution instruction, then the historical execution instruction collected by the parameter collection module is obtained.

[0012] According to the first operation action, the second operation action and the historical execution instruction, the first execution instruction and the second execution instruction are corrected to obtain a first correction instruction corresponding to the first execution instruction and a second correction instruction corresponding to the second execution instruction.

[0013] The first correction instruction and the second correction instruction are stored in a preset storage location in the data processing and recording module.

[0014] Furthermore, the motion acquisition module includes: a main driver's camera, a co-driver's camera and an auxiliary camera; wherein the main driver's camera is used to identify the operating actions of the main driver user, the co-driver's camera is used to identify the operating actions of the co-driver user, and the auxiliary camera is used to cover the entire cockpit area.

[0015] The method of correcting the first execution instruction and the second execution instruction according to the first operation action, the second operation action and the historical execution instruction to obtain a first correction instruction corresponding to the first execution instruction and a second correction instruction corresponding to the second execution instruction includes the following steps:

[0016] Each historical execution instruction collected by the parameter collection module within a preset historical time period is obtained to obtain a historical execution instruction list; wherein the historical execution instruction list includes a plurality of historical execution instructions; the preset historical time period is determined according to the first operation time or the second operation time.

[0017] If the number of historical execution instructions in the historical execution instruction list is 1, and the only historical execution instruction in the historical execution instruction list is the same as the first execution instruction, then the instruction type of the only historical execution instruction in the historical execution instruction list is obtained.

[0018] If the instruction type of the only historical execution instruction in the historical execution instruction list is a manual type, then the image of the main driver user's operation action captured by the main driver's camera within the preset historical time period, the image of the co-driver user's operation action captured by the co-driver camera, and the image of the entire cockpit area captured by the auxiliary camera are obtained.

[0019] The image of the main driver's operating action and the image of the entire cockpit area are input into a preset action recognition model to obtain a first confidence level that the execution instruction corresponding to the operation of the main driver within a preset historical time period is a first execution instruction, and the image of the co-pilot user's operating action and the image of the entire cockpit area are input into the preset action recognition model to obtain a second confidence level that the execution instruction corresponding to the operation of the co-pilot user within a preset historical time period is a second execution instruction.

[0020] If the first confidence level is greater than the second confidence level, the first correction instruction is determined to be the first execution instruction, and the second correction instruction is determined to be empty.

[0021] If the first confidence level is less than the second confidence level, the first correction instruction is determined to be empty, and the second correction instruction is determined to be the second execution instruction.

[0022] If the first confidence level is equal to the second confidence level, a preset annotation prompt is performed on the first execution instruction and the second execution instruction; wherein the annotation prompt is used to perform manual judgment.

[0023] Further, the correcting the first execution instruction and the second execution instruction according to the first operation action, the second operation action and the historical execution instruction to obtain a first correction instruction corresponding to the first execution instruction and a second correction instruction corresponding to the second execution instruction also includes the following steps:

[0024] If the number of historical execution instructions in the historical execution instruction list is greater than 1, each manual type historical execution instruction in the historical execution instruction list is obtained in turn to obtain a manual type historical execution instruction list; wherein the manual type historical execution instruction list includes several manual type historical execution instructions.

[0025] Obtain the position of the operation button corresponding to each historical execution instruction in the manual type historical execution instruction list to obtain an operation button position list set corresponding to the manual type historical execution instruction list; wherein the operation button position list set includes an operation button position list corresponding to each manual type historical execution instruction, and each operation button position list includes a number of operation button positions.

[0026] A first operation position corresponding to the first operation action and a second operation position corresponding to the second operation action are obtained.

[0027] Traverse the operation button position list set, if the first operation position and any operation button position in the operation button position list set meet the preset position judgment condition, and the second operation position and each position in the operation button position list set do not meet the preset position judgment condition, then determine that the first correction instruction is the execution instruction corresponding to the button position in the operation button position list set that meets the preset judgment condition with the first operation position, and determine that the second correction instruction is empty.

[0028] If the first operation position and each position in the operation button position list set do not meet the preset position judgment condition, and the second operation position and any operation button position in the operation button position list set meet the preset position judgment condition, then the first correction instruction is determined to be empty, and the second correction instruction is determined to be the execution instruction corresponding to the button position in the operation button position list set that meets the preset judgment condition with the second operation position.

[0029] Further, the correcting the first execution instruction and the second execution instruction according to the first operation action, the second operation action and the historical execution instruction to obtain a first correction instruction corresponding to the first execution instruction and a second correction instruction corresponding to the second execution instruction also includes the following steps:

[0030] If the first operation position and each position in the operation button position list set do not meet the preset position judgment condition, and the second operation position and each position in the operation button position list set do not meet the preset position judgment condition, it is determined that the first operation action and the second operation action are recognized incorrectly.

[0031] The first correction instruction is determined to be empty, and the second correction instruction is determined to be empty.

[0032] If the first operation position and any operation button position in the operation button position list set meet the preset position judgment condition, and the second operation position and any operation button position in the operation button position list set meet the preset position judgment condition, then the first execution instruction and the second execution instruction are given a preset annotation prompt.

[0033] Furthermore, the first operation position and any operation button position in the operation button position list set meet a preset position judgment condition, comprising the following steps:

[0034] Get a first distance between the first operation position and any operation button position in the operation button position list set.

[0035] If the first distance is less than or equal to a preset position distance threshold, it is determined that the first operation position and any operation button position in the operation button position list set meet a preset position judgment condition.

[0036] Further, the correcting the first execution instruction and the second execution instruction according to the first operation action, the second operation action and the historical execution instruction to obtain a first correction instruction corresponding to the first execution instruction and a second correction instruction corresponding to the second execution instruction also includes the following steps:

[0037] If the instruction type of the only historically executed instruction in the historically executed instruction list is an automatic type, it is determined that the first operation action and the second operation action are recognized incorrectly.

[0038] The first correction instruction is determined to be empty, and the second correction instruction is determined to be empty.

[0039] Furthermore, the start time of the preset historical time period is the first operation time minus the preset duration, and the end time is the first operation time plus the preset duration.

[0040] Furthermore, the preset duration ranges from 1 to 2 seconds.

[0041] Furthermore, the auxiliary camera includes a front auxiliary camera and a rear auxiliary camera; wherein the front auxiliary camera is used to cover the cockpit area from the front of the cockpit area, and the rear camera is used to cover the cockpit area from the rear of the cockpit area.

[0042] Furthermore, the system also includes a data display module; wherein the data display module is used to reproduce the cockpit area.

[0043] According to a second aspect of the present invention, there is provided an electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned method for processing flight simulator data when executing the computer program.

[0044] According to a third aspect of the present invention, there is provided a computer-readable storage medium storing a computer program, wherein the computer program implements the above-mentioned method for processing flight simulator data when executed by a processor.

[0045] The present invention has at least the following beneficial effects:

[0046] The method for processing flight simulator data of the present invention obtains a first operation action of a main driver user and a second operation action of a co-driver user collected by an action collection module; if the first operation action and the second operation action both meet preset identification conditions, the first operation action and the second operation action are identified to obtain a first execution instruction corresponding to the first operation action and a second execution instruction corresponding to the second operation action; a first operation time corresponding to the first operation action and a second operation time corresponding to the second operation action are obtained; if the first operation time is different from the second operation time, the first execution instruction and the second execution instruction are stored in a preset storage position in a data processing and recording module; thereby, according to the operation action of the main driver user and the operation action of the co-driver user, the execution instruction corresponding to the operation action of the main driver user and the execution instruction corresponding to the operation action of the co-driver user are respectively determined, and the execution instruction is stored. Since the execution instruction is in the form of code and occupies less storage space, the problem that the data recording system is often unable to save new data due to full memory can be avoided.

[0047] Furthermore, when it is identified that the main driver user and the front passenger user operate the button corresponding to the same execution instruction at the same time, the execution instructions corresponding to the operating actions of the main driver user and the front passenger user are corrected according to the historical execution instructions to ensure the accuracy of the execution instructions corresponding to the identified user operating actions. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0049] Figure 1 A flowchart of a method for processing flight simulator data provided in Embodiment 1 of the present invention;

[0050] Figure 2 A flowchart of the first instruction correction provided in the first embodiment of the present invention;

[0051] Figure 3 A flowchart of the second instruction correction provided in the first embodiment of the present invention;

[0052] Figure 4 A flowchart of the third instruction correction provided in the first embodiment of the present invention;

[0053] Figure 5 A flowchart of position determination provided in Embodiment 1 of the present invention;

[0054] Figure 6 This is a flowchart of the fourth instruction correction provided in the first embodiment of the present invention. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0056] It should be noted that, based on the present disclosure, those skilled in the art should understand that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement a device and / or practice a method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this device and / or practice this method.

[0057] Embodiment 1:

[0058] The following will refer to Figure 1 The flowchart of the method for processing flight simulator data shown in the figure introduces a method for processing flight simulator data.

[0059] The flight simulator data processing method is applied to a flight simulator data processing system, the processing system comprising: a data acquisition module and a data processing and recording module; wherein the data acquisition module and the data processing and recording module are communicatively connected; the data acquisition module comprises an action acquisition module and a parameter acquisition module, the data acquisition module is used to identify the operating actions of a main driver user and a co-driver user, and the parameter acquisition module is used to acquire each execution instruction.

[0060] In this embodiment, the flight simulator data processing system includes three parts, among which the first part is a data acquisition module, including an action acquisition module and a parameter acquisition module. The action acquisition module mainly uses multiple cameras to shoot all the user's operating actions in the cockpit; the parameter acquisition module records various parameter information, which may include the content of the display and the operating device, such as: the control stick, the steering wheel, the side stick, the offset of the pedals, the position of the landing gear, the instructor's desk setting parameters, etc. The collected data can be customized; the execution instructions collected by the parameter acquisition module include instructions generated by the user's operation button, that is, manual type execution instructions, and also include automatically generated execution instructions, that is, automatic type execution instructions.

[0061] The second part is the data processing and recording module. The operating actions captured by shooting need to be recognized. After recognition, these actions are converted into digital signals with low memory requirements; the parameter information on the cockpit also needs to be converted into digital signals.

[0062] The third part is the data display module, including the virtual cockpit and digital virtual human area, the third-view area, the display area, the control device area, and the custom area. The content of each area can be customized, and you can also choose the area you want to observe and zoom in and out. Use virtual technology to create a virtual cockpit and digital virtual human for the virtual cockpit and digital virtual human area, a virtual aircraft and external environment for the third-view area, a virtual display for the display area, a virtual control device for the control device area, and customized display content for the custom area. Then use the recorded digital data signals to drive each virtual module to change the display and reproduce the cockpit scene.

[0063] The method comprises the following steps:

[0064] S100, obtaining a first operating action of a main driver user and a second operating action of a co-driver user collected by an action collection module.

[0065] In this embodiment, when the main driver and the co-driver operate buttons in the cockpit, the action collection module can collect the first operation action of the main driver and the second operation action of the co-driver in real time.

[0066] Furthermore, the motion acquisition module includes: a main driver's camera, a co-driver's camera and an auxiliary camera; wherein the main driver's camera is used to identify the operating actions of the main driver user, the co-driver's camera is used to identify the operating actions of the co-driver user, and the auxiliary camera is used to cover the entire cockpit area.

[0067] Furthermore, the auxiliary camera includes a front auxiliary camera and a rear auxiliary camera; wherein the front auxiliary camera is used to cover the cockpit area from the front of the cockpit area, and the rear camera is used to cover the cockpit area from the rear of the cockpit area.

[0068] In this embodiment, four cameras are preset, namely, a main driver's camera, a co-driver's camera, a front auxiliary camera and a rear auxiliary camera; the front auxiliary camera is installed below the eye position meter, and the rear auxiliary camera is installed at the rear of the overhead panel. The front auxiliary camera and the rear auxiliary camera can simultaneously capture the operating actions of the main driver and the co-driver, and can cover the entire cockpit operating area one by one, and are used as auxiliary cameras; the main driver's camera is installed on the frame between the left main windshield and the left windshield, as the main camera of the main driver's seat, for tracking the operating actions of the main driver; the co-driver's camera is installed on the frame between the right main windshield and the right windshield, as the main camera of the co-driver's seat, for tracking the operating actions of the co-driver.

[0069] In this embodiment, the main driver's camera can be used in conjunction with the auxiliary camera, and the co-driver's camera can also be used in conjunction with the auxiliary camera. For example: the main driver user presses the button on the right side of the light hood, and the co-driver user also operates the control panel on the right. The main driver's camera tracks and confirms that the operation is performed by the main driver user, but may not capture which button is pressed due to hand occlusion. At the same time, the co-driver's camera also tracks the co-driver user's operation. Because the button is too small, it is impossible to determine which button it is. At this time, the auxiliary camera needs to be added, but the front auxiliary camera may not be able to capture it due to blind spots. The rear auxiliary camera needs to be used to clarify the operating areas of the two drivers. The auxiliary camera and the main driver's camera jointly confirm the buttons operated by the main driver user and the co-driver user. At the same time, by comparing the background parameter changes, the specific operation button is determined to prevent misjudgment.

[0070] S200: If the first operation action and the second operation action both meet the preset recognition condition, the first operation action and the second operation action are recognized to obtain a first execution instruction corresponding to the first operation action and a second execution instruction corresponding to the second operation action.

[0071] If both A and B meet the preset recognition conditions, then A and B are recognized to obtain the execution instruction Z corresponding to A. A The execution instruction Z corresponding to B B .

[0072] In this embodiment, the first operation action is recorded as A, the second operation action is recorded as B; the first execution instruction is recorded as Z A , and the second execution instruction is recorded as Z B .

[0073] A and B both meet the preset recognition conditions that the clarity of the image corresponding to A and the image corresponding to B is greater than the preset clarity threshold; the trained neural network model can be used to recognize the image corresponding to A and the image corresponding to B to obtain the execution instructions corresponding to the operation buttons corresponding to A and B; for example: A is the operation joystick, Z A It is the execution instruction corresponding to the operation joystick.

[0074] Furthermore, if A or B does not meet the preset recognition conditions, the image that does not meet the preset recognition conditions can be processed to improve the clarity of the image so that the processed image can be recognized.

[0075] S300: Acquire a first operation time corresponding to a first operation action and a second operation time corresponding to a second operation action.

[0076] In this embodiment, the first operation time corresponding to the first operation action is recorded as T A , the second operation time corresponding to the second operation action is recorded as T B .

[0077] In this embodiment, it should be noted that the operation action of the main driver and the operation action of the co-driver are independently identified. Therefore, the operation time T corresponding to A can be obtained according to the acquisition time of the image corresponding to A and the acquisition time of the image corresponding to B. A The operation time T corresponding to B B .

[0078] S400: If the first operation time is different from the second operation time, the first execution instruction and the second execution instruction are stored in a preset storage location in the data processing and recording module.

[0079] In this embodiment, if T A ≠T B , indicating that the time when the main driver user operated the button is not the same as the time when the co-driver user operated the button, there is a sequential relationship between the operations, there is no operation conflict, and the execution instruction Z corresponding to A is recognized A And the execution instruction Z corresponding to B B, then, directly change Z A and Z B It can be stored in the preset storage location in the data processing and recording module; it should be noted that when Z A and Z B When storing, you need to associate storage Z A The corresponding operation time T A and Z B The corresponding operation time T B Therefore, when the user's operation action is reproduced, it can be reproduced according to the timing of the operation action.

[0080] S500: If the first operation time is the same as the second operation time and the first execution instruction is different from the second execution instruction, the first execution instruction and the second execution instruction are stored in a preset storage location in the data processing and recording module.

[0081] In this embodiment, if T A =T B And Z A ≠Z B , indicating that the main driver and the co-driver operated the button at the same time. However, the execution command corresponding to the button operated by the main driver is different from the execution command corresponding to the button operated by the co-driver. There is no conflict between the execution commands between the two. It can be determined that Z A and Z B is correct, so directly change Z A and Z B Stored to the preset storage location in the data processing and recording module.

[0082] S600: If the first operation time is the same as the second operation time and the first execution instruction is the same as the second execution instruction, then obtain the historical execution instruction collected by the parameter collection module.

[0083] In this embodiment, if T A =T B And Z A =Z B , indicating that it is recognized that the main driver and the co-driver operate the same function button at the same time. At this time, there may be a situation where the main driver's operation action is recognized incorrectly or the co-driver's operation action is recognized incorrectly, or there may be a situation where the main driver's operation action is recognized incorrectly and the co-driver's operation action is recognized incorrectly. At this time, the historical execution instructions collected by the parameter collection module can be obtained to further judge Z A and Z B Is it correct.

[0084] S700, correcting the first execution instruction and the second execution instruction according to the first operation action, the second operation action and the historical execution instruction to obtain a first correction instruction corresponding to the first execution instruction and a second correction instruction corresponding to the second execution instruction.

[0085] In this embodiment, the first correction instruction is recorded as G A , and the second correction instruction is recorded as G B .

[0086] Further, such as Figure 2 As shown, step S700 may include the following steps:

[0087] S710, obtaining each historical execution instruction collected by the parameter collection module within a preset historical time period to obtain a historical execution instruction list; wherein the historical execution instruction list includes a plurality of historical execution instructions; the preset historical time period is determined according to the first operation time or the second operation time.

[0088] Specifically, each historical execution instruction collected by the parameter collection module in the preset historical time period TL is obtained to obtain a historical execution instruction list C=(C1, C2, ..., C i , …, C n ), i=1, 2, …, n; where C i is the i-th historical execution instruction collected by the TL internal parameter collection module, and n is the number of historical execution instructions collected by the TL internal parameter collection module; C r The generation time is earlier than C r+1 The generation time, r = 1, 2, ..., n-1; TL is based on T A or T B Sure.

[0089] In this embodiment, the parameter acquisition module can collect each execution instruction in real time. The execution instructions include manual type execution instructions and automatic type execution instructions; manual type execution instructions can be understood as execution instructions generated by the main driver user and the co-driver user operating buttons; other execution instructions, for example: execution instructions generated during automatic cruising are automatic type execution instructions.

[0090] Furthermore, TL=[TL1, TL2]; where TL1 is the start time of TL and TL2 is the end time of TL; TL1=T A -ΔT, TL2=T A +ΔT; ΔT is the preset duration.

[0091] In this embodiment, it can be understood that T A and T BAll of them are obtained through image recognition. There are delays and inaccurate time in the image recognition process. Based on this, TL is set to make the redundancy of time comparison higher, thereby improving Z A and Z B Accuracy of judgment; specifically, the value range of ΔT is 1 to 2 seconds, for example, the value of ΔT is 1 second.

[0092] S720: If the number of historical execution instructions in the historical execution instruction list is 1, and the only historical execution instruction in the historical execution instruction list is the same as the first execution instruction, obtain the instruction type of the only historical execution instruction in the historical execution instruction list.

[0093] In this embodiment, if n=1 and Z A =Z B =C1, then get the instruction type of C1; if n=1 and Z A =Z B =C1, indicating that it is recognized that the main driver and the co-driver operate the button with the same function at the same time; however, under normal circumstances, this situation does not exist, so it may be caused by an image recognition error; at this time, the instruction type of C1 can be obtained.

[0094] Further, such as Figure 6 As shown, after step S720 and before step S730, step S700 may further include the following steps:

[0095] S721: If the instruction type of the only historically executed instruction in the historically executed instruction list is an automatic type, it is determined that the first operation action and the second operation action are recognized incorrectly.

[0096] That is, if the instruction type of C1 is the automatic type, it is determined that A and B recognize errors.

[0097] S722, determine that the first correction instruction is empty, and determine that the second correction instruction is empty.

[0098] In this embodiment, if there is only one execution instruction in C, and the execution instruction is an automatic type execution instruction, it means that the main driver user and the co-driver user have not operated any button. Therefore, it is determined that G A =NULL and G B =NULL.

[0099] S730, if the instruction type of the only historical execution instruction in the historical execution instruction list is a manual type, then obtain the image of the main driver user's operation action captured by the main driver's camera within a preset historical time period, the image of the co-driver user's operation action captured by the co-driver camera, and the image of the entire cockpit area captured by the auxiliary camera.

[0100] Specifically, if the instruction type of C1 is manual, the image QA of the main driver user's operation action captured by the main driver camera in TL, the image QB of the co-driver user's operation action captured by the co-driver camera, and the image QU of the entire cockpit area captured by the auxiliary camera are obtained.

[0101] In this embodiment, if the instruction type of C1 is manual, it is not possible to directly determine Z A and Z B To determine which one has an error, we can obtain the image QA of the main driver's user operation action collected by the main driver's camera, the image QB of the co-driver's user operation action collected by the co-driver's camera, and the image QU of the entire cockpit area collected by the auxiliary camera in the TL to further determine Z A and Z B Which one has the error?

[0102] S740, input the image of the main driver's operating action and the image of the entire cockpit area into a preset action recognition model to obtain a first confidence level that the execution instruction corresponding to the operation of the main driver within a preset historical time period is a first execution instruction, and input the image of the co-pilot user's operating action and the image of the entire cockpit area into a preset action recognition model to obtain a second confidence level that the execution instruction corresponding to the operation of the co-pilot user within a preset historical time period is a second execution instruction.

[0103] Specifically, QA and QU are input into the preset action recognition model to obtain the execution instruction corresponding to the operation of the main driver in TL as Z A The confidence level η A , and input QB and QU into the preset action recognition model to obtain the execution instruction corresponding to the operation of the co-pilot user in TL as Z B The confidence level η B .

[0104] In this embodiment, the preset action recognition model can be a trained neural network model, which outputs the confidence of the corresponding execution instruction based on the input image; QU can be used as an auxiliary image to assist in judging QA and QB respectively, so as to increase the richness of the images of the main driver's operating actions and the co-driver's operating actions, and improve the accuracy of the corresponding confidence output.

[0105] S750: If the first confidence level is greater than the second confidence level, determine that the first correction instruction is the first execution instruction, and determine that the second correction instruction is empty.

[0106] Specifically, if η A >η B , then determine G A =Z A and GB =NULL.

[0107] In this embodiment, since in TL, the parameter acquisition module only acquires one manual type execution instruction, the execution instruction corresponds to A or B; if n A >η B , indicating that the probability corresponding to A is greater, therefore, G is determined A =Z A and G B =NULL; NULL means a null command, that is, the passenger seat user did not press the button.

[0108] S760: If the first confidence level is less than the second confidence level, determine that the first correction instruction is empty, and determine that the second correction instruction is the second execution instruction.

[0109] Specifically, if η A <η B , then determine G A =NULL and G B =Z B If η A <η B , indicating that the probability corresponding to B is greater, therefore, G is determined A =NULL and G B =Z B ; That is, the main driver did not press the button.

[0110] S770: If the first confidence level is equal to the second confidence level, a preset annotation prompt is provided for the first execution instruction and the second execution instruction; wherein the annotation prompt is used to perform manual judgment.

[0111] Specifically, if η A =η B , then for Z A and Z B Perform preset annotation prompts; if η A =η B , indicating that the possibility corresponding to A is the same as the possibility corresponding to B. In this case, for Z A and Z B Provide preset annotation prompts for subsequent manual judgment.

[0112] Further, such as Figure 3 As shown, after step S710 and before step S720, step S700 may further include the following steps:

[0113] S711, if the number of historical execution instructions in the historical execution instruction list is greater than 1, then obtain each manual type historical execution instruction in the historical execution instruction list in turn to obtain a manual type historical execution instruction list; wherein the manual type historical execution instruction list includes several manual type historical execution instructions.

[0114] Specifically, if n>1, then obtain each manual type historical execution instruction in C in turn to obtain a manual type historical execution instruction list D=(D1, D2, ..., D j , …, D m ), j = 1, 2, …, m; where D j is the jth manual type historical execution instruction obtained from C, and m is the number of manual type historical execution instructions obtained from C.

[0115] In this embodiment, when n>1, there are multiple execution instructions in C, and each manual type historical execution instruction in C can be obtained according to the type label corresponding to each execution instruction.

[0116] S712, obtain the position of the operation button corresponding to each historical execution instruction in the manual type historical execution instruction list to obtain an operation button position list set corresponding to the manual type historical execution instruction list; wherein the operation button position list set includes an operation button position list corresponding to each manual type historical execution instruction, and each operation button position list includes a number of operation button positions.

[0117] Specifically, the position of the operation button corresponding to each historical execution instruction in D is obtained to obtain the operation button position list set E corresponding to D = (E1, E2, ..., E j ,…,E m ); where E j D j The location list of the corresponding operation buttons; E j =(E j，1 , E j，2 ,…,E j，x ,…,E j，f（j） ), j=1,2,…,f(j);E j，x D j The corresponding position of the x-th operation button.

[0118] In this embodiment, for each historical execution instruction in D, there is at least one corresponding button, and the position of each button corresponding to each historical execution instruction in D can be obtained to obtain E; a three-dimensional coordinate system corresponding to the cockpit area can be generated, and the position of the button can be represented by the corresponding three-dimensional coordinates.

[0119] S713: Acquire a first operation position corresponding to the first operation action and a second operation position corresponding to the second operation action.

[0120] Specifically, the first operation position is recorded as YA, and the second operation position is recorded as YB. In this embodiment, by identifying the image corresponding to A and the image corresponding to B, the operation position YA corresponding to A and the operation position YB corresponding to B can be obtained. It should be noted that those skilled in the art can use existing image processing methods according to actual needs to obtain the operation position YA corresponding to A and the operation position YB corresponding to B, which will not be elaborated here.

[0121] S714, traverse the operation button position list set, if the first operation position and any operation button position in the operation button position list set meet the preset position judgment condition, and the second operation position and each operation button position in the operation button position list set do not meet the preset position judgment condition, then determine that the first correction instruction is the historical execution instruction corresponding to the operation button position in the operation button position list set that meets the preset judgment condition with the first operation position, and determine that the second correction instruction is empty.

[0122] Specifically, traverse E, if YA and E j，x If the preset position judgment condition is met, and each position in YB and E does not meet the preset position judgment condition, then determine G A =D j and G B =NULL.

[0123] Further, such as Figure 5 As shown, YA and E j，x Meeting the preset position determination condition may include the following steps:

[0124] S71, obtaining a first distance between a first operation position and any operation button position in the operation button position list set.

[0125] Specifically, get YA and E j，x Distance L A .

[0126] S72: If the first distance is less than or equal to a preset position distance threshold, determine whether the first operation position and any operation button position in the operation button position list satisfy a preset position judgment condition.

[0127] Specifically, if L A ≤L', then determine YA and E j，x The preset position judgment condition is met; wherein L' is the preset position distance threshold.

[0128] In this embodiment, the above method can also be used to judge between YB and each position in E, which will not be elaborated here.

[0129] If YA and E j，x The preset position judgment condition is met, and each position in YB and E does not meet the preset position judgment condition, which means that the position of the button operated by the main driver user is in E, while the button operated by the co-driver user is not in E. Therefore, it can be determined that G A =D j and G B =NULL, thereby achieving the purpose of correcting the execution instruction.

[0130] S715, if the first operation position and each operation button position in the operation button position list set do not meet the preset position judgment condition, and the second operation position and any operation button position in the operation button position list set meet the preset position judgment condition, then determine that the first correction instruction is empty, and determine that the second correction instruction is the historical execution instruction corresponding to the operation button position in the operation button position list set that meets the preset judgment condition with the second operation position.

[0131] Specifically, if each position of YA and E does not meet the preset position judgment condition, and YB and E j，x If the preset position judgment conditions are met, G is determined A =NULL and G B =D j .

[0132] If each position of YA and E does not meet the preset position judgment condition, and YB and E j，x The preset position judgment condition is met, which means that the button operated by the co-pilot user is in E, while the button operated by the main driver user is not in E. Therefore, it can be determined that G A =NULL and G B =D j .

[0133] Further, such as Figure 4 As shown, after step S715, step S700 further includes the following steps:

[0134] S716, if the first operation position and each operation button position in the operation button position list set do not meet the preset position judgment condition, and the second operation position and each operation button position in the operation button position list set do not meet the preset position judgment condition, it is determined that the first operation action and the second operation action are recognized incorrectly.

[0135] Specifically, if each operation button position in YA and E does not meet the preset position judgment condition, and each operation button position in YB and E does not meet the preset position judgment condition, it is determined that A and B are recognized incorrectly.

[0136] S717, determining that the first correction instruction is empty, and determining that the second correction instruction is empty.

[0137] If the position of each operation button in YA and E does not meet the preset position judgment condition, and the position of each operation button in YB and E does not meet the preset position judgment condition, it means that the image corresponding to A and the image corresponding to B are both recognized incorrectly, and neither the main driver user nor the co-driver user has operated the button. Therefore, it is determined that G A =NULL and G B =NULL.

[0138] S718, if the first operation position and any operation button position in the operation button position list set meet the preset position judgment condition, and the second operation position and any operation button position in the operation button position list set meet the preset position judgment condition, then the first execution instruction and the second execution instruction are given a preset annotation prompt.

[0139] Specifically, if YA and E j，x The preset position judgment conditions are met, and YB and E j，x If the preset position judgment conditions are met, then Z A and Z B Provide preset annotation prompts.

[0140] In this embodiment, if YA and E j，x The preset position judgment conditions are met, and YB and E j，x If the preset position judgment condition is met, it means that the operation of the main driver and the co-driver conflicts, then Z A and Z B Provide preset annotation prompts for subsequent manual judgment.

[0141] S800: Store the first correction instruction and the second correction instruction in a preset storage location in the data processing and recording module.

[0142] In this embodiment, G is obtained by the above method. A and G B Then, G A and G B Stored to the preset storage location in the data processing and recording module to facilitate subsequent scene reproduction.

[0143] Furthermore, the above system also includes a data display module; wherein the data display module is used to reproduce the cockpit area.

[0144] The flight simulator data processing method of the present embodiment obtains the first operation action of the main driver user and the second operation action of the co-pilot user collected by the action collection module; if the first operation action and the second operation action both meet the preset recognition conditions, the first operation action and the second operation action are recognized to obtain a first execution instruction corresponding to the first operation action and a second execution instruction corresponding to the second operation action; a first operation time corresponding to the first operation action and a second operation time corresponding to the second operation action are obtained; if the first operation time is different from the second operation time, the first execution instruction and the second execution instruction are stored in a preset storage position in the data processing and recording module; thereby, according to the operation action of the main driver user and the operation action of the co-pilot user, the execution instruction corresponding to the operation action of the main driver user and the execution instruction corresponding to the operation action of the co-pilot user are respectively determined, and the execution instructions are stored. Since the execution instructions are in the form of code and occupy less storage space, the problem that the data recording system is often unable to save new data due to full memory can be avoided.

[0145] Furthermore, when it is identified that the main driver user and the front passenger user operate the button corresponding to the same execution instruction at the same time, the execution instructions corresponding to the operating actions of the main driver user and the front passenger user are corrected according to the historical execution instructions to ensure the accuracy of the execution instructions corresponding to the identified user operating actions.

[0146] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0147] Embodiment 2:

[0148] An embodiment of the present invention also provides a non-transitory computer-readable storage medium, which can be set in an electronic device to store at least one instruction or at least one program related to implementing a method in a method embodiment. The at least one instruction or the at least one program is loaded and executed by the processor to implement the method provided in the above embodiment.

[0149] The program product may adopt any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0150] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0151] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0152] Program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0153] Embodiment three:

[0154] An embodiment of the present invention further provides an electronic device, comprising a processor and the aforementioned non-transitory computer-readable storage medium.

[0155] The electronic device is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0156] The electronic device is presented in the form of a general-purpose computing device. The components of the electronic device may include, but are not limited to: the at least one processor mentioned above, the at least one memory mentioned above, and a bus connecting different system components (including the memory and the processor).

[0157] The memory stores program codes, which can be executed by the processor, so that the processor executes the steps in various embodiments described in this specification.

[0158] The memory may include readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory, and may further include read only memory (ROM).

[0159] The memory may also include a program / utility having a set (at least one) of program modules, such program modules including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include an implementation of a network environment.

[0160] The bus may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures.

[0161] The electronic device may also communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may communicate with one or more devices that enable a user to interact with the electronic device, and / or may communicate with any device (e.g., routers, modems, etc.) that enables the electronic device to communicate with one or more other computing devices. Such communication may be performed through an input / output (I / O) interface. Furthermore, the electronic device may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) through a network adapter. The network adapter communicates with other modules of the electronic device through a bus. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0162] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0163] An embodiment of the present invention further provides a computer program product, which includes program code. When the program product is run on an electronic device, the program code is used to enable the electronic device to execute the steps of the method according to various exemplary embodiments of the present invention described above in this specification.

[0164] Although some specific embodiments of the present invention have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are only for illustration, not for limiting the scope of the present invention. It should also be understood by those skilled in the art that various modifications may be made to the embodiments without departing from the scope and spirit of the present invention.

Claims

1. A method for processing flight simulator data, characterized in that: The method is applied to a processing system for flight simulator data, the processing system comprising: a data acquisition module and a data processing and recording module; wherein the data acquisition module and the data processing and recording module are in communication connection; the data acquisition module comprises an action acquisition module and a parameter acquisition module, the data acquisition module is used to identify the operation actions of the main driver and the co-driver, and the parameter acquisition module is used to collect each execution instruction; the action acquisition module comprises: a main driver camera, a co-driver camera and an auxiliary camera; wherein the main driver camera is used to identify the operation actions of the main driver, the co-driver camera is used to identify the operation actions of the co-driver, and the auxiliary camera is used to cover the entire cockpit area; The method comprises the following steps: Acquire a first operation action of the main driver user and a second operation action of the co-driver user collected by the action collection module; If both the first operation action and the second operation action meet the preset recognition condition, the first operation action and the second operation action are recognized to obtain a first execution instruction corresponding to the first operation action and a second execution instruction corresponding to the second operation action; Acquire a first operation time corresponding to the first operation action and a second operation time corresponding to the second operation action; If the first operation time is different from the second operation time, storing the first execution instruction and the second execution instruction in a preset storage location in the data processing and recording module; If the first operation time is the same as the second operation time and the first execution instruction is different from the second execution instruction, storing the first execution instruction and the second execution instruction in a preset storage location in the data processing and recording module; If the first operation time is the same as the second operation time and the first execution instruction is the same as the second execution instruction, then acquiring each historical execution instruction collected by the parameter collection module within the preset historical time period to obtain a historical execution instruction list; wherein the historical execution instruction list includes a plurality of historical execution instructions; the preset historical time period is determined according to the first operation time or the second operation time; Correct the first execution instruction and the second execution instruction according to the first operation action, the second operation action, and the number of historical execution instructions in the historical execution instruction list and the instruction type of the historical execution instructions to obtain a first correction instruction corresponding to the first execution instruction and a second correction instruction corresponding to the second execution instruction; The first correction instruction and the second correction instruction are stored in a preset storage location in the data processing and recording module.

2. The method for processing flight simulator data according to claim 1, characterized in that: The first execution instruction and the second execution instruction are corrected according to the first operation action, the second operation action, and the number of historical execution instructions in the historical execution instruction list and the instruction type of the historical execution instructions to obtain a first correction instruction corresponding to the first execution instruction and a second correction instruction corresponding to the second execution instruction, including the following steps: If the number of historical execution instructions in the historical execution instruction list is 1, and the only historical execution instruction in the historical execution instruction list is the same as the first execution instruction, then obtaining the instruction type of the only historical execution instruction in the historical execution instruction list; If the instruction type of the only historical execution instruction in the historical execution instruction list is a manual type, then obtaining an image of the main driver user's operation action collected by the main driver's camera, an image of the co-driver user's operation action collected by the co-driver's camera, and an image of the entire cockpit area collected by the auxiliary camera within a preset historical time period; Inputting the image of the operating action of the main driver user and the image of the entire cockpit area into a preset action recognition model to obtain a first confidence that the execution instruction corresponding to the operation of the main driver user within a preset historical time period is a first execution instruction, and inputting the image of the operating action of the co-pilot user and the image of the entire cockpit area into the preset action recognition model to obtain a second confidence that the execution instruction corresponding to the operation of the co-pilot user within the preset historical time period is a second execution instruction; If the first confidence level is greater than the second confidence level, determining that the first correction instruction is the first execution instruction, and determining that the second correction instruction is empty; If the first confidence level is less than the second confidence level, determining that the first correction instruction is empty, and determining that the second correction instruction is the second execution instruction; If the first confidence level is equal to the second confidence level, a preset annotation prompt is performed on the first execution instruction and the second execution instruction; wherein the annotation prompt is used to perform manual judgment.

3. The method for processing flight simulator data according to claim 2, characterized in that: The first execution instruction and the second execution instruction are corrected according to the first operation action, the second operation action, and the number of historical execution instructions in the historical execution instruction list and the instruction type of the historical execution instructions to obtain a first correction instruction corresponding to the first execution instruction and a second correction instruction corresponding to the second execution instruction, and further includes the following steps: If the number of historical execution instructions in the historical execution instruction list is greater than 1, then obtaining each manual type historical execution instruction in the historical execution instruction list in turn to obtain a manual type historical execution instruction list; wherein the manual type historical execution instruction list includes a plurality of manual type historical execution instructions; Obtaining the position of the operation button corresponding to each historical execution instruction in the manual type historical execution instruction list, so as to obtain an operation button position list set corresponding to the manual type historical execution instruction list; wherein the operation button position list set includes an operation button position list corresponding to each manual type historical execution instruction, and each operation button position list includes a plurality of operation button positions; Acquire a first operation position corresponding to the first operation action and a second operation position corresponding to the second operation action; Traversing the operation button position list set, if the first operation position and any operation button position in the operation button position list set meet the preset position judgment condition, and the second operation position and each operation button position in the operation button position list set do not meet the preset position judgment condition, then determining that the first correction instruction is a historical execution instruction corresponding to the operation button position in the operation button position list set that meets the preset judgment condition with the first operation position, and determining that the second correction instruction is empty; If the first operation position and each operation button position in the operation button position list set do not meet the preset position judgment condition, and the second operation position and any operation button position in the operation button position list set meet the preset position judgment condition, then the first correction instruction is determined to be empty, and the second correction instruction is determined to be the historical execution instruction corresponding to the operation button position in the operation button position list set that meets the preset judgment condition with the second operation position.

4. The method for processing flight simulator data according to claim 3, characterized in that: The first execution instruction and the second execution instruction are corrected according to the first operation action, the second operation action, and the number of historical execution instructions in the historical execution instruction list and the instruction type of the historical execution instructions to obtain a first correction instruction corresponding to the first execution instruction and a second correction instruction corresponding to the second execution instruction, and further includes the following steps: If the first operation position and each operation button position in the operation button position list set do not satisfy the preset position judgment condition, and the second operation position and each operation button position in the operation button position list set do not satisfy the preset position judgment condition, it is determined that the first operation action and the second operation action are recognized incorrectly; Determining that the first correction instruction is empty, and determining that the second correction instruction is empty; If the first operation position and any operation button position in the operation button position list set meet the preset position judgment condition, and the second operation position and any operation button position in the operation button position list set meet the preset position judgment condition, then the first execution instruction and the second execution instruction are given a preset annotation prompt.

5. The method for processing flight simulator data according to claim 3, characterized in that: The first operation position and any operation button position in the operation button position list set meet the preset position judgment condition, including the following steps: Obtaining a first distance between the first operation position and any operation button position in the operation button position list set; If the first distance is less than or equal to a preset position distance threshold, it is determined that the first operation position and any operation button position in the operation button position list set meet a preset position judgment condition.

6. The method for processing flight simulator data according to claim 2, characterized in that: The first execution instruction and the second execution instruction are corrected according to the first operation action, the second operation action, and the number of historical execution instructions in the historical execution instruction list and the instruction type of the historical execution instructions to obtain a first correction instruction corresponding to the first execution instruction and a second correction instruction corresponding to the second execution instruction, and further includes the following steps: If the instruction type of the only historically executed instruction in the historically executed instruction list is an automatic type, determining that the first operation action and the second operation action are recognized incorrectly; The first correction instruction is determined to be empty, and the second correction instruction is determined to be empty.

7. The method for processing flight simulator data according to claim 2, characterized in that: The start time of the preset historical time period is the first operation time minus the preset duration, and the end time is the first operation time plus the preset duration.

8. The method for processing flight simulator data according to claim 7, characterized in that: The preset duration ranges from 1 to 2 seconds.

9. The method for processing flight simulator data according to claim 1, characterized in that: The auxiliary camera includes a front auxiliary camera and a rear auxiliary camera; wherein the front auxiliary camera is used to cover the cockpit area from the front of the cockpit area, and the rear camera is used to cover the cockpit area from the rear of the cockpit area.

10. The method for processing flight simulator data according to claim 1, characterized in that: The system further comprises a data display module; wherein the data display module is used to reproduce the cockpit area.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method for processing flight simulator data as described in any one of claims 1 to 10 is implemented.

12. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for processing flight simulator data as claimed in any one of claims 1 to 10 is implemented.

Images