Information control system, information control method and program

Abstract

To allow an operator to undergo training, evaluation, and certification of operating skills more effectively and appropriately.SOLUTION: An information control system available for supporting the training, evaluation, or certification of operating skills of an operator who operates an object to be operated, including a flight vehicle or another mobile body, comprises: an information import part that acquires first video data recorded on the state of the object to be operated that is operated by the operator and also acquires second video data recorded by an operating device on the state of the operation inputted by the operator; an integrated data generation part that generates or records integrated video data obtained by integrating the first and second video data; and a display output part that outputs the integrated video data during or after operation of the object to be operated.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information control system, an information control method, and a program. [Background technology]

[0002] Patent Document 1 discloses a piloting training system that includes an aircraft, a controller for controlling the aircraft, and a training system, wherein the training system is capable of abnormal flight control that simulates abnormal conditions such as external disturbances and flies the aircraft under those abnormal conditions, and the aircraft flying under the control can be piloted by operating the controller. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-191225 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, as described in Patent Document 1, consideration has been given to the practical application of systems that support training in piloting skills for piloted objects such as unmanned aerial vehicles, and systems that quantitatively evaluate piloting skills.

[0005] In order to effectively conduct pilot training for aircraft and the like, it is desirable to immediately review one's own flight performance after pilot training and consider areas for improvement in piloting. However, the pilot training system described in Patent Document 1 only discloses that information regarding the status of the aircraft being piloted, such as attitude information, remaining battery power, and communication status, is displayed on the flight information display unit, and no support function was considered to allow the pilot to immediately review one's own flight performance after pilot training.

[0006] Furthermore, in order to effectively improve piloting skills, it is important to review and improve the pilot's finger and eye movements when operating the control device; however, there was no piloting training system that allowed the pilot to review the pilot's finger and eye movements.

[0007] Furthermore, although the flight training system described in Patent Document 1 discloses a technology for evaluating a pilot's flight skills based on flight performance information such as the position and speed of the aircraft, the pilot's finger and eye movements when operating the control device, as well as other pilot-related conditions, are not taken into consideration in the flight skill evaluation, leaving room for improvement in the accuracy of the flight skill evaluation.

[0008] Furthermore, in order to more effectively improve pilots' flying skills, it was necessary to provide instructors and pilots with information that would be useful for improving skills, such as shortcomings in the piloting operations performed, their causes, areas for improvement, and future training methods.

[0009] Therefore, the present invention has been made in consideration of at least one of the above problems, and one of its objectives is to provide a system or method, etc., that can more effectively or appropriately train, evaluate, and certify pilots' piloting skills. [Means for solving the problem]

[0010] According to the present invention, there is provided an information control system that can be used to support training, evaluation, or certification of the piloting skills of a pilot who pilots a piloted object, including an aircraft or other moving object, and that includes an information import unit that acquires first video data that captures the piloted object being piloted by the pilot and second video data that captures the pilot inputting operations into a control device that controls the piloted object, an integrated data generation unit that generates or records integrated video data that integrates the first video data and the second video data, and a display output unit that displays the integrated video data during or after piloting the piloted object.

[0011] Furthermore, according to the present invention, there is provided an information control system that can be used to assist in training, evaluating, or certifying the piloting skills of a pilot who pilots a piloted object, including an aircraft or other moving object, and that includes an information import unit that acquires second video data that captures the pilot inputting operations into a control device that controls the piloted object, and a piloting skill evaluation unit that, based on the second video data, performs at least one of the following: evaluating and judging the pilot's piloting skills; making a pass / fail judgment as to whether the pilot meets the passing criteria for a piloting skill certification test; determining the cause of a low piloting skill evaluation; and determining areas for improvement in the piloting skills.

[0012] Furthermore, according to the present invention, there is provided an information control system that can be used to assist in training, evaluating, or certifying the piloting skills of a pilot who pilots a piloted object, including an aircraft or other moving object, and that includes an information import unit that acquires second video data that captures the pilot inputting operations into a control device that pilots the piloted object, and a training proposal generation unit that generates, based on the second video data, proposal information regarding at least one of the content and schedule of future training or certification tests for the pilot. [Effects of the Invention]

[0013] According to the present invention, it is possible to provide a system or method that can more effectively or appropriately train, evaluate, and certify pilots' piloting skills. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is an overall configuration diagram of an information control system 1 according to one embodiment of the present invention. [Figure 2] FIG. 1 is a functional block diagram showing the functional configuration of an aircraft 1000. [Figure 3] FIG. 2 is a hardware configuration diagram showing the external configuration of the control device 2000. [Figure 4] FIG. 3 is a functional block diagram showing the functional configuration of a training support system 3000. [Figure 5] FIG. 6 is a functional block diagram showing the functional configuration of a data analysis system 6000. [Figure 6] FIG. 10 is a diagram showing an example of pre-registered information registered in an evaluation criterion registration unit 3120. [Figure 7] 10 is a diagram showing an example of various pieces of acquired information acquired by an information import unit 3200. FIG. [Figure 8] FIG. 2 is a flowchart showing the processing flow of the information control system 1. [Figure 9] 10 is a diagram showing an example of integrated video data generated by an integrated data generating unit 3300. FIG. [Figure 10] FIG. 10 is a flowchart showing an example of the integrated video data output processing flow executed by the integrated data display output unit 3400. [Figure 11] FIG. 10 is a diagram showing an example of a method for receiving a display command and tag information for integrated video data by an integrated data display output unit 3400. [Figure 12] FIG. 10 is a diagram showing an example of a piloting skill evaluation result displayed and output by the piloting skill evaluation output unit 3610. [Figure 13] 10 is a diagram showing another example of a piloting skill evaluation result displayed and output by the piloting skill evaluation output unit 3610. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0015] The present invention will be described below with reference to the following embodiments. [Item 1] An information and control system that can be used to support training, evaluation, or certification of pilot skills of pilots who pilot objects including aircraft or other mobile objects, an information import unit that acquires first video data capturing a state of the control object being controlled by the operator and second video data capturing a state of the operator inputting an operation to a control device that controls the control object; an integrated data generating unit that generates or records integrated video data that integrates the first video data and the second video data; An information control system comprising a display output unit that displays and outputs the integrated video data during or after the operation of the object to be operated. [Item 2] In the information control system according to item 1, a display command receiving unit that receives a display command including at least one of playback, slow playback, pause, fast forward, and rewind from the operator, training instructor, or other user; The display output unit controls a display output mode of the integrated video data in response to the display command. [Item 3] In the information control system according to item 1 or 2, An information control system, wherein the integrated video data generated by the integrated data generation unit is video data in which the first video data and the second video data are integrated into a single video data, and the first video data is displayed in at least a portion of a display area and the second video data is displayed in at least another portion of a display area. [Item 4] In the information control system according to any one of items 1 to 3, the information import unit acquires third video data capturing an image of the eyes or line of sight of the pilot or a posture of the pilot while piloting the object to be piloted, An information control system, wherein the integrated video data generated by the integrated data generation unit is video data that integrates the first video data, the second video data, and the third video data. [Item 5] In the information control system according to any one of items 1 to 4, the information import unit acquires fourth video data of the surroundings of the target object captured by a mobile camera mounted on the target object; An information control system, wherein the integrated video data generated by the integrated data generation unit is video data that integrates the first video data, the second video data, and the fourth video data. [Item 6] In the information control system according to any one of items 1 to 5, the information import unit acquires control target information relating to the control target, including at least one of the position, movement path, speed, acceleration, attitude angle, angular velocity, angular acceleration, remaining battery charge, control communication radio wave strength, positioning signal strength, wind direction or wind speed around the control target, and control commands input to the control device by the operator; The integrated video data generated by the integrated data generation unit is video data in which the control target information is integrated in addition to the first video data and the second video data. [Item 7] In the information control system according to any one of items 1 to 6, the information import unit acquires pilot status information regarding the pilot, the pilot including at least one of a gaze direction or a focus position, a heart rate, a body temperature, a blood pressure, and sweating of the pilot while piloting the pilot object; An information control system, wherein the integrated video data generated by the integrated data generation unit is video data that integrates the first video data, the second video data, and the pilot status information. [Item 8] In the information control system according to any one of items 1 to 7, An information control system including a tag information assigning unit that records tag information for the integrated video data received from the pilot, training instructor, or other user in association with the integrated video data. [Item 9] In the information control system according to any one of items 1 to 8, An information control system comprising a piloting skill evaluation unit that performs at least one of the following: evaluation and judgment of the pilot's piloting skills; pass / fail judgment of whether the pilot meets the passing criteria for the piloting skill certification test; judgment of the causes of shortcomings in the piloting skills; and judgment of areas for improvement in the piloting skills. [Item 10] In the information control system according to any one of items 1 to 9, The piloting skill evaluation unit performs at least one of an evaluation and judgment of the pilot's piloting skills, a pass / fail judgment of whether the pilot meets the passing criteria for a piloting skill certification test, a judgment of the causes of shortcomings in the piloting skills, and a judgment of areas for improvement in the piloting skills, based on status information of the piloted object including at least one of the position, movement path, speed, acceleration, attitude angle, angular velocity, and angular acceleration of the piloted object, or piloting commands input to the piloting device by the pilot. [Item 11] In the information control system according to any one of items 1 to 10, The piloting skill evaluation unit performs at least one of the following based on the operational input operations of the control device captured as the second video data: an evaluation of the pilot's piloting skills, a pass / fail determination of whether the pilot meets the passing criteria for the piloting skill certification test, a determination of the cause of shortcomings in the piloting skills, and a determination of areas for improvement in the piloting skills. [Item 12] In the information control system according to any one of items 1 to 11, An information control system in which the criteria used for the judgment performed by the piloting skill evaluation unit include not releasing the pilot's finger from the operating stick of the piloting device, or not releasing it for more than a predetermined period of time. [Item 13] In the information control system according to any one of items 1 to 12, An information control system in which operating the control stick of the control device with the area around the tip of the pilot's finger is included in the judgment criteria used for the judgment performed by the piloting skill evaluation unit. [Item 14] In the information control system according to any one of items 1 to 13, The piloting skill evaluation unit performs at least one of an evaluation and judgment of the pilot's piloting skills, a pass / fail judgment of whether the pilot meets the passing criteria for a piloting skills certification test, a judgment of the cause of shortcomings in the piloting skills, and a judgment of areas for improvement in the piloting skills, based on third video data that captures the direction or focal position of the pilot's line of sight while piloting the piloted object, or the state of the pilot's eyes or line of sight or the pilot's posture while piloting the piloted object. [Item 15] In the information control system according to any one of items 1 to 14, An information control system in which moving the pilot's line of sight to the periphery of the object to be piloted is included in the criteria used for the judgment performed by the piloting skill evaluation unit. [Item 16] In the information control system according to any one of items 1 to 15, An information control system comprising a training proposal generator that generates proposal information regarding at least one of the content and schedule of future training or certification testing for the pilot. [Item 17] In the information control system according to any one of items 1 to 16, An information control system comprising a piloting skill evaluation recording unit that records, in association with the integrated video data, at least one of the judgment information of the pilot's piloting skills performed by the piloting skill evaluation unit, a pass / fail judgment of whether the piloting skills meet the passing criteria for the piloting skill certification test, a judgment of the cause of shortcomings in the piloting skills, and a judgment of areas for improvement in the piloting skills. [Item 18] In the information control system according to any one of items 1 to 17, The display output unit displays and outputs at least one of the judgment information, which is an evaluation judgment of the pilot's piloting skills performed by the piloting skill evaluation unit, a pass / fail judgment of whether the piloting skills meet the passing criteria for the piloting skills certification test, a judgment of the cause of shortcomings in the piloting skills, and a judgment of areas for improvement in the piloting skills, in association with the integrated video data. [Item 19] In the information control system according to any one of items 1 to 18, An information control system comprising a proposal information recording unit that records the proposal information generated by the training proposal generating unit in association with the integrated video data. [Item 20] In the information control system according to any one of items 1 to 19, The display output unit displays and outputs the proposal information generated by the training proposal generation unit in association with the integrated video data. [Item 21] An information control method that can be used to support training, evaluation, or certification of pilot skills of pilots who pilot piloting objects including aircraft or other moving objects, The computer an information acquisition step of acquiring first video data capturing a state of the control object being controlled by the operator and second video data capturing a state of the control device controlling the control object being operated and input by the operator; an integrated data generating step of generating or recording integrated video data by integrating the first video data and the second video data; and a display output step of displaying and outputting the integrated video data during or after the operation of the object to be operated. [Item 22] A program that can be used to train, evaluate, or assist in certifying the piloting skills of a pilot who pilots an object, including an aircraft or other mobile object, On the computer, an information acquisition command to acquire first video data capturing a state of the control object being controlled by the operator, and second video data capturing a state of a control device for controlling the control object being operated and input by the operator; an integrated data generation command for generating or recording integrated video data that integrates the first video data and the second video data; A program that executes a display output command to display and output the integrated video data during or after the operation of the object to be operated. [Item 23] An information control system that can be used to support training, evaluation, or certification of the piloting skills of a pilot who pilots a piloted object, including an aircraft or other moving object, and includes an information import unit that acquires second video data that captures the pilot inputting operations into a control device that controls the piloted object, and a piloting skill evaluation unit that performs at least one of the following based on the second video data: an evaluation of the pilot's piloting skills, a pass / fail determination of whether the pilot meets the passing criteria for a piloting skill certification test, a determination of the cause of a low evaluation of the piloting skills, and a determination of areas for improvement in the piloting skills. [Item 24] An information control system that can be used to assist in training, evaluating, or certifying the flight skills of a pilot who operates a controlled object including an aircraft or other moving object. The information control system includes an information import unit that acquires second video data obtained by photographing the state in which a control device for controlling the controlled object is operated by the pilot, and a training proposal generation unit that generates proposal information regarding at least one of the content and schedule of future training or certification tests of the pilot based on the second video data.

[0016] <A. First Embodiment> Hereinafter, embodiments of the present invention will be described with reference to the drawings. In this specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant description is omitted. Further, the embodiments shown below are merely examples, and other known elements and alternative means can be adopted according to the use, purpose, scale, etc.

[0017] [A-1. Configuration] (A-1-1. Overview) FIG. 1 is an overall configuration diagram of an information control system 1 (hereinafter also referred to as "system 1") according to an embodiment of the present invention. As shown in FIG. 1, the information control system 1 includes an aircraft 1000, a control device 2000, a training support system 3000, a controlled object information acquisition system 4000, a pilot information acquisition system 5000, and a data analysis system 6000. Further, these systems can be connected to each other so as to be communicable via a wired or wireless communication network.

[0018] The aircraft 1000 is an aircraft 1000 that is controlled by a control device 2000 operated by a pilot, and is, for example, an unmanned multicopter, a VTOL aircraft, or other aircraft. In this embodiment, the aircraft 1000 is described as an example of a controlled object, but the controlled object in the present invention is not limited to the aircraft 1000, and any device having movement or an actuator, such as a vehicle moving on land, a boat moving on water, a submarine moving underwater, or a robot operating in space, can be used as the controlled object.

[0019] The control device 2000 transmits control commands (forward, backward, left turn, right turn, ascend, descend, move left, move right, etc.) input by the pilot to the flying object 1000, and acquires from the flying object 1000 the flying object status (position, speed, acceleration, route, attitude, remaining battery charge), external environmental status (control communication radio wave strength, positioning signal strength, etc.), and surrounding image data captured by one or more cameras mounted on the flying object 1000 (such as image data captured below or to the side of the flying object 1000). In addition, the control device 2000 provides the control commands, flying object status, external environmental status, and surrounding image data to the training support system 3000.

[0020] The control target information acquisition system 4000 is a system that acquires information about the control target, such as the flying object 1000. For example, it acquires image data of the target captured by a camera showing the external operating state of the flying object 1000 during flight, or operation target information (including position, speed, acceleration, path, attitude, operation timing, etc.) of the flying object 1000's flight operation, or information about the wind speed and direction around the control target as the external environmental state, and provides this information to the training support system 3000.

[0021] The operator information acquisition system 5000 is a system that acquires information about the operator, such as finger movement video data of the operator that captures the control device 2000 and the movements of the operator's fingers operating the control device 2000 (i.e., the manner in which the control device 2000 is operated by the operator's fingers), gaze video data that captures the movements of the operator's eyes and gaze, posture video data regarding the posture of the operator holding the control device in his or her hand, posture measurement data regarding the posture including the direction of the operator's body and head, and even biological detection information that indicates the operator's condition (heart rate, blood pressure, sweating, body temperature, etc.), and provides this information to the training support system 3000.

[0022] Here, the finger movement video data is, for example, video data of the control device captured by a camera fixed to the control device, the operator's chest, or another location. The gaze video data is, for example, video data of eye movements captured by eyewear, or data in which gaze information obtained by image analysis of the video data is integrated into the video data. The posture video data is video data of the operator holding the control device captured by a camera installed behind or around the operator. The posture measurement data and biometric detection information are, for example, biometric information of the operator measured by a wearable device worn by the operator.

[0023] Here, the photographed object video data acquired by the control object information acquisition system 4000 and the finger movement video data, gaze video data, and posture video data acquired by the operator information acquisition system 5000 may be video data captured by individual cameras, or may be video data obtained by cutting out a partial area of ​​a wide-angle video captured by a common wide-angle camera. In other words, at least two of the photographed object video data, finger movement video data, gaze video data, and posture video data may be video data captured by a common camera.

[0024] The training support system 3000 acquires various data from the control device 2000, the control target information acquisition system 4000, and the pilot information acquisition system 5000, integrates this data to generate integrated data, and displays and outputs the integrated data to the pilot and his / her instructor.

[0025] The data analysis system 6000 generates piloting skill evaluation information and training suggestion information for the pilot based on various data acquired from the training support system 3000, and provides this information to the training support system 3000. Note that the transfer of various data from the training support system 3000 to the data analysis system 6000 may be performed by wired or wireless data transmission between the training support system 3000 and the data analysis system 6000, but is not limited to this. For example, the data may be transferred by a user manually removing a recording device on which various data is recorded from the training support system 3000 and connecting it to the data analysis system 6000.

[0026] Furthermore, the data analysis system 6000 can be connected to the training support system 3000 via an internet line and implemented in a server device located in a remote location away from the site where the pilot pilots the aircraft 1000, but is not limited to this, and the data analysis system 6000 can be implemented in a computer device provided at the site where the pilot pilots the aircraft 1000. In this case, the data analysis system 6000 may be implemented in a computer device common to the training support system 3000.

[0027] (A-1-2. Configuration of Aircraft 1000) 2 is a functional block diagram showing the functional configuration of the flying object 1000. The flying object 1000 includes a measurement unit 1100, an aircraft state determination unit 1200, a flying unit 1300, and a communication unit 1400.

[0028] The measurement unit 1100 is a functional unit that acquires surrounding image data obtained by capturing images of the surroundings of the aircraft 1000 using one or more optical cameras mounted on the aircraft 1000. The measurement unit 1100 can also be equipped with other sensors, such as an infrared camera, instead of or in addition to the optical camera.

[0029] In addition, the above-mentioned camera may be installed facing downward on the aircraft 1000 to photograph the ground, or may be installed facing sideways to photograph walls or buildings to the side, or may be installed facing upward on the aircraft 1000 to photograph the ceiling, in order to more accurately grasp the behavior of the aircraft 1000.

[0030] Next, the aircraft state determination unit 1200 is a functional unit that determines the state of the aircraft itself, including its position, speed, etc. The aircraft state determination unit 1200 can determine, for example, the three-dimensional position, speed, acceleration, attitude angle, angular velocity of the attitude angle, angular acceleration, flight path so far, and remaining battery charge of the aircraft itself (air vehicle 1000). The aircraft state determination unit 1200 can also determine the external environmental state of the aircraft itself, such as the communication radio wave strength of the control command from the control device 2000 and the strength of satellite signals for positioning received from GNSS satellites, etc.

[0031] The method for measuring the attitude (orientation) of the aircraft is to measure the current attitude of the aircraft using, for example, a geomagnetic sensor, a GNSS compass, etc. Attitude information includes at least the attitude angle (orientation) in a planar view around the Z axis, and preferably includes attitude information around three axes: the X axis, the Y axis, and the Z axis. In addition, angular velocity and angular acceleration can be calculated based on the amount of change over time in the measured attitude information.

[0032] Next, flying unit 1300 is a functional unit for flying the aircraft, and is a functional unit that controls the flight operations of aircraft 1000 in accordance with control commands received from control device 2000. Flying unit 1300 includes thrust generation unit 1310 and flight control unit 1320. Thrust generation unit 1310 is composed of multiple propellers and other devices capable of generating thrust.

[0033] The flight control unit 1320 is a functional unit that controls the output from the thrust generation unit 1310 to control the flight operation of the air vehicle 1000. The flight control unit 1320 can be configured with a processing unit also called a flight controller. The processing unit is configured to control the flight state of the air vehicle 1000. For example, the processing unit adjusts the spatial position, velocity, acceleration, attitude angle, angular velocity, and angular acceleration of the air vehicle 1000, which has six degrees of freedom (translational motion x, y, and z, and rotational motion θx, θy, and θz).

[0034] Next, the communication unit 1400 is a functional unit that transmits and receives control commands, the state of the flying object, the state of the external environment, and surrounding image data between the flying object 1000 and the control device 2000.

[0035] (A-1-3. Control unit 2000) Figure 3 is a hardware configuration diagram showing the external configuration of the control device 2000. As shown in Figure 3, the control device 2000 is composed of a display unit 2100 provided on the top and an operation unit 2200 provided on the bottom, and the display unit 2100 and the operation unit 2200 are connected to each other so that they can communicate with each other via wire or wirelessly. Note that the example shown in Figure 3 shows a case where mode 1 is selected as the operation mode of the control device.

[0036] Display unit 2100 displays in real time various information such as the aircraft status and surrounding video data transmitted from aircraft 1000. In addition to the various information transmitted from aircraft 1000, display unit 2100 may also display information on the remaining battery charge of control device 2000.

[0037] The operation unit 2200 is provided with a right operation stick 2210, a left operation stick 2220, and a communication antenna 2230. The communication antenna 2230 is an antenna that enables the control device 2000 to communicate with the flying object 1000 and the training support system 3000.

[0038] 3, the right operating stick 2210 can be tilted forward to input an operation command to raise the aircraft 1000, or tilted backward to input an operation command to lower the aircraft 1000, and can also be tilted right to input an operation command to move the aircraft 1000 horizontally to the right, or tilted left to input an operation command to move the aircraft 1000 horizontally to the left. The left operating stick 2220 can be tilted forward to input an operation command to move the aircraft 1000 forward, or tilted backward to input an operation command to turn the aircraft 1000 right, or tilted left to input an operation command to turn the aircraft 1000 left.

[0039] The operation mode of the control device is not limited to mode 1, and other modes such as mode 2, mode 3, and mode 4 are also possible. In mode 2, the right operating stick 2210 can be used to input forward movement, backward movement, right movement, and left movement, and the left operating stick 2220 can be used to input ascending, descending, turning right, and turning left. In mode 3, the right operating stick 2210 can be used to input ascending, descending, turning right, and turning left, and the left operating stick 2220 can be used to input forward movement, backward movement, right movement, and left movement. Finally, in mode 4, the right operating stick 2210 can be used to input forward movement, backward movement, right turn, and turning left, and the left operating stick 2220 can be used to input ascending, descending, turning right, and turning left.

[0040] The pilot holds the operation unit 2200 with both hands and tilts the right operation stick with the thumb of his right hand and the left operation stick with the thumb of his left hand in any direction and at any angle, thereby inputting a desired operation command to the flying object 1000. Note that, although Fig. 3 shows an example of a control device equipped with operation sticks on both the left and right sides, the control device of the present invention is not limited to this, and may be a control device configured to control the operation stick with one hand.

[0041] (A-1-4. Training Support System 3000) 4 is a functional block diagram showing the functional configuration of training support system 3000. Training support system 3000 includes pre-registration information accepting unit 3100, information import unit 3200, integrated data generating unit 3300, integrated data display output unit 3400, data analysis request command unit 3500, and data analysis result output unit 3600.

[0042] (A-1-4-1. Pre-registration information reception unit 3100) The pre-registration information receiving unit 3100 is a functional unit having a function of receiving pre-registration information from a user. The pre-registration information receiving unit 3100 includes an integration information registration unit 3110, an evaluation criterion registration unit 3120, and a proposal criterion registration unit 3130.

[0043] The integration information registration unit 3110 pre-registers information related to the video data integration process performed by the integrated data generation unit 3300, which will be described later. For example, the integration information registration unit 3110 can register the data types to be integrated as the video data integration process and detailed conditions for the video data integration. As the data types, for example, the target video data, surrounding video data, video data of the pilot's fingers, video data of the pilot's line of sight, video data of the pilot's posture, and biometric data of the pilot can be registered. Furthermore, as detailed conditions for the video data integration, for example, the designation of main video data and sub video data, the designation of a display area for the sub video data, and the like can be registered for each data type.

[0044] The evaluation criterion registration unit 3120 is a functional unit that registers criteria information for judgment regarding skill evaluation performed by a piloting skill evaluation unit 6200 of the data analysis system 6000, which will be described later. A detailed example of the registered information registered in the evaluation criterion registration unit 3120 will be described later.

[0045] The proposal criteria registration unit 3130 pre-registers information about training proposals to be implemented by a training proposal generation unit 6300 of the data analysis system 6000, which will be described later. For example, the information includes training items to be proposed according to the skill evaluation results, the number of training sessions, the training schedule, the training method, and the re-examination date for the skill certification test.

[0046] Fig. 6 is a diagram showing an example of pre-registered information registered in evaluation criterion registration unit 3120. As shown in Fig. 6, evaluation criterion registration unit 3120 includes operation target information of the aircraft, target information of the operation command input by the pilot, operation target information of the pilot's fingers, operation target information of the pilot's eyes, target information of the pilot's posture, judgment criteria for skill evaluation, and pass criteria for the skill certification test. The operation target information of the aircraft includes, for example, the movement path (position), speed, acceleration, attitude angle, angular velocity, angular acceleration, and behavior timing of the aircraft.

[0047] The motion target information of the operator's fingers includes, for example, keeping the thumb on the control stick, operating the stick with the tip of the thumb, touching the stick in the same position with the thumb, and other motion targets of the thumb (position, trajectory, speed, attitude angle, movement timing).

[0048] The target information of the operation command input by the operator includes, for example, an input command value for ascending or descending, an input command value for moving left or right, an input command value for forward update, an input command value for turning left or right, and the like.

[0049] The pilot's eye movement target information includes, for example, shifting the gaze to the aircraft and its surroundings at a predetermined frequency, shifting the gaze forward in the direction of flight while the aircraft is moving, and shifting the gaze diagonally downward and forward a predetermined distance from the aircraft while it is moving. Furthermore, the pilot's posture target information includes holding the control device horizontally, holding the control device at a predetermined height, and facing the body or head toward the aircraft.

[0050] The skill evaluation criteria may include, for example, an evaluation rank according to a difference value from the aircraft's operation target, an evaluation rank according to a difference value from the control command target, an evaluation rank according to a difference value from the finger operation target, an evaluation rank according to a difference value from the eye operation target, and an evaluation rank according to a difference value from the posture target. In other words, a registered evaluation criterion is one in which the skill evaluation rank is higher if the difference value between the pilot's operation performance and each operation target is small, and conversely, the skill evaluation rank is lower if the difference value from each operation target is large. Note that the skill evaluation criteria may also be criteria that comprehensively evaluate the differences between the pilot's operation performance and each target of the aircraft, control command, fingers, eyes, and posture.

[0051] Alternatively, the skill evaluation criteria are not limited to the rule-based evaluation described above, and can also be calculated as a similarity evaluation with expert data based on a trained model that uses pilot performance data (aircraft movement, finger movement, eye movement, piloting commands) by an expert as teaching data. Alternatively, the skill evaluation criteria may be a combination of the rule-based evaluation described above and evaluation using a trained model.

[0052] The pass criterion for the skills certification test may be, for example, that the rank of the skills evaluation determined based on the above-mentioned skill evaluation criteria is a predetermined rank or higher.

[0053] (A-1-4-2. Information import unit 3200) Next, information import unit 3200 is a functional unit that acquires information used in the integrated data generated by training support system 3000, or information used in skill evaluations and the generation of training proposals performed by data analysis system 6000. Information import unit 3200 includes a control device information acquisition unit 3210, a control target information acquisition unit 3220, and a pilot information acquisition unit 3230.

[0054] The control device information acquisition unit 3210 is a functional unit that acquires various information transmitted from the control device 2000. The control device information acquisition unit 3210 acquires, for example, control commands, the state of the flying object, the state of the external environment, and surrounding image data.

[0055] The control target information acquisition unit 3220 is a functional unit that acquires various information transmitted from the control target information acquisition system 4000. The control target information acquisition unit 3220 acquires, for example, image data of the target, operation target information, and external environment information related to wind speed and wind direction.

[0056] The operator information acquisition unit 3230 is a functional unit that acquires various pieces of information related to the operator transmitted from the operator information acquisition system 5000. The operator information acquisition unit 3230 acquires the operator's finger movement video data, eye movement video data, gaze estimation information, posture video data, posture measurement data, and biological information detection information.

[0057] Fig. 7 is a diagram showing an example of various acquired information acquired by the information import unit 3200. As shown in Fig. 7, the information import unit 3200 acquires information related to control commands, aircraft status, external environment status, surrounding video data, target video data, and pilot status.

[0058] The control commands include, for example, command input values ​​for ascending and descending, command input values ​​for moving left and right, command input values ​​for forward update, command input values ​​for turning left and right, etc. The aircraft state includes the aircraft's movement path (position), speed, acceleration, attitude angle, angular velocity, angular acceleration, remaining battery charge, etc. The external environment state includes control communication radio wave strength, positioning signal strength, wind direction, wind speed, etc. The surrounding image data includes surrounding images of the aircraft 1000 captured by a camera mounted on the aircraft 1000 (images taken downward, images taken sideways, etc.).

[0059] The controlled object video data includes video of the flying object 1000 captured from the periphery, showing the behavior of the flying object 1000. The pilot status includes photographed video data of the pilot's fingers manipulating the left and right control sticks of the control device 2000, photographed video data of the pilot's eye movements, estimated information on the gaze direction or gaze focus position estimated based on the pilot's eye movements, posture video data on the posture of the pilot holding the control device in his / her hand, posture measurement data on the orientation of the pilot's body and head, and the pilot's heart rate, blood pressure, sweating, body temperature, and other biological information.

[0060] (A-1-4-3. Integrated data generation unit 3300) The integrated data generating unit 3300 is a functional unit that generates integrated data by integrating various pieces of acquired information. The integrated data generating unit 3300 includes a data integrating unit 3310 and an integrated data recording unit 3320.

[0061] The data integration unit 3310 is a functional unit that performs integration processing of various types of data based on the data type for which video data integration processing is performed and detailed conditions for video data integration registered in the integration information registration unit 3110. Here, the data integration processing is, for example, processing of integrating multiple pieces of video data including first and second video data specified from various acquired video data into a single piece of video data, and the integrated video data is video data in which the first video data is displayed in at least a portion of the display area of ​​the integrated video data after integration, and the second video data is displayed in at least another portion of the display area.

[0062] The integrated data recording unit 3320 is a functional unit that records the integrated data generated by the data integration unit 3310 .

[0063] Here, when using the control device 2000 to control a control target such as the aircraft 1000, improving the finger movements when operating the control device is effective in improving piloting skills. However, while piloting the aircraft 1000, the pilot needs to operate the control stick of the control device 2000 while looking at the aircraft 1000 and its surroundings. Therefore, the pilot does not see and check the movements of his or her own fingers, making it difficult to recognize the habits of his or her own finger movements. Furthermore, the pilot's actual finger movements often differ from the finger movements the pilot imagines. Therefore, the pilot is required to correctly recognize the actual movements of his or her own fingers and consider areas for improvement.

[0064] The data integration unit 3310 can integrate video data and the like using any combination of data registered in the integration information registration unit 3110, and in particular can generate integrated video data that integrates video data of the object to be controlled (video data capturing the behavior of the air vehicle 1000) with captured video data of the pilot's fingers (control device 2000). In this way, by integrating the video data of the air vehicle 1000 and the video data of the pilot's fingers, the pilot or training instructor can simultaneously check the behavior of the air vehicle 1000 and the movement of the pilot's fingers (i.e., the operation of the control stick of the control device 2000), thereby enabling the pilot to more effectively review the piloting content and consider areas for improvement in the piloting.

[0065] Furthermore, the data integrator 3310 can generate integrated video data by integrating the video data of the control object (video data capturing the behavior of the air vehicle 1000) and the captured video data of the pilot's fingers (control device 2000), as well as the captured video data of the pilot's eye movements or the pilot's posture. Alternatively, the data integrator 3310 can generate integrated video data by integrating the video data of the control object (video data capturing the behavior of the air vehicle 1000) and the captured video data of the pilot's eye movements or the pilot's posture. In this way, by integrating the video data of the air vehicle 1000 and the video data of the pilot's eyes or posture, the pilot or training instructor can simultaneously check the behavior of the air vehicle 1000, the pilot's eye movements (i.e., the pilot's line of sight and visual range), the position and angle at which the pilot holds the control device, the orientation of the pilot's body and head, etc., thereby enabling the pilot to more effectively review the pilot's operation and consider areas for improvement in the operation.

[0066] Furthermore, the data integration unit 3310 can generate integrated video data by integrating the control target video data (video data capturing the behavior of the air vehicle 1000) and the captured video data of the pilot's fingers (control device 2000), as well as the peripheral video data of the air vehicle 1000 captured by a camera mounted on the air vehicle 1000. In this way, by integrating the peripheral video data (video data captured in a downward and lateral direction) in addition to the video data of the air vehicle 1000 and the video data of the pilot's fingers, the pilot or training instructor can grasp the behavior of the air vehicle 1000 (such as changes in attitude angle) in more detail based on not only the video captured from the outside of the air vehicle 1000 but also the video data captured in a downward and lateral direction, thereby enabling the pilot to more effectively review the operation and consider areas for improvement in the operation. Note that the peripheral video data can also be generated as an around-view video by integrating multiple peripheral video data captured by multiple cameras.

[0067] Furthermore, the data integration unit 3310 can generate integrated video data that integrates not only the video data of the aircraft to be controlled (video data capturing the behavior of the aircraft 1000) and the captured video data of the pilot's fingers (control device 2000), but also the aircraft's status, such as its movement path (position), speed, acceleration, attitude angle, angular velocity, angular acceleration, and remaining battery charge, as well as the external environmental status, such as control communication radio wave strength and positioning signal strength. In this way, by integrating the aircraft status and the external environmental status in addition to the video data of the aircraft 1000 and the video data of the pilot's fingers, the pilot or training instructor can more accurately grasp the behavior of the aircraft 1000 and can also grasp the status of the aircraft 1000 and its external environment, allowing the pilot to more effectively review the pilot's operations and consider areas for improvement in their operations.

[0068] Furthermore, the data integration unit 3310 can generate integrated video data that integrates not only the video data of the object to be controlled (video data capturing the behavior of the flying object 1000) and the captured video data of the pilot's fingers (control device 2000), but also estimated information on the pilot's line of sight or the focal position of the line of sight, and the pilot's heart rate, blood pressure, sweating, body temperature, and other biological information. In this way, by integrating the video data of the flying object 1000 and the video data of the pilot's fingers, as well as the line of sight and biological information, the pilot and training instructor can more easily grasp the pilot's level of tension, physical condition, or piloting tasks that the pilot feels uncomfortable with, thereby more effectively allowing the pilot to review the piloting content and consider areas for improvement in the piloting.

[0069] The timing for executing the data integration process by the data integration unit 3310 may be immediately after each piece of information is acquired by the information import unit 3200, but is not limited to this. The data integration process can also be executed immediately after a display command is received by the display command receiving unit 3420, which will be described later.

[0070] (A-1-4-4. Integrated data display output unit 3400) The integrated data display output unit 3400 is a functional unit that displays and outputs the integrated video data generated by the integrated data generation unit 3300 during or after operation. The integrated data display output unit 3400 includes a real-time output unit 3410, a display command receiving unit 3420, a display control unit 3430, and a tag information assignment unit 3440.

[0071] The real-time output unit 3410 is a functional unit that displays and outputs the integrated video data generated by the integrated data generation unit 3300 from the output device 200 in real time while the flying object 1000 is being operated. The integrated video data that is displayed and output in real time is particularly useful for training instructors to grasp the operation situation and the status of the pilot in real time.

[0072] The display command receiving unit 3420 is a functional unit that receives, from the pilot, training instructor, or other user, a display command relating to a method for displaying and outputting the integrated video data generated by the integrated data generating unit 3300 and recorded in the integrated data recording unit 3320. The display command receiving unit 3420 can receive display commands including, for example, at least one of play, slow play, pause, fast forward, and rewind. The display command receiving unit 3420 can also input, as search information, video at the time when a tag was added using tag information added by the tag information adding unit 3440, which will be described later.

[0073] The display control unit 3430 is a functional unit that controls the display output mode of the integrated video data recorded in the integrated data recording unit 3320 in response to a display command received by the display command receiving unit 3420. For example, the display control unit 3430 controls display in response to commands received as display commands, such as play, slow play, pause, fast forward, and rewind. The display control unit 3430 can also display and output video at the time when tag information received as a display command was added.

[0074] The tag information assigning unit 3440 is a functional unit that receives tag information for the integrated video data that has been displayed from the pilot, the training instructor, or another user, and records the tag information in association with the integrated video data. Here, the tag information is, for example, information such as shortcomings, findings, and areas for improvement in the operation that is input by the pilot after checking the integrated video data.

[0075] (A-1-4-5. Data analysis request command unit 3500) The data analysis request command unit 3500 is a functional unit that transmits a request command for data analysis processing to the data analysis system 6000. The data analysis request command unit 3500 includes a piloting skill evaluation request unit 3510 and a training suggestion request unit 3520.

[0076] The piloting skill evaluation request unit 3510 is a functional unit that requests the data analysis system 6000 to perform at least one of the following judgment processes: evaluation and judgment of the pilot's piloting skills, pass / fail judgment of whether the pilot meets the passing criteria for the piloting skill certification test, judgment of the cause of a low piloting skill evaluation, and judgment of areas for improvement in the piloting skills.

[0077] The training proposal request unit 3520 is a functional unit that requests the data analysis system 6000 to generate proposal information regarding at least one of the contents and schedule of future pilot training or certification testing.

[0078] (A-1-4-6. Data analysis result output unit 3600) The data analysis result output unit 3600 is a functional unit that displays, outputs, or records each determination result determined by the data analysis processing by the data analysis system 6000 and the generated proposal information. The data analysis result output unit 3600 includes a piloting skill evaluation output unit 3610, a piloting skill evaluation recording unit 3620, a training proposal output unit 3630, and a training proposal recording unit 3640.

[0079] The piloting skill evaluation output unit 3610 is a functional unit that associates at least one of the judgment information executed by the data analysis system 6000, including an evaluation of the pilot's piloting skills, a pass / fail judgment of whether the pilot meets the passing criteria for the piloting skill certification test, a judgment of the cause of a low piloting skill evaluation, and a judgment of areas for improvement in the piloting skills, with the integrated video data and displays and outputs it from the output device 200.

[0080] The piloting skill evaluation recording unit 3620 is a functional unit that records at least one of the judgment information performed by the data analysis system 6000, such as an evaluation of the pilot's piloting skills, a pass / fail judgment of whether the pilot meets the passing criteria for the piloting skill certification test, a judgment of the cause of a low piloting skill evaluation, and a judgment of areas for improvement in the piloting skills, in association with the integrated video data.

[0081] The training proposal output unit 3630 is a functional unit that associates proposed information regarding at least one of the content and schedule of future pilot training or certification testing generated by the data analysis system 6000 with integrated video data and displays and outputs it from the output device 200.

[0082] The training proposal recording unit 3640 is a functional unit that records proposal information regarding at least one of the content and schedule of future pilot training or certification testing generated by the data analysis system 6000 in association with the integrated video data.

[0083] (A-1-5. Data Analysis System 6000) 5 is a functional block diagram showing the functional configuration of a data analysis system 6000. The data analysis system 6000 includes an information acquisition unit 6100, a piloting skill evaluation unit 6200, a training suggestion generation unit 6300, an information output unit 6400, and a data recording unit 6500.

[0084] (A-1-5-1. Information acquisition unit 6100) The information acquisition unit 6100 is a functional unit that acquires data analysis request commands and integrated video data from the training support system 3000. The information acquisition unit 6100 includes a piloting skill evaluation request unit 6110, a training suggestion request unit 6120, and an integrated data acquisition unit 6130.

[0085] The piloting skill evaluation request unit 6110 is a functional unit that obtains a request command from the piloting skill evaluation request unit 3510 for at least one of the following judgment processes: evaluation and judgment of the pilot's piloting skills; pass / fail judgment of whether the pilot meets the passing criteria for the piloting skill certification test; judgment of the cause of a low piloting skill evaluation; and judgment of areas for improvement in the piloting skills.

[0086] The training proposal request unit 6120 is a functional unit that acquires from the training proposal request unit 3520 a request command for processing to generate proposal information relating to at least one of the contents and schedule of future pilot training or certification testing.

[0087] The integrated data acquisition unit 6130 is a functional unit that acquires various information used in the data processing performed by the piloting skill evaluation unit 6200 and training suggestion generation unit 6300 described below, and acquires, for example, integrated video data generated by the integrated data generation unit 3300 or various information acquired by the information import unit 3200.

[0088] (A-1-5-2. Pilot Skills Evaluation Section 6200) The piloting skill evaluation unit 6200 is a functional unit that executes evaluation and judgment of the pilot's piloting skills, pass / fail judgment of whether the pilot meets the passing criteria for the piloting skill certification test, judgment of the cause of a low piloting skill evaluation, judgment of areas for improvement in piloting skills, or other judgment processing related to piloting skills. The piloting skill evaluation unit 6200 includes a skill evaluation judgment unit 6210, a certification test pass / fail judgment unit 6220, a defect cause judgment unit 6230, and an improvement method proposal unit 6240.

[0089] The skill evaluation determination unit 6210 is a functional unit that evaluates and determines the pilot's piloting skills based on the integrated video data acquired by the integrated data acquisition unit 6130 or various other information. The skill evaluation determination unit 6210 can, for example, determine a skill evaluation for each maneuver item. For example, a skill evaluation can be performed for each type of operation of the aircraft 1000 (high-speed curves, figure-eight flight, return flight toward the pilot, turning, takeoff, landing, hovering, sudden stops, sudden acceleration, etc.). For example, in the evaluation of figure-eight flight, an evaluation criterion can be set such that the evaluation of figure-eight flight is lowered if the aircraft 1000 is unable to fly at a constant speed or if the timing or angle of the rudder (turn) is inappropriate for the flight speed. Furthermore, an evaluation criterion can be set such that the evaluation of the aircraft 1000 is lowered if the position or attitude of the aircraft 1000 is unstable in general.

[0090] Alternatively, the skill evaluation unit 6210 can evaluate and judge the movements of the pilot's fingers (operational movements of the control stick), the movements of the pilot's eyes, or the pilot's posture (the angle and position at which the control device is held, and the orientation of the body and head relative to the controlled object). The evaluation can be calculated as a number of predefined evaluation ranks (ranks 1 to 5) or evaluation scores (with 100 being the perfect score). Furthermore, the skill evaluation unit 6210 is not limited to absolute evaluations; it can also perform a relative evaluation of the pilot's past driving history by analyzing the difference from the pilot's past driving history. In other words, based on the integrated video data acquired by the integrated data acquisition unit 6130 or various other information, it can determine whether the skill level has significantly improved or significantly deteriorated if there is a large difference from the past driving history.

[0091] In this way, skill evaluation is performed for each movement of each aircraft 1000, the movement of the pilot's fingers, the movement of the pilot's eyes, and the pilot's posture, so pilots and training instructors can understand the strengths and weaknesses of piloting skills in more detail, allowing them to more appropriately review the piloting content and decide on future training directions, thereby more effectively improving the pilot's skills.

[0092] The certification test pass / fail determination unit 6220 is a functional unit that determines whether or not the pass criteria for the pilot skill certification test are met based on the integrated video data acquired by the integrated data acquisition unit 6130 or various other information. For example, it can determine whether or not the pass criteria for the certification test to certify a pilot as a pilot to pilot an object to be piloted, such as the aircraft 1000, are met.

[0093] The defect cause determination unit 6230 is a functional unit that determines the cause of defects in piloting skills based on the integrated video data acquired by the integrated data acquisition unit 6130 or various other information. The defect cause determination unit 6230 can determine that the cause of low piloting skills is finger movement based on the movement of the pilot's fingers. For example, if there is a large difference between the flight performance of the aircraft 1000 and the flight target, and the finger is away from the control stick, the control stick is not being operated with the tip of the finger, or the finger is touching the control stick in a different position, it can be determined that the above-mentioned finger movement is the cause of the inability to fly along the flight target.

[0094] The defect cause determination unit 6230 can determine that the cause of low piloting skill is eye movement based on the eye movement. For example, if there is a large difference between the flight performance of the aircraft 1000 and the flight target, and the pilot's line of sight is fixed on the aircraft 1000 and not looking at the surroundings, it can be determined that the cause of inability to fly along the flight target is the position of the line of sight. In another example, if the flight path of the aircraft 1000 deviates from the target path, and the pilot's line of sight is not directed ahead in the direction of travel of the aircraft 1000, it can be determined that the cause of inability to fly along the flight target is the position of the line of sight.

[0095] Furthermore, the defect cause determination unit 6230 can determine that the cause of low piloting skill is the pilot's attitude based on the pilot's attitude. For example, if there is a large difference between the flight performance of the air vehicle 1000 and the flight target, and the pilot holds the control device at an angle rather than horizontally, or if the pilot holds the control device in a position that is too high or too low while piloting, it can be determined that the cause of the inability to fly along the flight target is the pilot's attitude. In another example, if the flight path of the air vehicle 1000 deviates from the target path, and the pilot's body or head is not facing the direction of the target, it can be determined that the cause of the inability to fly along the flight target is the pilot's attitude.

[0096] The defect cause determination unit 6230 can also determine the pilot's level of tension and the pilot's perceived weakness in flight operations based on the acquired biometric information. When there is a large difference between the flight performance of the aircraft 1000 and the flight target, and the pilot's heart rate is higher than a predetermined value (or there is increased sweating, blood pressure, or body temperature), it can be determined that the pilot's inability to fly in accordance with the flight target is due to excessive tension. When the pilot's heart rate is higher than a predetermined value (or there is increased sweating, blood pressure, or body temperature) during a common aircraft 1000 operation over multiple flights, it can be determined that the pilot is perceived as being weak in that aircraft 1000 operation. The defect cause determination unit 6230 can also determine the cause of the large difference between the flight performance of the aircraft 1000 and the flight target based on a combination of the pilot's finger movements, eye movements, posture, and biometric information.

[0097] The improvement method suggestion unit 6240 is a functional unit that determines areas for improvement in piloting skills based on the integrated video data acquired by the integrated data acquisition unit 6130 or various other information. For example, it can generate suggestion information that identifies the cause of a large difference between the flight performance and the flight target, as determined by the defect cause determination unit 6230, as an area for improvement in piloting.

[0098] In other words, the improvement method suggestion unit 6240 can generate suggested information on areas for improvement in the pilot's finger movements (fingers away from the control stick, not operating the control stick with the finger tips, touching the control stick with different finger positions, etc.), areas for improvement in the pilot's eye movements (staring at the aircraft 1000 and not looking at the surroundings, not looking ahead in the direction of the aircraft 1000's travel, not shifting the gaze diagonally downward and forward a predetermined distance from the aircraft while moving, etc.), and areas for improvement in the pilot's posture (the angle or position at which the control device is held is inappropriate, the body or head is not facing the direction of the object to be controlled, etc.).

[0099] In addition, the improvement method suggestion unit 6240 can generate suggested information that includes not only the defect causes determined by the defect cause determination unit 6230, but also the timing, speed, and trajectory of the operator's finger movements, the timing, speed, and trajectory of the operator's eye movements, or other tips for operation.

[0100] Here, the skill evaluation determination unit 6210, the certification test pass / fail determination unit 6220, the defect cause determination unit 6230, and the improvement method proposal unit 6240 can perform evaluation and determination of piloting skills, pass / fail determination of the certification test, determination of the cause of a low evaluation of piloting skills, or determination of areas for improvement in piloting skills based on the skill evaluation determination criteria registered in the evaluation criteria registration unit 3120 of the training support system 3000. The skill evaluation determination criteria can be evaluation criteria based on the difference between the operational performance of the flying object 1000 (at least one of the position, movement path, speed, acceleration, attitude angle, angular velocity, and angular acceleration of the piloted object) and the operational target of the flying object 1000.

[0101] Alternatively, the skill evaluation criteria may be based on a difference between the operation command (up / down, left / right movement, forward / backward movement, left / right turn) input by the pilot to the control device 2000 and the target of the operation command. Alternatively, the skill evaluation criteria may be based on a difference between the operation input motion of the pilot's fingers and the target operation input motion of the fingers, as determined from video data of the pilot's fingers capturing the state in which the pilot operates the control device 2000. Alternatively, the skill evaluation criteria may be based on a difference between the actual eye movement, including the direction or focus position of the pilot's gaze, and the target eye movement, as determined from video data of the pilot's eyes. Alternatively, the skill evaluation criteria may be based on a difference between the angle or position at which the control device is held, or the posture state, including the orientation of the pilot's body or head, as determined from posture video data, and the posture target.

[0102] The piloting skill evaluation unit 6200 can make various judgments (evaluation / determination of the pilot's piloting skill, pass / fail determination of whether the pilot meets the pass criteria for the piloting skill certification test, determination of the cause of a low piloting skill evaluation, determination of areas for improvement in piloting skill, or other judgments related to piloting skill) using a judgment criterion in which the finger movement target includes not releasing the pilot's finger from the control stick of the piloting device 2000 (or, even if the pilot releases the finger, the time for which the release is made is within a predetermined time). This is because if the finger releases the control stick, the timing of the next operation input is delayed, the amount of operation input is likely to be excessive and difficult to adjust, or the position of the finger touching the control stick changes with each operation, making it difficult to adjust the operation input. Therefore, by setting not releasing the finger from the control stick during operation as a finger movement target, it is possible to more appropriately evaluate / determine the pilot's piloting skill, determine whether the pilot passes the piloting skill certification test, determine the cause of a low piloting skill evaluation, and determine areas for improvement in piloting skill.

[0103] Furthermore, the piloting skill evaluation unit 6200 can perform various evaluations using evaluation criteria in which the finger movement target includes operating the control stick of the control device 2000 with the area around the pilot's fingertip. When operating the control stick with, for example, the pad of a finger rather than the tip of a finger, it is difficult to move the fingertip to an arbitrary position using the first joint of the finger. The relatively limited range of motion of the fingertip makes it difficult to move the control stick in the intended direction. Therefore, a control stick operated with the pad of a finger is likely to move in an arc-shaped trajectory even when the pilot intends to move it in a straight line. Therefore, by setting the finger movement target to operate the control stick with the pilot's fingertip, it becomes possible to more appropriately evaluate and evaluate the pilot's piloting skills, determine whether the pilot passed the piloting skill certification test, determine the cause of a low piloting skill evaluation, and determine areas for improvement in the piloting skills.

[0104] Furthermore, the piloting skill evaluation unit 6200 can make various judgments using a judgment criterion in which the finger movement target includes touching the control stick at the same position of the pilot's finger. If the pilot does not touch the control stick at the same position of the finger but touches the control stick at a different position, it becomes difficult for the pilot to move the control stick at the timing intended by the pilot or tilt the control stick by the intended angle. Therefore, by setting touching the control stick at the same position of the pilot's finger as the finger movement target, it becomes possible to more appropriately evaluate and judge the pilot's piloting skills, judge whether the pilot has passed the piloting skill certification test, determine the cause of a low piloting skill evaluation, and determine areas for improvement in the piloting skills.

[0105] Furthermore, the piloting skill evaluation unit 6200 can make various determinations using a determination criterion in which the eye movement target includes moving the pilot's line of sight to the periphery of the aircraft 1000, which is the target of piloting. If the pilot pilots the aircraft 1000 by staring at the aircraft 1000 without moving his or her line of sight to the periphery of the aircraft 1000, which is the target of piloting, it becomes difficult for the pilot to grasp the relative distance between the aircraft 1000 and the ground or surrounding objects, and it becomes difficult to move the aircraft 1000 along the flight path, which is the movement target. Therefore, by setting the eye movement target to move the pilot's line of sight to the periphery of the aircraft 1000, it becomes possible to more appropriately evaluate and determine the pilot's piloting skills, determine whether the pilot has passed the piloting skill certification test, determine the cause of a low piloting skill evaluation, and determine areas for improvement in the piloting skills.

[0106] Furthermore, the piloting skill evaluation unit 6200 can perform various determinations using a determination criterion in which the eye movement target includes shifting the gaze to a movement target point ahead in the direction of movement of the piloted aircraft 1000 when the aircraft is moving or accelerating. Alternatively, the piloting skill evaluation unit 6200 can perform various determinations using a determination criterion in which the eye movement target includes shifting the gaze diagonally downward and ahead in the direction of movement of the aircraft 1000 when the aircraft is moving or accelerating. When moving or accelerating the aircraft 1000, it is important to check ahead in the direction of movement and recognize obstacles and the distance to the movement target position, so it is desirable to shift the gaze to a movement target point ahead in the direction of movement. Furthermore, during movement, it is important to pilot the aircraft 1000 while understanding the relative positions of the aircraft 1000, the movement target position, and the ground. Therefore, it is desirable to shift the gaze diagonally downward and ahead in the direction of movement of the aircraft 1000 and understand the relative positions of the aircraft 1000, the movement target position, and the ground through indirect vision.

[0107] (A-1-5-3. Training proposal generation unit 6300) The training proposal generation unit 6300 is a functional unit that generates proposal information related to at least one of the content and schedule of future pilot training or certification testing. The training proposal generation unit 6300 includes a training method proposal unit 6310 and a test schedule proposal unit 6320.

[0108] The training method suggestion unit 6310 is a functional unit that suggests training items, number of training sessions, training schedules, training methods, etc. based on various judgment results by the piloting skill evaluation unit 6200, or based on the integrated video data acquired by the integrated data acquisition unit 6130, or various other information.

[0109] For example, the skill evaluation unit 6210 can suggest, as training items, movements of the aircraft 1000, the pilot's finger movements, the pilot's eye movements, and the posture of holding the control device that have been given low evaluation ranks or scores relative to the evaluation criteria. Furthermore, the number of training sessions can be suggested as the number of training sessions required for the current skill evaluation level to reach the evaluation criteria. Furthermore, the training schedule can be suggested as the training time required for the current skill evaluation level to reach the evaluation criteria, or a training schedule that takes into account the required training time. Furthermore, the skill evaluation unit 6210 can suggest, as training methods, training methods for improving the movements of the aircraft 1000, the pilot's finger movements, the pilot's eye movements, and the posture of holding the control device that have been given low evaluation ranks or scores relative to the evaluation criteria.

[0110] The test schedule suggestion unit 6320 is a functional unit that suggests a date or schedule for a re-examination of the skill certification test based on the various assessment results by the piloting skill evaluation unit 6200, or based on the integrated video data acquired by the integrated data acquisition unit 6130, or various other information. The information suggested by the test schedule suggestion unit 6320 is useful suggested information for pilots and training instructors, and is particularly generated as suggested information for training instructors to determine or advise on the date of the next re-examination.

[0111] For example, if the certification test pass / fail determination unit 6220 determines that the cause of a pilot who failed the certification test was excessive tension on the part of the pilot and that the piloting skill itself was not poorly evaluated, the test schedule proposal unit 6320 proposes a retest schedule for a retest to be conducted on a date not too far away.On the other hand, if the certification test pass / fail determination unit 6220 determines that the cause of a pilot who failed the certification test was insufficient piloting skill on the part of the pilot, the test schedule proposal unit 6320 proposes a retest schedule for a retest to be conducted on a date with a sufficient period of time left for training to improve piloting skill.

[0112] (A-1-5-4. Information output unit 6400) The information output unit 6400 is a functional unit that transmits or displays the evaluation result by the piloting skill evaluation unit 6200 and the proposal information generated by the training proposal generation unit 6300 to the training support system 3000 or the like. The information output unit 6400 includes a skill evaluation output unit 6410 and a training proposal output unit 6420.

[0113] The skill evaluation output unit 6410 is a functional unit that transmits or displays the evaluation result by the piloting skill evaluation unit 6200 to the training support system 3000, etc. The training suggestion output unit 6420 is a functional unit that transmits or displays the proposal information generated by the training suggestion generation unit 6300 to the training support system 3000, etc.

[0114] (A-1-5-5. Data recording unit 6500) The data recording unit 6500 is a functional unit that records the evaluation results by the piloting skill evaluation unit 6200 and the proposal information generated by the training proposal generation unit 6300. The data recording unit 6500 also has a function of recording information acquired by the integrated data acquisition unit 6130 of the information acquisition unit 6100. When the data analysis system 6000 is implemented on the cloud, pilots and instructors can download and review the evaluation results, proposal information, integrated video data, etc. stored in the data recording unit 6500 on the cloud when necessary.

[0115] (A-1-6. Hardware Configuration) Next, a hardware configuration diagram of the training support system 3000, the data analysis system 6000, etc. will be described. Here, the training support system 3000 and the data analysis system 6000 constituting the information control system 1 of the present invention are information processing devices such as PCs. The data analysis system 6000 can also be configured as a server device. The training support system 3000 and the data analysis system 6000 have an input device 100, an output device 200, a processing device 300, a main memory device 400, an auxiliary memory device 500, a communication device 600, and a bus 700 that electrically connects these devices.

[0116] The input device 100 is a device that allows a user such as a pilot or a training instructor to input information and instructions to the information control system 1. Specifically, the input device 100 is, for example, a touch panel, a keyboard, a mouse, or an audio input device such as a microphone.

[0117] The output device 200 is a device that outputs information generated by the information control system 1. Specifically, the output device 200 is a display unit 2221 (including eyewear, AR, VR display devices, etc.), a printer, or a speaker.

[0118] The processing device 300 is, for example, a device that performs arithmetic processing. Specifically, the processing device 300 is, for example, a CPU, a microprocessor, a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array), or other semiconductor devices capable of performing calculations.

[0119] The main storage device 400 is a memory device such as a RAM that temporarily stores various read information and a ROM that stores programs executed by the processing device 300, application programs, and various other information. The auxiliary storage device 500 is a non-volatile storage device such as an HDD (Hard Disk Drive), SSD (Solid State Drive), or flash memory that can store digital information. The communication device 600 is a device that performs information communication with external devices wirelessly or via a cable.

[0120] Some of the functions of the above-mentioned data analysis system 6000 can be implemented in the training support system 3000, and some of the functions of the piloted object information acquisition system 4000 and the pilot information acquisition system 5000 can be implemented in the training support system 3000.

[0121] (A-1-7. Control flow of information control system 1) Next, a description will be given of the overall control flow of the information control system 1. FIG.

[0122] First, the pre-registration information receiving unit 3100 receives pre-registration information (step 101). In this step, for example, pre-registration information such as that shown in FIG.

[0123] Next, when the training instructor and pilot have completed preparations for the practical flight training, an input to start flight is accepted (step 102).

[0124] Next, the information import unit 3200 acquires piloting-related data during the piloting practical training (step 103). In this step, piloting commands, aircraft status, external environment status, surrounding video data, video data of the subject being photographed, pilot's finger movement video data, eye movement video data, gaze estimation information, posture video data, and living body detection information are acquired.

[0125] Next, the integrated video data is generated by the integrated data generation unit 3300 (step 104). In this step, at least a plurality of pieces of data acquired by the information import unit 3200 are integrated. Since it is required to review the flight practice using the integrated video data as soon as possible after the flight practice is completed, it is desirable that the process of generating the integrated video data in this step be performed in real time during the flight practice. However, the process of generating the integrated video data may also be performed after the flight practice is completed.

[0126] Next, the integrated video data is displayed and output by the integrated data display output unit 3400 (step 105). The display and output of the integrated video data in this step is performed immediately after the process of generating the integrated video data is completed.

[0127] Next, the piloting skill evaluation unit 6200 of the data analysis system 6000 evaluates and determines the piloting skill of the pilot (step 106).

[0128] Next, the training proposal generator 6300 of the data analysis system 6000 generates proposal information regarding at least one of the content and schedule of training or certification testing for future pilots (step 107).

[0129] Next, the data analysis result output unit 3600 displays and outputs or records the evaluation results of the pilot's piloting skills and the details and schedule of future pilot training or certification tests (step 108).

[0130] (A-1-8. Example of integrated video data display) Next, a description will be given of the integrated video data generated by the integrated data generating unit 3300. Fig. 9 is a diagram showing an example of the integrated video data generated by the integrated data generating unit 3300.

[0131] In the example shown in FIG. 9, (1) image data of the object to be photographed, (2) surrounding image data, (3) image data of the operator's finger movement, and (4) biometric detection information are integrated into a single piece of video data as integrated video data. In particular, (1) image data of the object to be photographed is video captured of a multicopter (air vehicle 1000) flying indoors. Furthermore, (2) surrounding image data is video captured of the ground directly below from the air vehicle 1000. Furthermore, (3) image data of the operator's finger movement is video captured of the operator's operation, including finger movement, using his or her thumb to operate the control stick of the control device 2000. Furthermore, (4) biometric detection information is heart rate information acquired by a sensor attached to the operator's body.

[0132] In the example shown in Fig. 9, (1) the image data of the subject to be photographed is displayed as the main image, occupying almost the entire display area, and (2) the peripheral image data, (3) the operator's finger movement image data, and (4) the living body detection information are generated as sub-images, which are displayed in a relatively small partial area at the bottom of the display area. Note that the image displayed in the wide screen area as the main image does not necessarily have to be (1) the image data of the subject to be photographed; (2) the peripheral image data, (3) the operator's finger movement image data, (4) the living body detection information, and other information may be displayed large as the main image, occupying almost the entire display screen. Furthermore, instead of providing a difference in size between the main image and the sub-image, each piece of integrated information may be displayed at approximately the same size.

[0133] (A-1-9. Output processing of integrated video data) Next, a description will be given of the integrated video data output process executed by the integrated data display output unit 3400. Fig. 10 is a flowchart showing an example of the integrated video data output process flow executed by the integrated data display output unit 3400.

[0134] First, the display command receiving unit 3420 receives a display command for the integrated video data input by the operator or the training instructor (step 201).

[0135] Next, the display control unit 3430 displays the integrated video data in response to a display command (step 202).

[0136] Next, the tag information assigning unit 3440 accepts input of tag information input by the pilot or training instructor (step 203).

[0137] Next, the tag information adding unit 3440 associates the tag information with the integrated video data and registers it (step 204).

[0138] (A-1-10. Integrated video data display command and tag information reception process) Next, a method for receiving a display command for integrated video data and tag information will be described. Fig. 11 is a diagram showing an example of a method for receiving a display command for integrated video data and tag information by integrated data display output unit 3400.

[0139] In the example shown in Fig. 11, the integrated video data is displayed on the display screen of the display device, and display command input icons for instructing the display output mode of the integrated video data, such as play, pause, stop, rewind, and fast forward, are displayed at the top of the screen. Although not shown, the display command input icons may also include slow play. By tapping these icons, the pilot or instructor can display the integrated video data in the intended mode.

[0140] By playing back the generated integrated video data immediately after the piloting demonstration, the pilot and instructor can review the piloting demonstration with their memories fresh. The integrated video data also integrates (1) the operational behavior of the aircraft 1000, (2) the video data of the target object, and (3) the video data of the pilot's finger movements. This allows the pilot and instructor to easily compare the aircraft's movements and the operational inputs, making it easier to identify piloting flaws and areas for improvement. The integrated video data shown in FIG. 11 also displays (2) the position and attitude of the aircraft 1000 relative to the ground, (3) the peripheral video data, and (4) the pilot's heart rate. Therefore, the pilot and instructor can (2) understand the aircraft's behavior in more detail using the peripheral video data, and can also understand the pilot's level of tension and other conditions, making it easier to identify piloting flaws and areas for improvement.

[0141] In addition, an icon for inputting tag information is displayed in the upper right corner of the screen, and by tapping this icon, pilots and instructors can register any tag information to the integrated video data. Tag information can be, for example, information about the results of the pilot's or instructor's review of the pilot's practical flying skills, or information about piloting shortcomings and areas for improvement.

[0142] (A-1-11. Display output of piloting skill evaluation results) Next, the piloting skill evaluation results generated by the piloting skill evaluation unit 6200 and displayed and output by the piloting skill evaluation output unit 3610 will be described with reference to FIGS.

[0143] FIG. 12 is a diagram showing an example of a piloting skill evaluation result displayed and output by the piloting skill evaluation output unit 3610. The example shown in FIG. 12 shows an example in which the evaluation score (75 points), which is the skill evaluation result information generated by the piloting skill evaluation unit 6200, and an improvement point ("The flight path is deviating from the target path") are displayed together with integrated video data. The integrated video data displayed in FIG. 12 includes (1) the image data of the subject of shooting displayed as the main image, occupying almost the entire display area, and (2) the peripheral image data, (3) the pilot's finger movement image data, and (4) the biological detection information displayed as sub-images in a relatively small area below the display area. In addition, in the example shown in this figure, the target path of the aircraft (solid line in the figure), which is the evaluation standard for the piloting skill, and the time-series movement record of the aircraft 1000 are superimposed and displayed.

[0144] In this way, along with the evaluation results of piloting skills, the target movement path of the aircraft (solid line in the figure), which is the information used in the evaluation judgment, and the time-series movement actual movement of the aircraft 1000 are displayed, and the difference between the target value and the piloting actual performance can be confirmed.Therefore, pilots and instructors who view this displayed information can more accurately grasp the points of piloting that need to be improved and the amount of piloting correction required, thereby enabling more effective piloting training.

[0145] FIG. 13 is a diagram showing another example of a piloting skill evaluation result displayed and output by the piloting skill evaluation output unit 3610. The example shown in FIG. 13 shows an example in which the skill evaluation result (79 points) generated by the piloting skill evaluation unit 6200 and an improvement point ("The difference between the operation input and the target input value is too large.") are displayed together with integrated video data. The integrated video data displayed in FIG. 13 includes (3) the pilot's finger movement video data displayed as the main video, occupying almost the entire display area, and (1) the captured image data, (2) the surrounding video data, and (4) the biological detection information displayed as sub-videos in a relatively small area below the display area. In the example shown in FIG. 13, the target position of the tip of the operating stick of the piloting device 2000 generated from the target information of the piloting command (dotted circle in the figure) and the position of the tip of the operating stick generated based on the operation command actually input by the pilot (solid circle in the figure) are superimposed on each other. The position of the tip of the operating stick may be generated by interpreting the pilot's finger movement video data.

[0146] In this way, along with the evaluation results of piloting skills, the target values ​​of the piloting commands (dotted circles in the figure), which are the information used in the evaluation judgment, and the piloting commands actually input (solid circles in the figure) are displayed, allowing the difference between the target values ​​and the piloting performance to be confirmed.This allows pilots and instructors who view this displayed information to more accurately grasp the points of piloting that need improvement and the amount of piloting correction required, thereby enabling more effective piloting training.

[0147] Figure 12 shows an example of a display output of an evaluation result based on the difference between the target value and the actual value of the drone's movement path, and Figure 13 shows an example of a display output of an evaluation result based on the difference between the target value and the actual value of the control command from the control device 2000.In addition to these, evaluation results based on the difference between the target movement and the actual value in finger movement, eye movement, or posture can also be displayed in a similar manner.

[0148] Furthermore, on the display screen of the piloting skill evaluation results as shown in FIG. 12 or 13, it is also possible to highlight items with shortcomings or areas for improvement. That is, for highly rated items with a small difference from the target value, integrated image data is displayed, and for shortcomings or areas for improvement with a large difference from the target value, the integrated image can be highlighted. Furthermore, in FIG. 12 or 13, areas for improvement are displayed along with the evaluation score, but in addition to this, improvement methods (such as advice on finger movements, eye movements, or posture) may also be displayed. Furthermore, on the display screen of the piloting skill evaluation results as shown in FIG. 12 or 13, it is also possible to highlight items whose relative evaluation has changed from the past piloting history. That is, it is also possible to highlight items whose piloting skill level has significantly improved or significantly deteriorated from the past piloting history.

[0149] The above-described embodiments are merely examples for facilitating understanding of the present invention, and are not intended to limit the present invention. The present invention can be modified and improved without departing from the spirit thereof, and it goes without saying that the present invention includes equivalents thereof.

[0150] [A-2. Effects of this embodiment] The above-described embodiment can provide a system or method that can more effectively or appropriately train, evaluate, and certify pilots' piloting skills. As an example, integrated video data is generated by integrating video of the piloted aircraft 1000 with other video of the pilot performing inputs with his / her fingers and the pilot's eye movements, allowing the pilot to immediately review his / her flight performance after piloting training.

[0151] Furthermore, the accuracy of piloting skill evaluation can be improved by evaluating piloting skills by taking into consideration not only the operational performance of the aircraft and the performance information of the piloting commands, but also the pilot's finger movements, eye movements, and other pilot-related conditions.Furthermore, by providing instructors and pilots with information useful for improving skills, such as shortcomings in the piloting operations, their causes, areas for improvement, or future training methods, by taking into consideration not only the operational performance of the aircraft and the performance information of the piloting commands, but also the pilot's finger movements, eye movements, and other pilot-related conditions, the training, evaluation, and certification of piloting skills can be carried out more effectively or appropriately. [Explanation of symbols]

[0152] 1...Information control system (system) 100...input device 200...output device 300...Processing device 400...Main storage device 500...Auxiliary storage device 600...Communication device 700...bus 1000...Flying object 1100: Measurement section 1200: Self-status determination section 1300...Flight section 1310...Thrust generation section 1320...Flight control unit 1400...Communication unit 2000...Control device 2100...Display section 2200...Operation section 2210...Right control stick 2220...Left control stick 2230…Communication antenna 3000...Training support system 3100: Pre-registration information reception unit 3110: Integrated information registration unit 3120: Evaluation criteria registration unit 3130: Proposal criteria registration unit 3200: Information import unit 3210: Control device information acquisition unit 3220: Control target information acquisition unit 3230: Operator information acquisition unit 3300...Integrated data generation unit 3310...Data integration unit 3320...Integrated Data Recording Unit 3400...Integrated data display output unit 3410...Real-time output unit 3420: Display command receiving unit 3430: Display control unit 3440...Tag information assignment section 3500...Data analysis request command unit 3510...Pilot skill evaluation request unit 3520...Training proposal request section 3600: Data analysis result output unit 3610: Pilot skill evaluation output unit 3620: Pilot skill evaluation recording unit 3630: Training proposal output unit 3640…Training Proposal Recording Department 4000...Pilot information acquisition system 5000...Pilot information acquisition system 6000...Data analysis system 6100: Information acquisition unit 6110: Pilot skill evaluation request unit 6120...Training proposal request unit 6130...Integrated data acquisition unit 6200: Pilot Skills Evaluation Department 6210: Skills Evaluation and Judgment Department 6220: Certification test pass / fail judgment section 6230: Defect cause judgment section 6240…Improvement method proposal department 6300…Training proposal generation unit 6310…Training method proposal unit 6320...Exam Schedule Proposal Department 6400...information output unit 6410...skill evaluation output unit 6420...Training suggestion output unit 6500...Data recording unit

Claims

1. An information and control system that can be used to support training, evaluation, or certification of pilot skills of pilots who pilot objects including aircraft or other mobile objects, an information import unit that acquires first video data capturing a state of the control object being controlled by the operator, and second video data capturing a state of a control device that controls the control object being operated by the operator; an integrated data generating unit that generates or records integrated video data that integrates the first video data and the second video data; An information control system comprising a display output unit that displays and outputs the integrated video data during or after the operation of the object to be operated.

2. 2. The information control system according to claim 1, a display command receiving unit that receives a display command including at least one of playback, slow playback, pause, fast forward, and rewind from the operator, training instructor, or other user; The display output unit controls a display output mode of the integrated video data in response to the display command.

3. 2. The information control system according to claim 1, An information control system, wherein the integrated video data generated by the integrated data generation unit is video data in which the first video data and the second video data are integrated into a single video data, and the first video data is displayed in at least a portion of a display area and the second video data is displayed in at least another portion of a display area.

4. 2. The information control system according to claim 1, the information import unit acquires third video data capturing an image of the eyes or line of sight of the pilot or a posture of the pilot while piloting the object to be piloted, An information control system, wherein the integrated video data generated by the integrated data generation unit is video data that integrates the first video data, the second video data, and the third video data.

5. 2. The information control system according to claim 1, the information import unit acquires fourth video data of the surroundings of the target object captured by a mobile camera mounted on the target object; An information control system, wherein the integrated video data generated by the integrated data generation unit is video data that integrates the first video data, the second video data, and the fourth video data.

6. 2. The information control system according to claim 1, the information import unit acquires control target information relating to the control target, including at least one of the position, movement path, speed, acceleration, attitude angle, angular velocity, angular acceleration, remaining battery charge, control communication radio wave strength, positioning signal strength, wind direction or wind speed around the control target, and control commands input to the control device by the operator; The integrated video data generated by the integrated data generation unit is video data in which the control target information is integrated in addition to the first video data and the second video data.

7. 2. The information control system according to claim 1, the information import unit acquires pilot status information regarding the pilot, the pilot including at least one of a gaze direction or a focus position, a heart rate, a body temperature, a blood pressure, and sweating of the pilot while piloting the pilot object; An information control system, wherein the integrated video data generated by the integrated data generation unit is video data that integrates the first video data, the second video data, and the pilot status information.

8. 2. The information control system according to claim 1, An information control system including a tag information assigning unit that records tag information for the integrated video data received from the pilot, training instructor, or other user in association with the integrated video data.

9. 2. The information control system according to claim 1, An information control system comprising a piloting skill evaluation unit that performs at least one of the following: evaluating and judging the pilot's piloting skills; determining whether the pilot meets the passing criteria for the piloting skill certification test; determining the causes of shortcomings in the piloting skills; and determining areas for improvement in the piloting skills.

10. 10. The information control system according to claim 9, The piloting skill evaluation unit performs at least one of an evaluation and judgment of the pilot's piloting skills, a pass / fail judgment of whether the pilot meets the passing criteria for a piloting skill certification test, a judgment of the causes of shortcomings in the piloting skills, and a judgment of areas for improvement in the piloting skills, based on status information of the piloted object including at least one of the position, movement path, speed, acceleration, attitude angle, angular velocity, and angular acceleration of the piloted object, or piloting commands input to the piloting device by the pilot.

11. 10. The information control system according to claim 9, The piloting skill evaluation unit performs at least one of the following based on the operational input operations of the control device captured as the second video data: an evaluation of the pilot's piloting skills, a pass / fail determination of whether the pilot meets the passing criteria for the piloting skill certification test, a determination of the cause of shortcomings in the piloting skills, and a determination of areas for improvement in the piloting skills.

12. The information control system according to claim 11, An information control system in which the criteria used for the judgment performed by the piloting skill evaluation unit include not releasing the pilot's finger from the operating stick of the piloting device, or not releasing it for more than a predetermined period of time.

13. The information control system according to claim 11, An information control system in which operating the control stick of the control device with the area around the tip of the pilot's finger is included in the judgment criteria used for the judgment performed by the piloting skill evaluation unit.

14. 10. The information control system according to claim 9, The piloting skill evaluation unit performs at least one of an evaluation and judgment of the pilot's piloting skills, a pass / fail judgment of whether the pilot meets the passing criteria for a piloting skills certification test, a judgment of the cause of shortcomings in the piloting skills, and a judgment of areas for improvement in the piloting skills, based on third video data that captures the direction or focal position of the pilot's line of sight while piloting the piloted object, or the state of the pilot's eyes or line of sight or the pilot's posture while piloting the piloted object.

15. 15. The information control system according to claim 14, An information control system in which moving the pilot's line of sight to the periphery of the object to be piloted is included in the criteria used for the judgment performed by the piloting skill evaluation unit.

16. 2. The information control system according to claim 1, An information control system comprising a training proposal generator that generates proposal information regarding at least one of the content and schedule of future training or certification testing for the pilot.

17. 10. The information control system according to claim 9, An information control system comprising a piloting skill evaluation recording unit that records, in association with the integrated video data, at least one of the judgment information of the pilot's piloting skills performed by the piloting skill evaluation unit, a pass / fail judgment of whether the piloting skills meet the passing criteria for the piloting skill certification test, a judgment of the cause of shortcomings in the piloting skills, and a judgment of areas for improvement in the piloting skills.

18. 10. The information control system according to claim 9, The display output unit displays and outputs at least one of the judgment information, which is an evaluation judgment of the pilot's piloting skills performed by the piloting skill evaluation unit, a pass / fail judgment of whether the piloting skills meet the passing criteria for the piloting skills certification test, a judgment of the cause of shortcomings in the piloting skills, and a judgment of areas for improvement in the piloting skills, in association with the integrated video data.

19. 17. The information control system according to claim 16, An information control system comprising a proposal information recording unit that records the proposal information generated by the training proposal generating unit in association with the integrated video data.

20. 17. The information control system according to claim 16, The display output unit displays and outputs the proposal information generated by the training proposal generation unit in association with the integrated video data.

21. An information control method that can be used to support training, evaluation, or certification of pilot skills of pilots who pilot piloting objects including aircraft or other moving objects, The computer an information acquisition step of acquiring first video data capturing a state of the control object being controlled by the operator and second video data capturing a state of the control device controlling the control object being operated and input by the operator; an integrated data generating step of generating or recording integrated video data by integrating the first video data and the second video data; and a display output step of displaying and outputting the integrated video data during or after the operation of the object to be operated.

22. A program that can be used to train, evaluate, or assist in certifying the piloting skills of a pilot who pilots an object, including an aircraft or other mobile object, On the computer, an information acquisition command to acquire first video data capturing a state of the control object being controlled by the operator, and second video data capturing a state of a control device for controlling the control object being operated and input by the operator; an integrated data generation command for generating or recording integrated video data that integrates the first video data and the second video data; A program that executes a display output command to display and output the integrated video data during or after the operation of the object to be operated.

Images