Mobile object piloting support method and mobile object piloting support system

Abstract

To enable an operator to confirm reference information such as flight items necessary for steering without greatly moving a visual line.SOLUTION: A steering assistance system comprises: a display unit (6) that displays image information in front of eyes of a user who steers a movable body; an image generation unit (40, 53) that collects image information about an environment of the movable body and generates an environment image on the basis of the information; a symbol information generation unit (52) that generates symbol information simplifying or imitating reference information associated with steering of the moving movable body; a state detection unit (1, 2, 7) that detects the state of the user, the state of the movable body, or the surrounding state of the movable body; and a display image forming unit (54) that forms image information displayed on the display unit with the symbol information superposed on the environment image so as to estimate a part to which the user directs the visual line on the screen of the display unit or within a displayable range on the basis of the detection result of the state detection unit and display one or both of the image information and the symbol information on the estimated part.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to a piloting assistance method and a piloting assistance system for various types of moving bodies such as aircraft. [Background technology]

[0002] In many aircraft such as helicopters and airplanes, image display devices such as a Head Up Display (HUD), a Head Mount Display (Helmet Mount Display: HMD), and a Head Down Display (HDD) are used by pilots to fly the aircraft.

[0003] In such an image display device for an aircraft, symbols indicating various flight information such as flight parameters are superimposed on an image of the outside world obtained by various sensors such as cameras such as a visible light camera, an infrared camera, and a night vision camera, an image sensor, and a range sensor. For example, by displaying an image of the outside world obtained by an infrared camera or the like on the image display device, it is possible to support the pilot in flying even when the pilot's naked eye visibility is poor due to nighttime or bad weather conditions. In addition, such an image display device is also effective in searching for abnormal locations, victims, and remains during disasters and accidents.

[0004] Conventionally, there is known an image display device for aircraft that displays a symbol (mark) indicating the aircraft's current position and the aircraft's traveling direction by superimposing the external image captured by a camera or the like (see Patent Documents 1 and 2, etc.). The former is, for example, a W-shaped or V-shaped mark, and is used to indicate the current direction of the aircraft's nose. The latter is a flight path vector (FPV, also called a flight path marker or velocity vector), which is a symbol that shows the position that the aircraft will reach after a predetermined time has elapsed if it continues flying as it is.

[0005] Usually, in HUDs and HMDs, the FPV is displayed near the center of the display screen when flying at approximately the same altitude so that the pilot can obtain the information necessary for operation without moving his / her eyes much, that is, while gazing forward in front of him / her (in front of the direction he / she is facing). In addition, in HMDs, image information obtained by cameras and sensors is also displayed near the center of the displayable range. However, pilots do not always gaze near the center of the display screen, and depending on the situation, they may gaze at a position away from the center on the display screen or within the displayable range. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2003-130677 A [Patent Document 2] JP 2012-104107 A Summary of the Invention [Problem to be solved by the invention]

[0007] For example, when visibility outside the window or in the image captured by the camera or sensor displayed on the display screen is poor, the pilot will turn his or her gaze downward within the display screen in an attempt to grasp the situation on the ground. This makes it difficult for the pilot to see the FPV displayed near the center of the display screen and information such as speed and altitude displayed nearby, and the pilot will tend to neglect to grasp such information. In addition, when using an HMD, if the pilot lowers his or her head (tilts forward) in an attempt to grasp the situation on the ground, the camera's shooting range moves forward accordingly. As a result, the FPV disappears from the display screen, making it difficult for the pilot to intuitively grasp the course of the aircraft.

[0008] Furthermore, when making a large turn to the left or right, the pilot often looks further ahead in the direction of the turn on the display screen, that is, at a position further to the side. When using an HMD, the camera image and key symbol information are displayed near the center of the pilot's field of vision, but the pilot's gaze is directed further ahead in the direction of the turn. Therefore, in order to check the captured image and key symbol information, the pilot must intentionally move his or her gaze or turn his or her head. This places a heavy burden on the pilot to grasp the information necessary for piloting, and the pilot's understanding of the information tends to be delayed.

[0009] The same problem is not limited to aircraft operated by a pilot on board, but also applies to remotely controlled unmanned aircraft and other flying objects (so-called drones, etc.).The same problem also applies to various moving objects other than flying objects and flying objects, such as submarines and submarines for exploring the ocean floor.

[0010] The present invention has been made to solve these problems, and its purpose is to provide a method and system for piloting a moving body that enables a pilot to easily visually obtain photographed images of the outside of the moving body and information necessary for piloting, without having to forcefully move his or her eyes or turn his or her head. [Means for solving the problem]

[0011] One aspect of a mobile object steering assistance method according to the present invention, which has been made to solve the above problems, is a mobile object steering assistance method that displays information for steering on a display unit capable of displaying image information arranged in front of a user who is steering a mobile object, comprising: an image creation step of collecting image information of the outside world of the moving body and creating an outside world image based on the information; a symbol information generating step of generating symbol information that simplifies or models reference information related to the operation of the moving object that is moving; a display execution step of displaying the symbol information on the display unit by superimposing the symbol information on the outside world image so that either or both of the outside world image and the symbol information are located at a position on the screen of the display unit or within a displayable range where the user is expected to direct his or her gaze; It has the following characteristics.

[0012] Here, "areas on the screen of the display unit or within the displayable range where the user is expected to direct his or her gaze" refers to areas excluding areas where image information and symbol information are typically displayed, that is, areas near the center of the screen of the display unit or within the displayable range.

[0013] In addition, the "area where the user is expected to direct his or her gaze" referred to here may be an area where the user is predicted to direct his or her gaze based on the results of detecting or understanding the user's movements at that time, the status of the moving object, the surrounding (environmental) status, etc., or it may be a specified area where it is predicted in advance (for example, at the system design stage, or at various setting stages before using the system, etc.) that the user will direct his or her gaze.

[0014] In the former case, the position of the "area where the user is expected to direct their gaze" changes depending on the results of detecting or understanding the events or situations described above, whereas in the latter case, the position of the "area where the user is expected to direct their gaze" does not change because it is determined in advance.

[0015] Additionally, the term "mobile body" as used herein includes flying bodies and aircraft, including manned and unmanned aircraft, as well as submarines and submarines that move underwater or in the sea.

[0016] In order to solve the above problems, one aspect of the vehicle driving assistance system according to the present invention is to A display unit that displays image information in front of the eyes of a user who operates the moving object; an image creation unit that collects image information of the outside world of the moving body and creates an outside world image based on the information; a symbol information generating unit that generates symbol information that simplifies or models reference information related to the operation of the moving object while the moving object is moving; a display image forming unit which forms image information to be displayed on the display unit by superimposing the symbol information on the outside world image, the display image forming unit forming the image information so that the symbol information is displayed in a portion of the outside world image displayed on the screen or within a displayable range of the display unit where the user is expected to direct his or her gaze; It is equipped with the following.

[0017] Another aspect of the vehicle operation assistance system according to the present invention, which is made to solve the above problems, is as follows: A display unit that displays image information in front of the eyes of a user who operates the moving object; an image creation unit that collects image information of the outside world of the moving body and creates an outside world image based on the information; a symbol information generating unit that generates symbol information that simplifies or models reference information related to the operation of the moving object while the moving object is moving; A situation detection unit that detects a situation of the user, a situation of the moving object, or a situation around the moving object; a display image forming unit which forms image information to be displayed on the display unit by superimposing the symbol information on the external image, and which estimates a portion on the screen or within a displayable range of the display unit to which the user will direct his or her gaze based on a detection result by the situation detecting unit, and forms the image information so that either the image information or the symbol information, or both, are displayed in the estimated portion; It is equipped with the following. Effect of the Invention

[0018] In a conventional piloting support system for a general aircraft, various symbol information is concentrated near the center of the display screen under the assumption that the pilot (user) stares at the center of the display screen during piloting. In contrast, in a piloting support method for a mobile body and a piloting support system for a mobile body, which are one aspect of the present invention, symbol information can be displayed in a part other than the center of the display screen or the displayable range where the user is expected to direct his / her gaze. For example, even when an aircraft is flying while maintaining its altitude, if the pilot directs his / her gaze toward the ground on the near side rather than forward in the direction of travel, symbol information is superimposed and displayed on the corresponding part of the ground in the external world image. Also, when the aircraft is turning widely, symbol information can be displayed together with the external world image further forward in the turning direction than the direction the pilot is facing within the displayable range of the display unit. That is, symbol information to be referred to for piloting is displayed in the direction the pilot directs his / her gaze to perform safe piloting.

[0019] In this way, according to the mobile object piloting assistance method and mobile object piloting assistance system, which are one aspect of the present invention, a user such as a pilot can visually and easily obtain captured images of the outside of the mobile object and information required for piloting without forcing the pilot to move his / her eyes or turn his / her head. This reduces the burden on the pilot during piloting and contributes to safe piloting by avoiding delays in the pilot's understanding of information required for piloting. [Brief description of the drawings]

[0020] [Figure 1] 1 is a schematic block diagram of an aircraft piloting assistance system according to a first embodiment of the present invention; [Diagram 2] Schematic diagram of a helicopter flight situation. [Diagram 3] Schematic diagram of FPV display state on the display screen. [Figure 4] FIG. 2 is an explanatory diagram of the flight path of a helicopter using the aircraft piloting assistance system of the first embodiment when flying at a constant altitude, and a schematic diagram of the display state of the GPV on the display screen at that time. [Diagram 5]FIG. 4 is a schematic block diagram of an aircraft piloting assistance system according to a second embodiment of the present invention. [Figure 6] 1A to 1C are schematic diagrams of display screens of aircraft flight control assistance systems according to a conventional example and a second embodiment of the present invention, and are explanatory diagrams of the pilot and camera pointing directions. [Figure 7] FIG. 6 is an explanatory diagram of a bank angle correction method in the aircraft flight control assistance system according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] [First embodiment] An aircraft piloting assistance system according to an embodiment of the present invention will be described with reference to the accompanying drawings. The aircraft piloting assistance system of the first embodiment is primarily intended to assist a pilot on board a helicopter when piloting the helicopter, or to assist in searching for a person or a specific object on the ground. In other words, the "moving body" in this embodiment is a helicopter.

[0022] Fig. 1 is a schematic block diagram of an aircraft flight support system according to the present embodiment, Fig. 2 is a schematic diagram of a flight situation of a helicopter. 1, this piloting support system includes functional blocks such as a navigation sensor 1, an aircraft attitude sensor 2, a terrain database 3, a sensor pod 4 including a camera 40, a data processing unit 5, and a display unit 6. The data processing unit 5 includes lower functional blocks such as a course calculation unit 50, a predicted altitude calculation unit 51, a GPV symbol generation unit 52, an image creation unit 53, and a display processing unit 54.

[0023] The display unit 6 is an HMD integrated with a helmet worn by a pilot, and includes a display element such as a transmissive liquid crystal display element, a visor placed in front of the pilot's eyes, and an illumination optical system placed between the display element and the visor. An image input to the display unit 6 is displayed on the transmissive liquid crystal display element, and the image light emitted from the display element passes through the illumination optical system, is reflected by the visor, and reaches the pilot's eyes as parallel light. The visor transmits external light coming from the outside world in the direction in which the pilot is facing. Therefore, in front of the pilot's eyes, a background image based on the external light, which is approximately parallel light, and a display image based on the reflected light on the visor are displayed in a superimposed manner. The displayable range of this HMD is wide in the horizontal direction, but usually, the display image based on the reflected light is displayed only in a predetermined range near the center of the displayable range, and only the background image is visible outside the range. The display unit 6 is not limited to an HMD, and may be a HUD or HDD.

[0024] As shown in Fig. 2, the sensor pod 4 is provided at the bottom of the body of the helicopter 10, and the camera 40 is installed in the sensor pod 4. The camera 40 is, for example, an infrared camera including a sensor capable of detecting any one of near infrared, mid infrared, and far infrared, or a plurality of different wavelength bands thereof, and acquires an infrared image of the outside (external world) of the helicopter 10. The sensor pod 4 has a function of adjusting the attitude in the azimuth (AZ) direction and the elevation (EL) direction. Note that instead of the infrared camera, a highly sensitive visible light camera, a night vision camera, or various sensors (e.g., a distance measuring sensor) capable of acquiring an external world image can be used.

[0025] In the piloting support system of this embodiment, the sensor pod 4 is used in any one of the following modes: a head direction tracking mode that operates according to the position, angle, and acceleration of the pilot's head so that the camera 40 is directed in the direction in which the pilot is presumed to be looking, a specific direction fixed mode that operates so that the camera is directed in a specified specific direction, and a scan mode that operates so that the direction of the camera is repeatedly changed over a predetermined range. The head direction tracking mode will be described in detail in the second embodiment.

[0026] The terrain database 3 includes terrain information for at least a predetermined range around the area currently being flown, and includes a transceiver that accesses an external digital map database to receive only the necessary data, or a storage device in which a portion of the data included in the digital map database is stored. In the former case, the database itself is provided outside the helicopter 10, for example in a host computer, and the transceiver collects the necessary information by communicating with the host computer. On the other hand, in the latter case, all the necessary information is stored in advance in a storage device provided inside the helicopter 10, and no communication with the outside is required during flight.

[0027] The navigation sensor 1 and aircraft attitude sensor 2 are GNSS / INS devices and gyro sensors that are generally installed on aircraft, and collect and output information such as the current position (latitude, longitude, altitude, direction of travel (azimuth), and left / right tilt angle (bank angle) of the helicopter 10 during flight.

[0028] Before describing the image display in the flight control support system of this embodiment, a flight path vector (FPV) displayed on an HMD in a conventional flight control support system for an aircraft will be described. Fig. 3 is a schematic diagram of the display state of the FPV. As shown in Figure 2, assume that helicopter 10 flying at a point at height H above ground 11 is at positions 10A and 10B after a predetermined time has elapsed. Position 10A is when helicopter 10 has risen above its current position, and position B is when helicopter 10 has descended below its current position. The route taken by helicopter 10 when it flies horizontally, that is, at a constant altitude, is shown by a two-dot chain line.

[0029] As shown in FIG. 3, the FPV 102 indicating the direction of travel (course) of the helicopter 10 and a V-shaped nose mark 100 are displayed in the approximate center of the display screen of the HMD. The dotted lines 101 on the left and right of the nose mark 100 indicate the current left-right attitude of the helicopter 10, and when the lines 101 are horizontal as shown in FIG. 3, the helicopter 10 is in a horizontal attitude in the left-right direction. The FPV 102 indicates the location where the helicopter 10 will be located after continuing the current flight state and a predetermined time has elapsed. Therefore, as shown in FIG. 3(a), the FPV 102 is located above the line 101 during ascent, and as shown in FIG. 3(b), the FPV 102 is located below the line 101 during descent. Xf in FIG. 3(a) indicates the horizontal deviation between the current position and the target position, and Yf ​​indicates the vertical deviation between the current position and the target position. For example, if the pilot wishes to fly straight while maintaining altitude, the pilot simply steers the aircraft so that the FPV 102 is roughly overlapped with the nose mark 100.

[0030] For example, when the visibility outside the window is poor due to weather conditions, the pilot tends to look downward in the display screen to visually check the situation on the ground in front of him, that is, the location of buildings, fields, rivers, etc., rather than looking forward in the direction of travel. In addition, when directing attention further downward, it is common to tilt the head forward rather than looking downward while keeping the face facing forward. As described above, the FPV 102 indicates the course of the aircraft, and is displayed near the center of the display screen when flying at an equal altitude. Therefore, if the pilot lowers his gaze to look at the ground in the external image displayed on the display screen, the FPV 102 becomes difficult to see, and it is necessary to raise his gaze to check the FPV 102. Also, as described above, when the pilot tilts his head forward, the camera 40 is linked to the movement of the head, so the external image is almost a photograph of the ground. Therefore, the FPV 102 indicating the course is no longer displayed on the screen, and the pilot needs to raise his head itself to check the FPV 102.

[0031] In contrast, in the piloting assistance system of this embodiment, instead of the FPV, a new mark named a Ground Path Vector (GPV) can be displayed on the display screen. FIG. 4 is an explanatory diagram of the flight path when the helicopter 10 flies at a constant altitude, and a schematic diagram of the display state of the GPV on the display screen at that time, (a) in the case of flight at a constant altitude above the target ground, and (b) in the case of flight at a constant altitude above sea level. As shown in the right of FIG. 3, the GPV 103 is a mark with a substantially trapezoidal shape, and its horizontal movement and position are defined the same as those of the FPV. On the other hand, its vertical movement and position are completely different from those of the FPV. The GPV 103 is displayed on the external world image displayed at that time so as to be attached to a position corresponding to the ground at a position a predetermined distance L [m] in a straight line ahead from the current position of the aircraft.

[0032] Therefore, when the camera 40 is directed generally horizontally forward, the GPV 103 moves downward on the screen as the altitude of the helicopter 10 increases, and moves upward on the screen as the altitude decreases. In either case, the GPV 103 is displayed on the ground in the external image at a position corresponding to the distance from the helicopter 10. Therefore, although the definition of the GPV is different from that of the FPV, the pilot can easily grasp the relative positional relationship between the ground surface or the terrain and the aircraft and the direction of the course from the positional relationship between the nose mark 100 or the line 101 and the GPV 103. However, the GPV itself does not have information on speed or altitude. Therefore, the speed deviation and altitude deviation are each shown as numerical values ​​near the GPV 103. Since the numerical information that the pilot wants to confirm differs between flight at a constant altitude above the ground and flight at a constant altitude above sea level, it is preferable to display appropriate numerical information according to each flight mode.

[0033] As mentioned above, the GPV103 is displayed as if it is attached to a specific position on the ground, so it will always be visible when the pilot looks at the image near the ground on the display screen due to poor visibility outside. Therefore, unlike conventional methods, the pilot does not need to raise his / her eyes or head to see the FPV, which reduces the burden on the pilot and reduces the chance of overlooking information. The distance L may be a fixed value, but may be desired to be changed depending on the flight speed, flight altitude, flight purpose, etc. Therefore, the distance L may be set by the pilot as appropriate.

[0034] Next, the image display operation during flight in the piloting support system of this embodiment will be described in detail. During flight of the helicopter 10, the camera 40 acquires image data of a predetermined range of the outside world, and the image creation unit 53 creates an image of the outside world based on the acquired image data. The angle of view of the camera 40 is determined in advance or can be adjusted by the pilot. In the head tracking mode, the range captured by the camera 40 changes depending on the position of the pilot's head, etc.

[0035] The navigation sensor 1 and the aircraft attitude sensor 2 constantly acquire and output information such as the position (latitude, longitude), altitude, direction, speed, and angle of the helicopter 10 during flight. Based on the output from the navigation sensor 1 and the aircraft attitude sensor 2, the course calculation unit 50 identifies a position (latitude, longitude) a distance L ahead of the helicopter. Then, it acquires terrain data of the surrounding area where the helicopter is flying from the terrain database 3, and identifies a ground position on the image corresponding to the identified position. On the other hand, the predicted altitude calculation unit 51 calculates a predicted altitude of the helicopter 10 when the helicopter 10 reaches the identified position from the ground altitude at the identified position based on the terrain data. However, this is the case when the helicopter 10 flies at a constant altitude above sea level. When the ground altitude is constant, the helicopter 10 moves upward or downward as it advances, thereby maintaining a constant ground altitude, so the predicted altitude is not necessarily required.

[0036] The GPV symbol generating unit 52 places a mark of a predetermined shape at a position on the image calculated by the course calculating unit 50. In addition, at a predetermined position near the display, the predicted altitude calculated by the predicted altitude calculating unit 51, information based on the airspeed obtained by the navigation sensor 1, etc., and numerical values ​​of flight parameters such as altitude deviation and speed deviation are displayed. Then, the display processing unit 54 superimposes reference information including the GPV 103 on the external image and displays it on the display unit 6. This GPV 103 is always displayed on the ground at a position L [m] straight ahead of the pilot, so that it appears to the pilot to be moving while adhering to the ground at a predetermined distance away, which has the advantages described above.

[0037] In addition, since there are cases where it is more appropriate to display the FPV rather than the GPV due to reasons such as good visibility, it is preferable to allow the pilot to manually select whether to display the GPV or the FPV. Alternatively, it may be possible to automatically determine whether the GPV or the FPV is appropriate and switch the display based on the results of processing an image from a visible light camera, or the movement of the pilot's line of sight detected by an eye tracker built into the helmet, or based on the results of detecting the movement of the helmet or the pilot's head.

[0038] [Second embodiment] Next, a piloting assistance system for an aircraft according to a second embodiment of the present invention will be described with reference to the accompanying drawings. Fig. 5 is a schematic block diagram of an aircraft flight control support system according to a second embodiment. Fig. 6 is a schematic diagram of the display screen of the conventional and present aircraft flight control support systems, and an explanatory diagram of the pilot and camera pointing directions. Fig. 7 is an explanatory diagram of a bank angle correction method in the aircraft flight control support system according to the second embodiment.

[0039] As shown in Fig. 5, this piloting support system includes functional blocks such as a navigation sensor 1, an aircraft attitude sensor 2, a sensor pod 4 including a camera 40, a sensor pod drive unit 8, a head position / angle / acceleration detection unit 7, a data processing unit 5A, and a display unit 6. The data processing unit 5A includes a sensor pod control unit 55 as a functional block in addition to an image creation unit 53 and a display processing unit 54, and the sensor pod control unit 55 includes a bank angle correction unit 550 as a characteristic functional block. Note that the same reference numerals are used to denote the same components as those in the piloting support system of the above embodiment shown in Fig. 1.

[0040] The sensor pod 4 including the camera 40 is mechanically driven by the sensor pod drive unit 8, and as a result, the imaging range of the camera 40 moves within a predetermined range in the vertical and horizontal directions. The head position / angle / acceleration detection unit 7 includes a plurality of components built into the helmet worn by the pilot, specifically, for example, a plurality of cameras that capture the surroundings and a six-axis gyro sensor. The plurality of cameras capture the scenery around the pilot or a predetermined marker in the aircraft (cockpit), and a signal processing unit (not shown) calculates the position and tilt angle of the pilot's head based on the captured images. The head position is, for example, position information on the three-dimensional reference coordinates XYZ within the aircraft. The head tilt angle is, for example, the angle between each axis of the three-dimensional reference coordinates XYZ within the aircraft and the central axis of the head. On the other hand, the six-axis gyro sensor detects the angular velocity of the head accompanying the movement of the pilot's head. Based on such information on the position, tilt angle, and angular velocity of the head, the field of view that the pilot is looking at at that time can be identified.

[0041] In the head direction tracking mode, the sensor pod control unit 55 generally receives the above-mentioned information from the head position / angle / acceleration detection unit 7, and changes the orientation of the sensor pod 4 via the sensor pod driving unit 8 so as to follow the pilot's field of view. This adjusts the orientation of the camera 40 so as to follow the pilot's field of view, and the image captured by the camera 40 (external world image) is always displayed in the center of the direction in which the pilot is facing.

[0042] 6(a) shows a state when the pilot is facing forward, and at this time, the camera 40, which tracks the movement of the pilot's head 300, is also facing forward. The display screen 200 of the HMD, which is the display unit 6, is wide in the left-right direction, and an image captured by the camera 40 is displayed in a predetermined imaging display range 201 around a center position 202 of the display screen 200. In this imaging display range 201, the image captured by the camera 40 is displayed superimposed on an image of the outside world seen through the helmet visor. On the other hand, outside the imaging display range 201 within the display screen 200, only the image of the outside world seen through the visor can be seen. When the pilot is facing forward, unless the outside visibility is extremely poor as described above, the pilot will usually be staring straight ahead, so there is no problem with the above display.

[0043] Fig. 6(b) shows the state when making a large turn to the left. The dashed line shown on the right of Fig. 6(b) is the turning trajectory, and at a certain point the pilot's head 300 faces directly ahead in the direction of the turn. In conventional piloting support systems, as described above, the camera 40 is linked to the movement of the pilot's head 300 and therefore faces the same direction as the pilot's head 300. Then, as in the case of Fig. 6(a), an image captured by the camera 40 is displayed in a predetermined imaging display range 201 around the center position 202 of the MHD display screen 200.

[0044] If the pilot is gazing directly ahead at the direction his face faces during a turn, there is no problem with the display state shown in Fig. 6(b), but when making a large turn, the pilot often looks further ahead in the direction of the turn to ensure flight safety. In other words, the pilot generally looks toward a position 303 that is further ahead than the point (indicated by a + sign) at which the pilot is actually facing along the turning trajectory 302. Therefore, in Fig. 6(b), the pilot is looking not at the center of the imaging display range 201 on the MHD display screen 200, but at a range 204 near the edge of the imaging display range 201.

[0045] This piloting support system assumes that the pilot is looking at the vicinity of the center of the imaging display range 201, and displays information on flight parameters such as altitude and speed in that area. However, since the pilot is not actually looking at the vicinity of the center of the imaging display range 201, he or she must move their line of sight left and right to check flight parameters, which is a burden for the pilot and can lead to overlooking information or delayed understanding.

[0046] Therefore, in the piloting support system of this embodiment, the bank angle correction unit 550 in the sensor pod control unit 55 corrects the direction in which the camera 40 (i.e., the sensor pod 4) is pointed based on the information on the left-right tilt angle of the helicopter 10, i.e., the bank angle, obtained from time to time by the navigation sensor 1 and the aircraft attitude sensor 2. As will be described in detail later, when the turning rate is large and the bank angle is larger than a certain level, as shown in FIG. 6(c), the camera 40 is rotated in the turning direction more than the pilot's head 300 has rotated. As a result, the camera 40 faces not in the direction in which the pilot's face is facing, but in the direction in which the pilot's line of sight is facing, that is, in the direction of the position 303 on the turning trajectory 302. At this time, the center position 202 of the MHD display screen 200 is essentially the center of the direction in which the pilot's face is facing, so the imaging display range 201 is not in the center of the display screen 200, but is in a position offset to the left. In other words, the imaging display range 201 is located near the position in which the pilot is actually looking.

[0047] Since information on flight parameters such as altitude, speed, etc. is displayed in a concentrated manner near the center of the imaging display range 201, such information on flight parameters, etc. is displayed where the pilot is actually looking, and the pilot can check such flight parameters, etc. without moving his / her line of sight significantly. In addition, the pilot can easily grasp the entire image captured by the camera 40.

[0048] Specifically, as shown in FIG. 7(a), when the angle of the pilot's head 300 is ψh (front direction is ψh=0) and the angle of the camera 40 is ψc (front direction is ψc=0), the angle of the camera 40 can be corrected using the following equation (1). ψc = f(φ) + ψh … (1) f(φ) is a function of the bank angle φ, and the relationship shown in Fig. 7(b) may be adopted. That is, when the bank angle is in the range of -5[deg]≦φ≦5[deg], f(φ)=0, and in the ranges of 5[deg]<φ≦15[deg] and -15[deg]≦φ<-5[deg], it linearly gives values of 0[deg]<f(φ)≦15[deg] and -15[deg]≦f(φ)<-0[deg] respectively. Of course, each value, the relationship in Fig. 7(b), or the formula (1) is just an example. As the bank angle increases, the camera 40 may be directed further forward in the turning direction than the direction in which the pilot's head 300 is facing. In addition to the bank angle, corrections can also be made using other parameters related to turning such as the Rate of Turn.

[0049] As described above, in the flight assistance system of the present embodiment, when the pilot operates the helicopter 10 to make a large turn, information such as the captured image by the camera 40 and flight parameters that the pilot wants to confirm during flight can be intensively displayed on the display screen in front of the turning direction where the pilot is actually looking. Of course, for pilots who are familiar with the image display of conventional flight assistance systems, the image display corrected based on the bank angle as described above may sometimes feel uncomfortable. Therefore, it may be possible to switch whether to perform the above correction or not according to the pilot's selection.

[0050] The above first and second embodiments apply the mobile body flight assistance system according to the present invention to an aircraft (specifically, a helicopter) operated by a pilot, but it can also be applied to other forms of mobile bodies. For example, the present invention can also be applied to an aircraft that is remotely operated (flown) like an unmanned aerial vehicle. Also, instead of flying objects, the present invention can be applied to submarines and submersible ships used for, for example, undersea resource exploration.

[0051] Furthermore, the above-described embodiment and modified examples are merely examples of the present invention, and it is natural that any appropriate changes, modifications, and additions made within the spirit of the present invention will also be encompassed within the scope of the claims of the present application.

[0052] [Various aspects] It will be apparent to those skilled in the art that the above-described exemplary embodiments are illustrative of the following aspects.

[0053] (1) One aspect of the mobile object steering assistance method according to the present invention is a mobile object steering assistance method, which displays information for steering on a display unit capable of displaying image information arranged in front of the eye of a user who is steering the mobile object, an image creation step of collecting image information of the outside world of the moving body and creating an outside world image based on the information; a symbol information generating step of generating symbol information that simplifies or models reference information related to the operation of the moving object that is moving; a display execution step of displaying the symbol information on the display unit by superimposing the symbol information on the outside world image so that either or both of the outside world image and the symbol information are located at a position on the screen of the display unit or within a displayable range where the user is expected to direct his or her gaze; It has the following.

[0054] (2) One aspect of the mobile object control assistance system according to the present invention is a system that embodies the mobile object control method described in the 1st paragraph, A display unit that displays image information in front of the eyes of a user who operates the moving object; an image creation unit that collects image information of the outside world of the moving body and creates an outside world image based on the information; a symbol information generating unit that generates symbol information that simplifies or models reference information related to the operation of the moving object while the moving object is moving; a display image forming unit which forms image information to be displayed on the display unit by superimposing the symbol information on the outside world image, the display image forming unit forming the image information so that the symbol information is displayed in a portion of the outside world image displayed on the screen or within a displayable range of the display unit where the user is expected to direct his or her gaze; It is equipped with the following.

[0055] (5) Another aspect of the mobile object control assistance system according to the present invention is a system that embodies the mobile object control method described in the 1st paragraph, and is different from the system described in the 2nd paragraph, A display unit that displays image information in front of the eyes of a user who operates the moving object; an image creation unit that collects image information of the outside world of the moving body and creates an outside world image based on the information; a symbol information generating unit that generates symbol information that simplifies or models reference information related to the operation of the moving object while the moving object is moving; A situation detection unit that detects a situation of the user, a situation of the moving object, or a situation around the moving object; a display image forming unit which forms image information to be displayed on the display unit by superimposing the symbol information on the external image, and which estimates a portion on the screen or within a displayable range of the display unit to which the user will direct his or her gaze based on a detection result by the situation detecting unit, and forms the image information so that either the image information or the symbol information, or both, are displayed in the estimated portion; It is equipped with the following.

[0056] In paragraphs 1, 2, and 5, "mobile body" includes flying bodies and air vehicles, including manned and unmanned aircraft, as well as submarines and submarines that move in the sea or underwater.

[0057] In the mobile object control assistance method described in paragraph 1 and the mobile object control assistance system described in paragraphs 2 and 5, symbol information including flight parameters that are referred to or important during control can be displayed in a part of the display screen or displayable range other than the center where the user is expected to direct his / her gaze. Therefore, according to the mobile object control assistance method described in paragraph 1 and the mobile object control assistance system described in paragraphs 2 and 5, a user such as a pilot can visually easily obtain a captured image of the outside of the mobile object and various information required for control without forcibly moving his / her gaze or turning his / her head. This reduces the burden on the pilot during control and enables safe movement by avoiding delays in understanding or oversight of information required for control by the pilot.

[0058] (3) In the mobile object piloting assistance system described in paragraph 2, the mobile object is an aircraft, and the part in the external world image to which the user is expected to direct his or her gaze can be within the ground range of the external world image.

[0059] According to the mobile object control assistance system described in paragraph 3, for example, when the user (pilot) tends to look at the ground due to factors such as poor visibility in the outside world, reference information to be checked when controlling the vehicle can be displayed in a part of the display screen where the user is likely to look. This makes it possible to avoid delays in understanding or oversight of information required for control by the pilot, enabling safe movement.

[0060] (4) In addition, in the vehicle piloting assistance system described in paragraph 3, the part in the outside world image to which the user is expected to direct his or her gaze can be a point on the ground corresponding to a position a predetermined distance forward from the flying object.

[0061] According to the mobile object piloting support system described in paragraph 4, when flying at a constant altitude at a constant speed, the reference information continues to be displayed at approximately the same position on the display screen. Therefore, the user does not need to move his / her eyes much to check the reference information, and the pilot can more easily avoid delays in understanding or oversight of information necessary for piloting.

[0062] (Item 6) In addition, in the mobile body steering assistance system described in Item 5, the situation detection unit detects the turning rate or the inclination of the mobile body associated with a turn when the mobile body is turning left or right, and the display image forming unit can change the part at which the user is estimated to direct his or her gaze depending on the turning rate or the inclination of the mobile body detected by the situation detection unit.

[0063] As described above, for example, when a pilot flies an aircraft to make a large turn, the pilot generally looks forward in the direction of the turn to check the situation ahead of the aircraft. In contrast, in the mobile object piloting assistance system described in paragraph 5, the larger the turning rate or the larger the tilt of the mobile object accompanying the turn, the more the user will look forward in the turning direction, and the display position of the reference information, etc. is changed. As a result, the reference information can be displayed on the display screen near the part where the pilot looks, regardless of the size of the turn, and the pilot does not need to move his or her eyes significantly to check the reference information.

[0064] (Item 7) In addition, in the mobile body piloting assistance system described in Item 6, the display unit is a head-mounted display or helmet-mounted display worn on the head of a user, and the display image forming unit includes a control unit that changes the shooting range of the imaging unit included in the image creation unit in accordance with the position, angle, and acceleration of the user's head, and the control unit can be configured to correct the change in the shooting range of the imaging unit in accordance with the turning rate or inclination of the mobile body detected by the situation detection unit.

[0065] According to the mobile object piloting support system described in paragraph 7, when the pilot faces forward during a turn and turns only his / her line of sight forward in the turning direction, the external image obtained by the photographing unit and the reference information can be displayed together on the display screen of the head mounted display or helmet mounted display near the part of the body to which the pilot is turning. This allows the pilot to check both the photographed image, such as an infrared image, and the reference information without moving his / her line of sight. [Explanation of symbols]

[0066] 1...Navigation sensor 2...Aircraft attitude sensor 3. Terrain database 4. Sensor pod 40…Camera 5, 5A: Data processing section 50…Course calculation section 51…Predicted altitude calculation section 52...GPV symbol generator 53…Image Creation Department 54...Display processing unit 55...Sensor pod control unit 550…Bank angle correction section 6…Display section 7...Head position / angle / acceleration detection unit 8…Sensor pod drive unit 10. Helicopter 100…nose mark 101...line 103...GPV

Claims

1. A method for assisting a vehicle in steering, comprising the steps of: displaying information for steering on a display unit that displays image information in front of a user who is steering the vehicle; an image creation step of collecting image information of the outside world of the moving body and creating an outside world image based on the information; a symbol information generating step of generating symbol information that simplifies or models reference information related to the operation of the moving object that is moving; a situation detection step of detecting a situation of the user and a situation of the moving object or a situation around the moving object; a display execution step of estimating a direction of the user's line of sight based on a detection result of the situation detection step when the symbol information is superimposed on the outside world image and displayed on the display unit, and displaying either the image information or the symbol information, or both, on a screen of the display unit or within a displayable range thereof at a position corresponding to the direction of the user's line of sight; A method for assisting a moving object in steering, comprising:

2. A display unit that displays image information in front of the eyes of a user who operates the moving object; an image creation unit that collects image information of the outside world of the moving body and creates an outside world image based on the information; a symbol information generating unit that generates symbol information that simplifies or models reference information related to the operation of the moving object while the moving object is moving; A situation detection unit that detects a situation of the user and a situation of the moving object or a situation around the moving object; a display image forming unit which forms image information to be displayed on the display unit by superimposing the symbol information on the external image, the display image forming unit estimating a direction of the user's line of sight based on a detection result by the situation detecting unit, and forms the image information so that either the image information or the symbol information, or both, are displayed on the screen of the display unit or in a displayable range at a position corresponding to the direction of the line of sight; A vehicle control assistance system comprising:

3. 3. The mobile body steering assistance system of claim 2, wherein the situation detection unit detects a turning rate or an inclination of the mobile body accompanying a turn when the mobile body is turning left or right, and the display image forming unit changes a portion corresponding to the direction of the user's line of sight in accordance with the turning rate or the inclination of the mobile body detected by the situation detection unit.

4. 4. The vehicle steering assistance system according to claim 3, wherein the display unit is a head-mounted display or helmet-mounted display worn on the head of a user, the display image forming unit includes a control unit that changes a shooting range by an imaging unit included in the image creation unit in accordance with a position, angle, and acceleration of the user's head, and the control unit corrects the change in the shooting range by the imaging unit in accordance with a turning rate or an inclination of the vehicle detected by the situation detection unit.

5. The moving object is an aircraft, The vehicle piloting assistance system of any one of claims 2 to 4, wherein the display image forming unit further has a function of forming the image information so as to display the symbol information at a point on the ground corresponding to a position a predetermined distance forward from the aircraft, the point being displayed on the screen or within a displayable range of the display unit.

Images