Obstacle detection device

Abstract

To make it easy to identify obstacles. [Solution] The obstacle detection device comprises a camera that captures images from the aircraft's body in a predetermined direction, a measuring device that measures the distance of an obstacle from the aircraft in at least one or more measurement directions including the camera's imaging direction, a processing unit that performs processing based on the imaging results from the camera and the measurement results from the measuring device, and a display device capable of displaying the processing results from the processing unit. The processor of the processing unit performs processing including, for each measurement direction, identifying a threat level that represents the likelihood of the obstacle contacting the aircraft based on the distance information included in the measurement results, generating a level image that can identify the threat level based on the identified threat level, generating a superimposed image by overlaying the level image on an image representing the imaging results, and displaying the generated superimposed image on the display device.

Description

Technical Field

[0001] The present invention relates to an obstacle detection device.

Background Art

[0002] For example, Patent Document 1 discloses an aircraft including a detection unit that detects an object that can become an obstacle during hovering. In such an aircraft of Patent Document 1, display data indicating the approach or approachability of the object to the aircraft is generated using the detection data of the detection unit, and the display data is output to the display unit.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the aircraft of Patent Document 1, it is difficult for the crew of the aircraft to grasp what kind of object the object (obstacle) indicated by the approachability display data is, for example, a cliff, a rock, vegetation, or a tower.

[0005] Therefore, an object of the present invention is to provide an obstacle detection device that can easily grasp an obstacle.

Means for Solving the Problems

[0006] In order to solve the above problems, an obstacle detection device according to an embodiment of the present invention includes a camera that images a predetermined direction from the fuselage of the aircraft, a measuring device that measures the distance of an obstacle from the fuselage in one or more measurement directions including at least the imaging direction of the camera, a processing device that executes processing based on the imaging result by the camera and the measurement result by the measuring device A display device capable of displaying the processing results from the aforementioned processing device, Equipped with, The aforementioned processing apparatus is One or more processors, One or more memory connected to the processor, It has, The aforementioned processor, For each of the measurement directions, a threat level representing the likelihood of the obstacle contacting the aircraft is identified based on the distance information included in the measurement results. Based on the identified threat level, a level image that can identify the threat level is generated, A superimposed image is generated by superimposing the level image onto the image representing the aforementioned imaging result, The generated superimposed image is displayed on the display device, Execute the process that includes this. [Effects of the Invention]

[0007] According to the present invention, obstacles can be easily detected. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 is a schematic side view showing the configuration of an aircraft to which the obstacle detection device according to this embodiment is applied. [Figure 2] Figure 2 is a schematic plan view showing the configuration of an aircraft to which the obstacle detection device according to this embodiment is applied. [Figure 3] Figure 3 is a block diagram showing the configuration of the obstacle detection device. [Figure 4] Figure 4 shows an example of a display on a display device. [Figure 5] Figure 5 illustrates an example of threat levels and level images. [Figure 6] Figure 6 is a diagram illustrating the sound output from a sound output device. [Figure 7] Figure 7 is a flowchart illustrating the operation flow of the imaging result processing unit. [Figure 8] FIG. 8 is a flowchart for explaining the operation flow of the measurement result processing unit and the image processing unit. [Figure 9] FIG. 9 is a diagram for explaining display position correction. [Figure 10] FIG. 10 is a diagram for explaining display position correction.

BEST MODE FOR CARRYING OUT THE INVENTION

[0009] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. The specific dimensions, materials, numerical values, etc. shown in such embodiments are merely examples for facilitating the understanding of the invention, and do not limit the present invention unless otherwise specified. In the present specification and drawings, elements having substantially the same functions and configurations are denoted by the same reference numerals to omit redundant descriptions, and elements not directly related to the present invention are not shown.

[0010] FIG. 1 is a schematic side view showing the configuration of an aircraft 2 to which an obstacle detection device 1 according to the present embodiment is applied. FIG. 2 is a schematic plan view showing the configuration of an aircraft 2 to which the obstacle detection device 1 according to the present embodiment is applied. The aircraft 2 is, for example, a hovering-capable rotary-wing aircraft such as a helicopter. Note that the aircraft 2 is not limited to a rotary-wing aircraft, and may be an airplane that flies by lift according to the propulsive force forward. Referring to FIGS. 1 and 2, the configuration of the aircraft 2 will be described.

[0011] The aircraft 2 includes a fuselage 10, a main rotor 12, a tail rotor 14, a drive device 16, a camera 20, a measurement device 22, a display device 24, a sound output device 26, and a processing device 30.

[0012] The aircraft body 10 includes a cabin 40 where crew members such as pilots can board, and a tail boom 42 extending rearward from the cabin 40. The main rotor 12 and the tail rotor 14 are rotary wings. The main rotor 12 is provided on the upper part of the aircraft body 10, more specifically, on the upper part of the cabin 40. The tail rotor 14 is provided at the end of the tail boom 42. Note that the number of rotary wings is not limited to two, and it can also be composed of one or three or more.

[0013] The drive device 16 is composed of, for example, a reciprocating engine, a jet engine, a motor, etc., and rotates the main rotor 12 and the tail rotor 14. When the main rotor 12 and the tail rotor 14 are rotated by the drive device 16, lift and thrust are generated on the aircraft 2. In addition, the aircraft 2 can hover, that is, substantially stationary in the air by rotating the main rotor 12 and the tail rotor 14.

[0014] The camera 20 is provided at the lower part of the aircraft body 10. For example, the camera 20 may be provided at the lower rear part of the cabin 40 at the part where the cabin 40 and the tail boom 42 are connected. The camera 20 is configured to be able to image a predetermined direction outside the aircraft body 10 from the aircraft body 10. That is, the predetermined direction here means the imaging direction serving as a reference in the camera 20. The imaging direction of the camera 20 may be substantially fixed with respect to the aircraft body 10.

[0015] In FIG. 1, the solid arrow A10 indicates an example of the imaging direction of the camera 20. Assume that the imaging direction of the camera 20 is, for example, the lower rear of the aircraft body 10. More specifically, the imaging direction of the camera 20 is a direction inclined downward by a predetermined angle with respect to the rear of the aircraft body 10 indicated by the dashed-dotted arrow A20. The predetermined angle is set to, for example, 30° etc., but it may be set to various angles in consideration of the actual operation situation of the obstacle detection device 1. Note that the imaging direction of the camera 20 is not limited to the lower rear of the aircraft body 10, and it may be, for example, the rear of the aircraft body 10 or the lower part of the aircraft body 10.

[0016] As shown in Figure 1, the elevation of camera 20 is the vertical angle of camera 20 relative to the imaging direction of camera 20. In other words, the vertical direction of camera 20 is the vertical direction in the coordinate system of camera 20. For convenience of explanation, elevation may be denoted as "EL". "+EL" means a positive elevation, and "-EL" means a negative elevation. For example, "+EL" may mean the downward elevation relative to the imaging direction of camera 20.

[0017] Furthermore, as shown in Figure 2, the azimuth of camera 20 is the angle in the left-right direction relative to the imaging direction of camera 20. In other words, the left-right direction of camera 20 is the horizontal direction in the coordinate system of camera 20. For convenience of explanation, azimuth may be denoted as "AZ". "+AZ" means a positive azimuth, and "-AZ" means a negative azimuth. For example, "+AZ" may be the azimuth to the left relative to the imaging direction of camera 20.

[0018] The camera 20 can capture images within a predetermined elevation range and a predetermined azimuth range, with respect to the imaging direction.

[0019] Camera 20 may have a fisheye lens and a wide-angle field of view. For example, camera 20 may be capable of imaging an elevation range from a predetermined lower limit EL of -45° or less to a predetermined upper limit EL of +45° or more in the elevation direction. Camera 20 may be capable of imaging an azimuth range from a predetermined lower limit AZ of -75° or less to a predetermined upper limit AZ of +75° or more in the azimuth direction.

[0020] The measuring device 22 is configured to measure the distance from the aircraft 10 to an obstacle. The measuring device 22 is, for example, a LiDAR (Light Detection And Ranging) device that measures the distance to an obstacle by emitting light and receiving the reflected light. However, the measuring device 22 is not limited to a LiDAR device; for example, it may be a RADAR (Radio Detection And Ranging) device that emits radio waves and receives the reflected waves to measure the distance to an obstacle.

[0021] The measuring device 22 is installed near the camera 20. The measuring device 22 is formed, for example, in a roughly cylindrical shape. The measuring device 22 is installed in a position that matches the orientation of the camera 20. For example, the measuring device 22 is installed in a position where the central axis of the cylinder of the measuring device 22 is perpendicular to the imaging direction of the camera 20.

[0022] The measuring device 22 can irradiate measuring light from its outer surface to the outside of the measuring device 22. The measurement direction in the measuring device 22 is the direction in which the distance to the obstacle is measured, specifically the direction in which the light is irradiated. The measuring device 22 can measure the distance from the aircraft body 10 to the obstacle in one or more measurement directions, including at least the imaging direction of the camera 20. The measuring device 22 can change the measurement direction and measure the distance to the obstacle present in each measurement direction.

[0023] The measuring device 22 can change its measurement direction in the circumferential direction. For example, the measuring device 22 may be configured to change its measurement direction over a 360° circumferential direction. The circumferential direction of the measuring device 22 corresponds to the azimuth direction of the camera 20. That is, the measuring device 22 can substantially change its measurement direction along the azimuth direction of the camera 20. The measurement interval in the azimuth direction of the measuring device 22 may be, for example, 1°.

[0024] Furthermore, the reference plane of the measuring device 22 is assumed to be a plane that includes the center of the measuring device 22 and is perpendicular to the central axis of the measuring device 22. The measuring device 22 can change its measurement direction in the depression and elevation directions relative to its reference plane. For example, the measuring device 22 may be configured to change its measurement direction by 45° or more in the depression direction and 45° or more in the elevation direction relative to its reference plane. The depression and elevation directions of the measuring device 22 correspond to the elevation direction of the camera 20. That is, the measuring device 22 can substantially change its measurement direction along the elevation direction of the camera 20. The measurement interval in the elevation direction of the measuring device 22 may be, for example, 1°.

[0025] The display device 24 consists of a liquid crystal display, an electroluminescent display, an LED (light-emitting diode) display, etc. The display device 24 may be installed, for example, on the instrument panel of the cabin 40. The display device 24 can display the processing results of the processing device 30, which will be described later, as an image. The image displayed on the display device 24 may be updated in substantially real time. The crew can view the image as flight-related information through the display device 24.

[0026] The sound output device 26 includes a headset with a microphone and speaker, and an ICS (Internal Communication System). The headset is worn by the crew. The ICS transmits and receives sound signals between the processing unit 30 and one or more headsets. The sound output device 26 can, for example, output the sound of the sound signal transmitted from the processing unit 30 through the speaker of the headset. The crew can hear sounds as flight-related information through the speaker of the headset.

[0027] The processing unit 30 is a computer capable of performing various processes necessary for flight. For example, the processing unit 30 may perform processing based on the imaging results from the camera 20 and the measurement results from the measuring device 22.

[0028] Figure 3 is a block diagram showing the configuration of the obstacle detection device 1. The processing unit 30 of the obstacle detection device 1 includes one or more processors 50 and one or more memories 52 connected to the processors 50. The memories 52 include ROM for storing programs and RAM as a work area. The memories 52 may also include non-volatile storage that allows data to be rewritten. The processors 50 work in cooperation with the programs contained in the memories 52 to perform various processes.

[0029] For example, the processor 50 may function as an imaging result processing unit 60, a measurement result processing unit 62, an image processing unit 64, and a sound processing unit 66 by executing a program.

[0030] The imaging result processing unit 60 acquires imaging results from the camera 20 and performs processing to correct the acquired imaging results. The correction of imaging results will be described in detail later.

[0031] The measurement result processing unit 62 acquires measurement results from the measurement device 22 and performs a process to correct the acquired measurement results. The correction of measurement results will be described in detail later. The measurement result processing unit 62 also identifies a threat level for each measurement direction, based on the distance information (more specifically, the corrected distance information) included in the measurement results, which represents the likelihood of an obstacle contacting the aircraft 10. For example, the measurement directions may be distributed in a mesh pattern with predetermined angular (e.g., 1°) intervals in the elevation direction and predetermined angular (e.g., 1°) intervals in the azimuth direction. The measurement result processing unit 62 identifies a threat level for each of these measurement directions.

[0032] The image processing unit 64 generates a level image that can identify the threat level based on the threat level identified by the measurement result processing unit 62. The image processing unit 64 generates a superimposed image by overlaying the level image on the image representing the imaging result after processing by the imaging result processing unit 60. The image processing unit 64 displays the generated superimposed image on the display device 24.

[0033] The sound processing unit 66 generates an audio signal corresponding to the highest threat level among the threat levels for each measurement direction. The sound processing unit 66 outputs the generated audio signal from the sound output device 26. The imaging result processing unit 60, measurement result processing unit 62, image processing unit 64, and sound processing unit 66 will be described in detail later.

[0034] Figure 4 shows an example of the display of the display device 24. The display screen 70 of the display device 24 has a first display area 72 set. The first display area 72 is the area in which the entire superimposed image 80 is displayed.

[0035] The superimposed image 80 is an image in which a level image 84, generated based on the measurement results of the measuring device 22, is superimposed on a camera image 82 based on the imaging results of the camera 20. In the example in Figure 4, the camera image 82 displays the scenery below and behind the aircraft 2. Note that a part of the aircraft 2 (for example, the tail boom 42) may be displayed in both the camera image 82 and the superimposed image 80.

[0036] Furthermore, in the example in Figure 4, the level image 84 is displayed using cross-hatching for the sake of explanation. In reality, the level image 84 may be a semi-transparent image of a predetermined color. For example, the difference in the fineness of the cross-hatching in Figure 4 may correspond to the difference in color in the level image 84. In the superimposed image 80, by making the level image 84 a semi-transparent image, the camera image 82 behind the level image 84 can also be seen in the area where the level image 84 is displayed.

[0037] The first display area 72 associates a specific projection plane, determined by the elevation direction and azimuth direction relative to the imaging direction of the camera 20, with the display screen 70 of the display device 24. For example, the first display area 72 may display a superimposed image 80 that is within a range of ±45° in the elevation direction and within a range of ±75° in the azimuth direction, relative to the imaging direction of the camera 20.

[0038] The first display area 72 may be divided into a plurality of second display areas 74. The second display areas 74 may be divided, for example, by a grid. The second display areas 74 may be squares with 15° intervals in the elevation direction and 15° intervals in the azimuth direction. The size of the second display areas 74 is not limited to the example shown, and may be larger or smaller than the example shown. Also, the shape of the second display areas 74 is not limited to a square, but may be a rectangle or a predetermined polygon.

[0039] As described above, the measuring device 22 can substantially change the measurement direction along the elevation direction of the camera 20, and can also change the measurement direction along the azimuth direction of the camera 20. In this case, the positions representing the measurement direction of the measuring device 22 on the first display area 72 will be distributed across the entire first display area 72, for example, in a mesh pattern with 1° intervals.

[0040] The multiple second display areas 74 are configured such that each of the multiple second display areas 74 corresponds to at least one measurement direction. In other words, since the measurement directions are distributed in both the azimuth direction and the elevation direction, each of the multiple second display areas 74 is substantially associated with one or more measurement results from the measuring device 22.

[0041] As described above, the measurement result processing unit 62 identifies the threat level based on the distance information included in the measurement results. When identifying the threat level, the measurement result processing unit 62 may identify the threat level for each second display area 74. In addition, if there are multiple distance information items in one second display area 74, the measurement result processing unit 62 may identify the threat level in the second display area 74 based on the shortest distance information among the multiple distance information items, that is, the distance information closest to the aircraft 10.

[0042] As described above, the image processing unit 64 generates a level image 84 based on the identified threat level. When generating the level image 84, the image processing unit 64 may generate a level image 84 for each second display area 74. In other words, the level image 84 may be able to identify the threat level for each second display area 74.

[0043] Figure 5 illustrates an example of a threat level and a level image 84. As the distance between the aircraft 10 and the obstacle, represented by distance information, decreases, it is inferred that the probability of the obstacle contacting the aircraft 10 increases. Therefore, the measurement result processing unit 62 identifies the threat level such that the threat level increases as the distance determined by distance information decreases. The image processing unit 64 generates a level image 84 color-coded according to the threat level.

[0044] For example, as shown in Figure 5, if the distance is longer than 20m, the threat level may be set to "0" and level image 84 may not be generated. If the distance is longer than 10m and 20m or less, the threat level may be set to "1" and a "green" level image 84 may be generated. If the distance is longer than 5m and 10m or less, the threat level may be set to "2" and a "yellow" level image 84 may be generated. If the distance is 5m or less, the threat level may be set to "3" and a "red" level image 84 may be generated.

[0045] Furthermore, the level image 84 may be displayed in various colors, not limited to the exemplified "green," "yellow," and "red." Also, the level image 84 may be displayed not only in specific colors, but also, for example, in specific patterns. In addition, the correspondence between specific distances and specific level images 84 may be various combinations, not limited to the exemplified ones. Moreover, the threat level may be divided into various stages, not limited to the example of four stages from "0" to "3."

[0046] Let's return to Figure 4 for explanation. As mentioned above, in Figure 4, the cross-hatching represents the level image 84. For example, level image 84A, in which the cross-hatching is relatively coarse, may be the "green" level image 84 corresponding to threat level "1". Level image 84B, in which the cross-hatching is relatively medium, may be the "yellow" level image 84 corresponding to threat level "2". Level image 84C, in which the cross-hatching is relatively fine, may be the "red" level image 84 corresponding to threat level "3". Note that the portion of the superimposed image 80 in which the level image 84 is not displayed means that the threat level is "0".

[0047] As described above, in the obstacle detection device 1 of this embodiment, a superimposed image 80, in which a level image 84 capable of identifying the threat level is superimposed on the camera image 82, is displayed on the display device 24. As a result, the crew of the aircraft 2 can instantly and intuitively grasp the position and distance of obstacles by looking at the superimposed image 80 on the display device 24, and can also instantly and appropriately grasp what kind of obstacles are nearby.

[0048] In other words, the obstacle detection device 1 of this embodiment makes it possible for the crew of the aircraft 2 to easily identify obstacles.

[0049] Furthermore, in the obstacle detection device 1 of this embodiment, the first display area 72 on which the superimposed image 80 is displayed is divided into a plurality of second display areas 74, and level images 84 capable of identifying threat levels are displayed in units of the second display areas 74. As a result, the crew of the aircraft 2 can more easily grasp the location of nearby obstacles by looking at the superimposed image 80 in the first display area 72.

[0050] Furthermore, in the obstacle detection device 1 of this embodiment, the resolution of the level image 84 in the superimposed image 80 can be improved by making the divisions of the second display area 74 finer. Also, in the obstacle detection device 1 of this embodiment, the processing load when generating the level image 84 based on distance information can be suppressed by making the divisions of the second display area 74 coarser.

[0051] Furthermore, as shown in Figure 4, a third display area 90 may be provided outside the first display area 72 on the display screen 70 of the display device 24. The position of the third display area 90 is, for example, to the upper right of the first display area 72, but it is not limited to this example and may be at various positions outside the first display area 72.

[0052] The image processing unit 64 may generate a sub-image 92 representing a threat level of a predetermined level or higher if at least a portion of the superimposed image 80 includes a level image 84 with a threat level of a predetermined level or higher. The image processing unit 64 may display the generated sub-image 92 in the third display area 90.

[0053] For example, as shown in Figure 4, if the image processing unit 64 includes a level image 84C with threat level "3" in at least a portion of the superimposed image 80, it may display a sub-image 92 labeled "WARNING" in the third display area 90. Note that the labeling of the sub-image 92 is not limited to the example "WARNING" and may be various other labels.

[0054] Furthermore, at least a portion of the superimposed image 80 may not contain a level image 84 with a threat level higher than "2," but may contain a level image 84B with a threat level of "2." In this case, the image processing unit 64 may display a sub-image 92 labeled "CAUTION" in the third display area 90. Note that the labeling of the sub-image 92 is not limited to the example "CAUTION," but may be various other labels.

[0055] Furthermore, if at least a portion of the superimposed image 80 does not include a level image 84 with a threat level of "2" or higher, the image processing unit 64 may prevent the display of the sub-image 92.

[0056] Thus, in the obstacle detection device 1 of this embodiment, if at least a portion of the superimposed image 80 includes a level image 84 with a threat level of a predetermined level or higher, a sub-image 92 is displayed. As a result, even if the resolution of the level image 84 is increased and the second display area 74 is reduced, for example, the crew can prevent missing the occurrence of a level image 84 with a threat level of a predetermined level or higher by looking at the sub-image 92.

[0057] Figure 6 is a diagram illustrating the sound output from the sound output device 26. As described above, the sound processing unit 66 may generate a sound signal corresponding to the highest threat level among the threat levels for each measurement direction, and output the sound of the generated sound signal from the sound output device 26.

[0058] For example, if the highest identified threat level is "0", the sound processing unit 66 may choose not to generate an audio signal corresponding to the threat level. In this case, no sound related to the approach of an obstacle will be output.

[0059] Furthermore, if the highest identified threat level is "1", the sound processing unit 66 may output a first output sound corresponding to threat level "1" from the sound output device 26. The first output sound may be short in length and output intermittently, for example, as shown in Figure 6.

[0060] Furthermore, if the highest identified threat level is "2", the sound processing unit 66 may output a second output sound corresponding to threat level "2" from the sound output device 26. The second output sound may be longer in duration than the first output sound and output discontinuously, as shown in Figure 6, for example.

[0061] Furthermore, if the highest identified threat level is "3", the sound processing unit 66 may output a third output sound corresponding to threat level "3" from the sound output device 26. The third output sound may be, for example, a long, continuous sound, as shown in Figure 6.

[0062] Thus, in the obstacle detection device 1 of this embodiment, a sound corresponding to the highest identified threat level is output from the sound output device 26. As a result, the crew can recognize the threat level by listening to the sound output from the sound output device 26, even in situations where they do not have time to look at the display device 24.

[0063] The provision for outputting sounds corresponding to the threat level from the sound output device 26 may be omitted.

[0064] Figure 7 is a flowchart illustrating the operation flow of the imaging result processing unit 60. The imaging result processing unit executes the series of processes shown in the flowchart of Figure 7 each time the first execution timing, which is repeated at the first time interval, arrives.

[0065] When the first execution timing arrives, the imaging result processing unit 60 acquires the current imaging result from the camera 20 (S10). As mentioned above, since the camera 20 includes a fisheye lens, the acquired image contains distortion due to the influence of the fisheye lens. Therefore, the imaging result processing unit 60 performs distortion correction processing to remove the distortion from the acquired image (S11). Distortion correction can be achieved by known methods. By performing distortion correction, an image from which distortion has been removed from the imaging result is obtained.

[0066] Next, the imaging result processing unit 60 performs a trimming process to extract an image from the distortion-corrected image that matches the range of the first display area 72 of the display device 24 (S12). For example, the imaging result processing unit 60 may extract an image from the distortion-corrected image that is within a range of ±45° in the elevation direction and within a range of ±75° in the azimuth direction, with reference to the imaging direction of the camera 20.

[0067] Next, the image processing unit 60 stores the cropped image in the memory 52 as the camera image 82 (S13), and terminates the series of processes shown in Figure 7. As a result, the camera image 82 is updated to the latest image.

[0068] Furthermore, if the camera 20 does not include a fisheye lens and the resulting image is substantially free of distortion, the distortion correction process may be omitted. Also, if the size of the resulting image is substantially the same as the size of the first display area 72, the cropping process may be omitted.

[0069] Figure 8 is a flowchart illustrating the operation flow of the measurement result processing unit 62 and the image processing unit 64. The measurement result processing unit 62 and the image processing unit 64 execute a series of processes shown in the flowchart of Figure 8 each time a second execution timing occurs, which is repeated at a second time interval. The second time interval may be substantially the same as the first time interval of the imaging result processing unit 60, or it may be different from the first time interval.

[0070] When the second execution timing arrives, the measurement result processing unit 62 acquires the current measurement result of the measuring device 22 (S20). The acquired measurement result includes, for example, information that associates direction information, which identifies the measurement direction, and distance information, which represents the distance to the obstacle, for all measurement directions that can be measured by the measuring device 22. The direction information includes the elevation and azimuth in the coordinate system of the measuring device. The position of the obstacle can be identified based on the direction information and distance information. In other words, the measurement result substantially includes point cloud information that represents the position of the obstacle.

[0071] Next, the measurement result processing unit 62 performs noise reduction on the acquired measurement results to remove the effects of weather conditions such as rain, fog, and snow (S21). This improves the obstacle detection accuracy of the obstacle detection device 1 in this embodiment.

[0072] For example, the measuring device 22 measures the distance to an obstacle based on the reflected light intensity received in response to light transmitted in the measurement direction. The measurement result processing unit 62 determines, for each measurement direction, whether the reflected light intensity corresponding to that measurement direction is below a predetermined threshold.

[0073] If the reflection intensity corresponding to the measurement direction is determined to be greater than a predetermined threshold, it is inferred that an obstacle is present. In this case, the measurement result processing unit 62 identifies the distance information to the obstacle based on the time the reflection intensity was received. On the other hand, if the reflection intensity corresponding to the measurement direction is obtained, but it is determined to be below a predetermined threshold, it is inferred that it was reflected by rain, fog, snow, etc. In this case, the measurement result processing unit 62 may update the distance information in this measurement direction as specific information indicating that there is no obstacle.

[0074] Next, if a part of the aircraft 10 is present in the measurement direction, the measurement result processing unit 62 performs a masking process to treat that part of the aircraft 10 as not present in the measurement result, since that part of the aircraft 10 will not be an obstacle to the aircraft 2 (S22). More specifically, the measurement result processing unit 62 may update the distance information in the measurement direction where it is known in advance that a part of the aircraft 10 is present as specific information indicating that there is no obstacle. This prevents the aircraft 10 from being mistakenly detected as an obstacle.

[0075] Here, although the measuring device 22 is located near the camera 20, it is physically separate from the camera 20, so the origin of the coordinate system of the measuring device 22 has a predetermined difference from the origin of the coordinate system of the camera 20. Therefore, the measurement result processing unit 62 performs an offset process to correct the position of the obstacle identified by the measurement results of the measuring device 22 (more specifically, the measurement results after noise reduction and masking) (S23). This improves the accuracy of the obstacle's position.

[0076] The difference between the position of the camera 20 and the position of the measuring device 22, that is, the amount of displacement between the origin of the coordinate system of the measuring device 22 and the origin of the coordinate system of the camera 20, is pre-stored in the memory 52. ​​The measurement result processing unit 62 corrects the position of obstacles for each measurement direction based on the difference between the position of the camera 20 and the position of the measuring device 22.

[0077] Next, the measurement result processing unit 62 extracts measurement results from the measurement results after offset processing that cover the range to be superimposed on the image 80 (S24). For example, the measurement result processing unit 62 may extract measurement results from the measurement results after offset processing that are within ±45° in the elevation direction and within ±75° in the azimuth direction, with respect to the imaging direction of the camera 20. This reduces the amount of information that will be processed in subsequent steps, thereby reducing the processing load in subsequent steps.

[0078] As described above, the position of an obstacle based on the measurement results is determined by directional information and distance information. If the values ​​of AZ and EL as directional information are used directly to map the position of the obstacle to the display screen 70 of the display device 24, it may be misaligned with the position of the obstacle in the camera image 82.

[0079] Therefore, the measurement result processing unit 62 performs display position correction to convert the position of the obstacle identified by the measurement result to a position within the display screen 70 of the display device 24 (more specifically, within the first display area 72) (S25).

[0080] Figures 9 and 10 illustrate display position correction. Figures 9 and 10 explain the azimuth direction, and for convenience, the explanation of the elevation direction is omitted, but the same approach as the explanation of the azimuth direction can be applied to the elevation direction as well.

[0081] Point P10 in Figure 9 indicates the origin of the coordinate system of the measuring device 22, which corresponds to the origin of the coordinate system of the camera 20. That is, in Figure 9, the offset processing has been performed, and it is assumed that the origin of the coordinate system of the measuring device 22 coincides with the origin of the coordinate system of the camera 20. In Figure 9, the solid arrow B10 indicates the reference measurement direction in the measuring device 22, which corresponds to the imaging direction of the camera 20.

[0082] Point P12 in Figure 9 shows an example of the location of an obstacle identified based on measurement direction and distance information. "AZdisp" refers to the azimuth when the obstacle is located at point P12.

[0083] For the sake of explanation, the plane perpendicular to the imaging direction and containing the position of the obstacle at point P12 is sometimes referred to as the first plane C10. "Xdisp" indicates the distance between point P10, which is the origin, and the first plane C10. "Ydisp" indicates the distance between point P14, the intersection of the line along the imaging direction and the first plane C10, and point P12, which is an example of an obstacle.

[0084] "AZdispmax" indicates the maximum field of view in the azimuth direction. "Ydispmax" at point P16 is on the first plane C10 and indicates the position of the maximum field of view in the azimuth direction "AZdispmax". Referring to Figure 9, the measurement result processing unit 62 can derive "Ydispmax" using the following equation (1). Ydispmax=Xdisp·tan(AZdispmax) ···(1)

[0085] Figure 10 shows the first display area 72 representing the two-dimensional screen of the display device 24. "Yref" indicates the horizontal position within the first display area 72, corresponding to "Ydisp" at point P12 in Figure 9. The dashed line B20 is the horizontal center line in the first display area 72 and corresponds to "Yref=0". "Yrefmax" indicates the maximum horizontal position in the first display area 72.

[0086] The measurement result processing unit 62 performs the process of converting "Ydisp" in Figure 9 to "Yref" in Figure 10. The measurement result processing unit 62 performs the conversion so that the ratio of "Yref" to "Yrefmax" in Figure 10 and the ratio of "Ydisp" to "Ydispmax" in Figure 9 are substantially equal. More specifically, the measurement result processing unit 62 can derive "Yref" using the following equation (2). Yref=Yrefmax·Ydisp / Ydispmax···(2)

[0087] In other words, during the display position correction process, the measurement result processing unit 62 converts the position information of the obstacle in the first plane C10 into two-dimensional position information on the two-dimensional screen of the display device 24, for each obstacle.

[0088] As a result, in the obstacle detection device 1 of this embodiment, the position of the obstacle based on the measurement results can be mapped to an appropriate position on the two-dimensional screen of the display device 24, making it possible to prevent the position of the obstacle from shifting relative to the camera image 82. Consequently, in the obstacle detection device 1 of this embodiment, it is possible to prevent the display position of the level image 84 based on the distance information of the obstacle from shifting, and to improve the accuracy of the superimposed image 80.

[0089] Let's return to Figure 8 for explanation. After display position correction, the measurement result processing unit 62 performs an aircraft coordinate transformation to convert the distance information of the measurement result into distance information expressed in the aircraft coordinate system, which is a coordinate system based on a specific position of the aircraft 10 (S26). More specifically, the distance information to be transformed by the aircraft coordinate transformation may be corrected to the coordinate system of the camera 20 by offset processing, and the position within the display screen 70 may be corrected by display position correction.

[0090] The aircraft coordinate system may be, for example, a coordinate system based on the position of the tail rotor 14. In this case, the position of the tail rotor 14 is set as the origin of the distance information after the aircraft coordinate transformation, and the distance information after the aircraft coordinate transformation is expressed as the distance from the tail rotor 14 to the obstacle.

[0091] In this case, the aircraft coordinate transformation only involves a coordinate transformation for distance information representing the distance to the obstacle, and does not transform the position of the obstacle. In other words, the position of the obstacle on the screen is maintained at the position identified by the display position correction process, and only the distance to the obstacle at that position is represented, for example, as the distance from the tail rotor 14.

[0092] In step S26, the measurement result processing unit 62 may convert the distance information into a first aircraft coordinate system based on the position of the tail rotor 14, and also convert it into a second aircraft coordinate system based on the position of the main rotor 12. The measurement result processing unit 62 may then compare the distance information in the first aircraft coordinate system with the distance information in the second aircraft coordinate system for each obstacle location, and select the distance information with a relatively shorter distance from among them as the distance information after the aircraft coordinate transformation.

[0093] Next, the measurement result processing unit 62 divides the point cloud information representing the obstacles (more specifically, the measurement results after the corrections from step S21 to step S26) into two display areas 74 (S27).

[0094] The measurement result processing unit 62 identifies the shortest distance information among the one or more distance information within the second display area 74 in each of the multiple second display areas 74 (S28).

[0095] The measurement result processing unit 62 determines the threat level in each of the multiple second display areas 74 based on the shortest distance information within the second display area 74 (S29).

[0096] Furthermore, since display position correction is performed in step S25, in step S29, the threat level will ultimately be determined for each converted 2D position information in the display position correction, based on the distance information corresponding to that converted 2D position information. Also, since aircraft coordinate transformation is performed in step S26, in step S29, the threat level will ultimately be determined based on the distance information expressed in the aircraft coordinate system.

[0097] The image processing unit 64 generates a level image 84 for each second display area 74 based on the identified threat level (S30). The level image 84 is a semi-transparent image having a predetermined transparency.

[0098] The image processing unit 64 reads the latest camera image 82 from the memory 52 (S31). The image processing unit 64 generates a superimposed image 80 by superimposing the level image 84 generated in step S30 onto the read latest camera image 82 (S32).

[0099] The image processing unit 64 determines whether or not at least a portion of the superimposed image 80 contains a level image 84 with a threat level of a predetermined level or higher (S33).

[0100] If it is determined that a level image 84 with a threat level of a predetermined level or higher is included (YES in S33), the image processing unit 64 generates a sub-image 92 based on the threat level indicated by the level image 84 (S34) and proceeds to the process in step S35.

[0101] If the image processing unit 64 determines that no level image 84 with a threat level of a predetermined level or higher is included (NO in S33), it proceeds to step S35 without generating a sub-image.

[0102] In step S35, the image processing unit 64 displays the superimposed image 80 in the first display area 72 of the display device 24, and if a sub-image 92 is generated, it displays the sub-image 92 in the third display area 90 (S35), and then terminates the series of processes shown in Figure 8. In this way, the superimposed image 80 displayed on the display device 24 is updated.

[0103] Furthermore, the noise reduction in step S21, the masking in step S22, the offset processing in step S23, the extraction in step S24, the display position correction in step S25, and the aircraft coordinate transformation in step S26 may be omitted as appropriate, for example, taking into account the trade-off between the accuracy of the superimposed image 80 and the processing load. Also, for example, if the resolution of the level image 84 is to be relatively coarse, the process of generating the sub-image 92 in step S34 may be omitted. In addition, although the first display area 72 was divided into a plurality of second display areas 74 in the above embodiment, the first display area 72 may not be divided, and a level image 84 may be generated and displayed for each measurement direction.

[0104] Embodiments of the present invention have been described above with reference to the attached drawings, but it goes without saying that the present invention is not limited to these embodiments. It is clear to those skilled in the art that various modifications or alterations can be conceived within the scope of the claims, and these will naturally also fall within the technical scope of the present invention.

[0105] Furthermore, the processes described herein do not necessarily have to be performed chronologically in the order shown in the flowchart; they may include parallel processing or processing by subroutines. [Explanation of Symbols]

[0106] 1. Obstacle detection device 2 aircraft 10 aircraft 20 cameras 22 Measuring device 24 Display device 26. Sound output device 30 Processing Unit 50 processors 52 memory 60 Imaging Result Processing Unit 62 Measurement Result Processing Unit 64 Image Processing Unit 66 Sound Processing Unit 72 1st display area 74 Second display area 80 superimposed images 82 Camera Images 84-level image 92 Sub-images

Claims

1. A camera that captures images in a predetermined direction from the aircraft's fuselage, A measuring device for measuring the distance of an obstacle from the aircraft in at least one or more measurement directions, including the imaging direction of the camera, A processing unit that performs processing based on the imaging results from the camera and the measurement results from the measuring device, A display device capable of displaying the processing results from the aforementioned processing device, Equipped with, The aforementioned processing apparatus is One or more processors, One or more memories connected to the processor, It has, The aforementioned processor, For each of the measurement directions, a threat level representing the likelihood of the obstacle contacting the aircraft is identified based on the distance information included in the measurement results. Based on the identified threat level, a level image that can identify the threat level is generated, A superimposed image is generated by superimposing the level image onto the image representing the aforementioned imaging result, The generated superimposed image is displayed on the display device, An obstacle detection device that performs processing including [specific actions].

2. The first display area in the aforementioned display device, in which the entire superimposed image is displayed, is divided into a plurality of second display areas. Each of the plurality of second display regions is associated with at least one of the measurement directions, The threat level is identified for each of the second display areas. The level image is capable of identifying the threat level for each of the second display areas. Obstacle detection device according to claim 1.

3. If there are multiple distance pieces of information in one second display area, the threat level in the second display area is determined based on the shortest distance piece of information among the multiple distance pieces of information. Obstacle detection device according to claim 2.

4. The aforementioned processor, Convert the distance information into distance information represented in an aircraft coordinate system, which is a coordinate system based on a specific position of the aircraft. Execute the process that includes, Based on the distance information represented in the aircraft coordinate system, the threat level is determined. Obstacle detection device according to claim 1.

5. The aforementioned processor, For each obstacle, the position information of the obstacle in a first plane that is perpendicular to the imaging direction and includes the position of the obstacle identified based on the measurement direction and the distance information is converted into two-dimensional position information on the two-dimensional screen of the display device. Execute the process that includes, For each converted two-dimensional location information, the threat level is identified based on the distance information corresponding to the converted two-dimensional location information. Obstacle detection device according to claim 1.

6. The aforementioned processor, If at least a portion of the superimposed image includes a level image in which the threat level is equal to or greater than a predetermined level, a sub-image representing the threat level equal to or greater than the predetermined level is generated. The sub-image is displayed outside the first display area in the display device where the entire superimposed image is displayed, An obstacle detection device according to claim 1, which performs a process including the following.

7. Equipped with an additional sound output device, The aforementioned processor, To generate an audio signal corresponding to the highest threat level among the threat levels for each measurement direction, The sound of the generated sound signal is output from the sound output device, An obstacle detection device according to claim 1, which performs a process including the following.

8. The aforementioned processor, Based on the difference between the position of the camera and the position of the measuring device, the position of the obstacle identified by the measurement results is corrected. An obstacle detection device according to claim 1, which performs a process including the following.

9. The measuring device measures the distance to the obstacle based on the reflected light intensity received in response to light transmitted in the measurement direction. The aforementioned processor, If the reflection intensity corresponding to the measurement direction is below a predetermined threshold, the distance information in the measurement direction is updated as specific information indicating the absence of an obstacle. An obstacle detection device according to claim 1, which performs a process including the following.

10. The aforementioned camera includes a fisheye lens, The aforementioned processor, To correct the distortion of the image captured by the aforementioned camera, Execute the process that includes, The superimposed image is generated by superimposing the level image onto the distortion-corrected image. Obstacle detection device according to claim 1.

Images