Control device, control method, and storage medium

Abstract

Provided are a control device, a control method, and a storage medium, which can superimpose information about an obstacle detected from an image of a vehicle's surroundings on the obstacle and display the information in a more comfortable manner. The control device includes an external recognition unit (55) that acquires an external recognition image representing a recognition result of the vehicle's (10) surroundings; an obstacle detection unit (57) that acquires obstacle information (81, 82) representing a detection result of an obstacle (71, 72) in the vehicle's surroundings; and a display control unit (56) that superimposes the obstacle information (81, 82) on the external recognition image and displays the information on a touch panel (42). The obstacle information (81, 82) represents a region in which the obstacle is detected in a plurality of fan-shaped detection regions (60) centered on the vehicle as viewed from above the vehicle, and the size of a central angle (θ1, θ2) of the fan-shaped detection region is set according to the distance of the fan-shaped detection region from the vehicle.

Description

[0001] This application is a divisional application of the invention patent application filed on February 21, 2023, with application number 202310152467.7 and title "Control Device, Control Method and Storage Medium". Technical Field

[0002] This invention relates to a control device, a control method, and a storage medium. Background Technology

[0003] In recent years, efforts to provide access to sustainable transportation systems that are considerate of vulnerable road users have become increasingly active. To achieve this goal, research and development are focused on further improving the safety and accessibility of transportation through the development of driver support technologies.

[0004] Previously, a technique was known to display images of the vehicle's surroundings captured by an onboard camera or similar device on a vehicle's display device when the vehicle was parked or moving at low speeds. Furthermore, a technique was known to overlay information related to obstacles detected by sensors onto the image of the vehicle's surroundings when displaying such an image. Patent Document 1 describes a top-down image generation apparatus that, when displaying a top-down image of the vehicle's surroundings, detects obstacles using sensors or similar devices and adjusts the display position of an icon representing an obstacle relative to an icon representing the vehicle based on the distance to the detected obstacle. Patent Document 2 describes an image display apparatus that, when an obstacle is detected by an obstacle detection unit, overlays obstacle information representing the detected obstacle onto a direct image captured by a camera and a top-down image of the vehicle's surroundings generated using that direct image, in the direction of obstacle detection.

[0005] Existing technical documents

[0006] Patent documents

[0007] Patent Document 1: Japanese Patent Application Publication No. 2020-182220

[0008] Patent Document 2: Japanese Patent Application Publication No. 2006-341641 Summary of the Invention

[0009] The problem that the invention aims to solve

[0010] In the overhead view image generation apparatus of Patent Document 1, a technique is described that varies the display position of obstacle information representing an obstacle based on the distance from the vehicle to the obstacle, but no description is provided of the respective observation methods for obstacle information displayed at different positions. Furthermore, the image display apparatus of Patent Document 2 can display an overhead image that reproduces obstacle information in a distorted form, and simultaneously display a direct image that represents obstacle information in a less distorted and more realistic form, but it lacks a description related to improving the observation method of obstacle information in the overhead image. Therefore, in driver support technology, there is still room for improvement in the method of displaying obstacle-related information overlaid on an image of the vehicle's surroundings.

[0011] The purpose of this invention is to provide a control device, control method, and storage medium capable of overlaying obstacle-related information detected in images surrounding a vehicle onto the obstacle and displaying it in a less inconvenient manner. This, in turn, contributes to the development of sustainable transportation systems.

[0012] Solution for solving the problem

[0013] This invention provides a control device, wherein the control device comprises:

[0014] The external recognition unit acquires an external recognition image representing the recognition results of the surroundings of the moving object;

[0015] An obstacle detection unit acquires obstacle information representing the detection results of obstacles surrounding the moving body; and

[0016] The display control unit overlays the obstacle information onto the external recognition image and displays it on the display device.

[0017] The obstacle information refers to the area where the obstacle was detected among multiple fan-shaped detection areas centered on the moving body when viewed from above the moving body.

[0018] The size of the central angle of the sector-shaped detection area is set according to the distance of the sector-shaped detection area from the moving body.

[0019] This invention provides a control method, wherein the control method performs the following processes through a control device:

[0020] Acquire an external recognition image representing the recognition results of the surroundings of the moving object;

[0021] Obtain obstacle information representing the detection results of obstacles surrounding the moving body; and

[0022] The obstacle information is overlaid on the external recognition image and displayed on a display device.

[0023] The obstacle information refers to the area where the obstacle was detected among multiple fan-shaped detection areas centered on the moving body when viewed from above the moving body.

[0024] The size of the central angle of the sector-shaped detection area is set according to the distance of the sector-shaped detection area from the moving body.

[0025] This invention provides a storage medium storing a control program, wherein the control program causes a processor of a control device to perform the following processes:

[0026] Acquire an external recognition image representing the recognition results of the surroundings of the moving object;

[0027] Obtain obstacle information representing the detection results of obstacles surrounding the moving body; and

[0028] The obstacle information is overlaid on the external recognition image and displayed on a display device.

[0029] The obstacle information refers to the area where the obstacle was detected among multiple fan-shaped detection areas centered on the moving body when viewed from above the moving body.

[0030] The size of the central angle of the sector-shaped detection area is set according to the distance of the sector-shaped detection area from the moving body.

[0031] Invention Effects

[0032] According to the present invention, a control device, control method, and storage medium can be provided that can overlay obstacle-related information detected in an image of the vehicle's surroundings onto the obstacle and display it in a less uncomfortable manner. Attached Figure Description

[0033] Figure 1 This is a side view showing an example of a vehicle whose movement is controlled by the control device of this embodiment.

[0034] Figure 2 yes Figure 1 The vehicle shown is shown in a top view.

[0035] Figure 3 It means Figure 1 The diagram shows a block diagram of the vehicle's internal structure.

[0036] Figure 4 This is a flowchart representing the display control of external recognition images and obstacle information performed by the computing unit.

[0037] Figure 5This is a diagram showing an example of a detection area that can be detected by the obstacle detection unit.

[0038] Figure 6 This is an example diagram showing obstacle information displayed on a front-view image.

[0039] Figure 7 This is an example diagram showing obstacle information displayed on an overhead image.

[0040] Explanation of reference numerals in the attached figures:

[0041] 10 vehicles (mobile vehicles)

[0042] 42 Touch panel (display device)

[0043] 55 External Identification Department

[0044] 56 Display Control Department

[0045] 57 Obstacle Recognition Department

[0046] 60 detection areas

[0047] Obstacles 71, 72, and 73

[0048] Obstacle information for 81, 82, and 83 Detailed Implementation

[0049] Hereinafter, an embodiment of the control device of the present invention will be described based on the accompanying drawings. The drawings are viewed along the direction of the symbols. Furthermore, in this specification and the like, for the sake of simplicity and clarity, the description follows the direction of the symbols. Figure 1 and Figure 2 The directions observed from the driver's position of the vehicle 10 shown are described as front-back, left-right, and up-down. In the accompanying drawings, the front of the vehicle 10 is denoted as Fr, the rear as Rr, the left as L, the right as R, the top as U, and the bottom as D.

[0050] <Vehicle 10 whose movement is controlled by the control device of the present invention>

[0051] Figure 1 This is a side view of a vehicle 10 whose movement is controlled by the control device of the present invention. Figure 2 yes Figure 1 The image shows a top view of vehicle 10. Vehicle 10 is an example of a mobile body according to the present invention.

[0052] Vehicle 10 is an automobile with a drive source (not shown) and wheels, the wheels including drive wheels driven by the power of the drive source and steering wheels capable of steering. In this embodiment, vehicle 10 is a four-wheeled automobile with a pair of left and right front wheels and a pair of left and right rear wheels. The drive source of vehicle 10 is, for example, an electric motor. Alternatively, the drive source of vehicle 10 can also be an internal combustion engine such as a gasoline engine or a diesel engine, or a combination of an electric motor and an internal combustion engine. Furthermore, the drive source of vehicle 10 can drive the pair of left and right front wheels, the pair of left and right rear wheels, or all four wheels. Both the front wheels and the rear wheels can be steering wheels capable of steering, or only one of them can be a steering wheel capable of steering.

[0053] The vehicle 10 also includes side rearview mirrors 11L and 11R. The side rearview mirrors 11L and 11R are mirrors (rearview mirrors) installed on the outside of the front doors of the vehicle 10 to allow the driver to see behind and to the sides. The side rearview mirrors 11L and 11R are each fixed to the main body of the vehicle 10 by a rotating axis extending in a vertical direction, and can be opened and closed by rotating around the rotating axis.

[0054] Vehicle 10 also includes a front camera 12Fr, a rear camera 12Rr, a left-side camera 12L, and a right-side camera 12R. The front camera 12Fr is a digital camera positioned in front of vehicle 10 to capture images of the front of vehicle 10. The rear camera 12Rr is a digital camera positioned behind vehicle 10 to capture images of the rear of vehicle 10. The left-side camera 12L is mounted on the left-side rearview mirror 11L of vehicle 10 to capture images of the left side of vehicle 10. The right-side camera 12R is mounted on the right-side rearview mirror 11R of vehicle 10 to capture images of the right side of vehicle 10.

[0055] <Internal structure of vehicle 10>

[0056] Figure 3 It means Figure 1 A block diagram illustrating an example of the internal structure of vehicle 10. (See diagram for reference.) Figure 3 As shown, vehicle 10 includes a sensor group 16, a navigation device 18, a control ECU (Electronic Control Unit) 20, an EPS (Electric Power Steering) system 22, and a communication unit 24. Vehicle 10 also includes a drive force control system 26 and a braking force control system 28.

[0057] Sensor group 16 acquires various detection values ​​used by the control ECU 20 for control. Sensor group 16 includes a front camera 12Fr, a rear camera 12Rr, a left-side camera 12L, and a right-side camera 12R. Furthermore, sensor group 16 includes a front sonar group 32a, a rear sonar group 32b, a left-side sonar group 32c, and a right-side sonar group 32d. Additionally, sensor group 16 includes wheel sensors 34a and 34b, a vehicle speed sensor 36, and an operation detection unit 38. Alternatively, sensor group 16 may also include radar.

[0058] The front camera 12Fr, rear camera 12Rr, left camera 12L, and right camera 12R acquire recognition data (e.g., external recognition images) for identifying the surroundings of the vehicle 10 by capturing images of the vehicle's perimeter. The external recognition image captured by the front camera 12Fr is called the front recognition image. The external recognition image captured by the rear camera 12Rr is called the rear recognition image. The external recognition image captured by the left camera 12L is called the left-side recognition image. The external recognition image captured by the right camera 12R is called the right-side recognition image. The image composed of the left-side and right-side recognition images is also called the side recognition image. The external recognition image generated by synthesizing the data captured by the front camera 12Fr, rear camera 12Rr, left camera 12L, and right camera 12R is also called the overhead view image of the vehicle 10.

[0059] The front sonar group 32a, rear sonar group 32b, left sonar group 32c, and right sonar group 32d emit sound waves to the periphery of the vehicle 10 and receive reflected sounds from other objects. The front sonar group 32a, for example, contains four sonars. The sonars constituting the front sonar group 32a are respectively positioned at the left diagonally forward, the left front, the right front, and the right diagonally forward of the vehicle 10. The rear sonar group 32b, for example, contains four sonars. The sonars constituting the rear sonar group 32b are respectively positioned at the left diagonally rear, the left rear, the right rear, and the right diagonally rear of the vehicle 10. The left sonar group 32c, for example, contains two sonars. The sonars constituting the left sonar group 32c are respectively positioned at the front left side and the rear left side of the vehicle 10. The right sonar group 32d, for example, contains two sonars. The sonars constituting the right sonar group 32d are respectively positioned at the front right side and the rear right side of the vehicle 10. The front sonar group 32a, the rear sonar group 32b, the left sonar group 32c, and the right sonar group 32d acquire detection data (e.g., obstacle information) for detecting obstacles around the vehicle 10.

[0060] Wheel sensors 34a and 34b detect the rotation angle of the wheels of vehicle 10. Wheel sensors 34a and 34b can be constructed from angle sensors or displacement sensors. Wheel sensors 34a and 34b output detection pulses whenever the wheels rotate a predetermined angle. The detection pulses output from wheel sensors 34a and 34b are used to calculate the wheel rotation angle and the wheel rotation speed. The distance traveled by vehicle 10 is calculated based on the wheel rotation angle. For example, wheel sensor 34a detects the rotation angle θa of the left rear wheel. Wheel sensor 34b, for example, detects the rotation angle θb of the right rear wheel.

[0061] Vehicle speed sensor 36 detects the speed of the vehicle body 10, i.e., vehicle speed V, and outputs the detected vehicle speed V to control ECU 20. Vehicle speed sensor 36 detects vehicle speed V, for example, based on the rotation of the transmission countershaft.

[0062] The operation detection unit 38 detects the content of the operation performed by the user using the operation input unit 14 and outputs the detected operation content to the control ECU 20. The operation input unit 14 includes, for example, various user interfaces such as a side mirror switch for switching the opening and closing state of the side rearview mirrors 11L and 11R, and a gear lever (shift lever, gear shift lever).

[0063] The navigation device 18 uses, for example, GPS (Global Positioning System) to detect the current location of the vehicle 10 and guides the user along the route to the destination. The navigation device 18 has a storage device (not shown) containing a map information database.

[0064] The navigation device 18 includes a touch panel 42 and a speaker 44. The touch panel 42 functions as an input device and display device for controlling the ECU 20. The speaker 44 outputs various guidance information to the user of the vehicle 10 through sound.

[0065] The touch panel 42 is configured to allow input of various commands to the control ECU 20. For example, a user can input a command via the touch panel 42 to display an external recognition image of the vehicle 10. Furthermore, the touch panel 42 is configured to display various screens related to the control content of the control ECU 20. For example, an external recognition image of the vehicle 10 can be displayed on the touch panel 42. Alternatively, components other than the touch panel 42, such as a HUD (Head-Up Display), smartphone, or tablet computer, can also be used as input or display devices.

[0066] The control ECU 20 includes an input / output unit 50, an arithmetic unit 52, and a storage unit 54. The arithmetic unit 52 is, for example, a CPU (Central Processing Unit). The arithmetic unit 52 controls various parts based on the program stored in the storage unit 54, thereby performing various controls. Furthermore, the arithmetic unit 52 exchanges input and output signals with various parts connected to the control ECU 20 via the input / output unit 50.

[0067] The arithmetic unit 52 includes: an external recognition unit 55 for recognizing external recognition images; a display control unit 56 for controlling the display of external recognition images; and an obstacle detection unit 57 for recognizing obstacles around the vehicle 10. The arithmetic unit 52 is an example of the control device of the present invention.

[0068] The external recognition unit 55 acquires an external recognition image from each camera, representing the recognition results of the surrounding images of the vehicle 10 captured by the front camera 12Fr, the rear camera 12Rr, the left camera 12L, and the right camera 12R.

[0069] The obstacle detection unit 57 acquires obstacle information from each sonar, representing the detection results of obstacles around the vehicle 10 detected by the front sonar group 32a, rear sonar group 32b, left sonar group 32c, and right sonar group 32d. For example, when the vehicle 10 is parked or leaving a parking space, the obstacle detection unit 57 acquires obstacle information from each sonar, representing the detection results of obstacles detected around the vehicle 10. Obstacles include objects that may impede the movement of the vehicle 10, such as people, animals, other vehicles, vegetation, etc. In addition, the parking or leaving of the vehicle 10 can be based on autonomous driving, driving assistance, or manual driving. Furthermore, in addition to sonar, the detection equipment for detecting obstacles around the vehicle 10 can also be, for example, LiDAR, radar, cameras 12Fr, 12Rr, 12L, 12R, etc.

[0070] The display control unit 56 displays the external recognition images acquired by the external recognition unit 55 on the display device of the vehicle 10. Specifically, the display control unit 56 displays the front recognition image captured by the front camera 12Fr, the rear recognition image captured by the rear camera 12Rr, the left-side recognition image captured by the left-side camera 12L, and the right-side recognition image captured by the right-side camera 12R on the touch panel 42. In addition, the display control unit 56 displays a top-down image of the vehicle 10 generated by combining the data captured by the front camera 12Fr, the rear camera 12Rr, the left-side camera 12L, and the right-side camera 12R on the touch panel 42.

[0071] Furthermore, the display control unit 56 overlays obstacle information onto the external recognition image and displays it on the touch panel 42. For example, the display control unit 56 overlays obstacle information onto the front recognition image, the rear recognition image, the left side recognition image, the right side recognition image, and the top-down image and displays it on the touch panel 42.

[0072] Specifically, when the user of vehicle 10 makes a specified input, the display control unit 56 overlays obstacle information onto the external recognition image and displays it on the touch panel 42. The specified input includes, for example, operation input from the touch panel 42, and switching operations. Furthermore, when the vehicle 10 is in a specified movement state, the display control unit 56 overlays obstacle information onto the external recognition image and displays it on the touch panel 42. Specified movement states include, for example, situations where the vehicle 10's movement speed is below a specified speed (e.g., 15 km / h), situations where the vehicle 10 has been identified as having moved into a specified parking area based on GPS information, external recognition images, etc., and situations where the vehicle 10's gears have moved to a specified position.

[0073] The EPS system 22 includes a steering angle sensor 100, a torque sensor 102, an EPS motor 104, a rotary transformer 106, and an EPS ECU 108. The steering angle sensor 100 detects the steering angle θst of the steering unit 110. The torque sensor 102 detects the torque TQ applied to the steering unit 110.

[0074] The EPS motor 104 can support the occupant's operation of the steering device 110 by applying driving or reaction force to the steering column 112 connected to the steering device 110, or perform automatic steering when the vehicle is parked. The rotary transformer 106 detects the rotation angle θm of the EPS motor 104. The EPSECU 108 controls the entire EPS system 22. The EPSECU 108 includes an input / output unit (not shown), an arithmetic unit (not shown), and a storage unit (not shown).

[0075] The communication unit 24 is capable of wireless communication with other communication devices 120. Other communication devices 120 refer to base stations, communication devices of other vehicles, smartphones or tablets carried by users of vehicle 10, etc.

[0076] The drive force control system 26 includes a drive ECU 130. The drive force control system 26 performs drive force control of the vehicle 10. The drive ECU 130 controls the engine (not shown) and other components based on user operations on the accelerator pedal (not shown), thereby controlling the drive force of the vehicle 10.

[0077] The braking force control system 28 includes a braking ECU 132. The braking force control system 28 performs braking force control of the vehicle 10. The braking ECU 132 controls the braking mechanism (not shown) and other components based on the user's operation of the brake pedal (not shown), thereby controlling the braking force of the vehicle 10.

[0078] <Control Example of Arithmetic Unit 52>

[0079] Next, refer to Figure 4 The flowchart shown illustrates an example of the display control of external recognition images and obstacle information performed by the arithmetic unit 52 of vehicle 10.

[0080] For example, a user driving vehicle 10 wants to park vehicle 10 in a parking space in a parking lot. Vehicle 10 may be in a low-speed driving state (e.g., below 15 km / h) or a stationary state. For example, when the driving state of vehicle 10 is detected based on the detection value of vehicle speed sensor 36, the processing unit 52 of vehicle 10 begins... Figure 4 The processing is shown.

[0081] The display control unit 56 of the arithmetic unit 52 determines whether to display the external recognition image acquired by the external recognition unit 55 on the touch panel 42 of the navigation device 18 (step S11). For example, when the stop button (illustration omitted) displayed on the touch panel 42 is clicked, the external recognition image is displayed. Therefore, the display control unit 56 determines, for example, whether the user has clicked the stop button.

[0082] In step S11, if no external recognition image is displayed on the touch panel 42, that is, if the parking button is not clicked (step S11: no), the display control unit 56 repeatedly performs the processing of step S11 and stands by until the parking button is clicked.

[0083] In step S11, if an external recognition image is displayed on the touch panel 42, i.e., if the parking button has been clicked (step S11: Yes), the display control unit 56 displays the external recognition image on the touch panel 42 (step S12). Additionally, for example, if the vehicle 10's gear is set to drive, neutral, or park, the display control unit 56 displays a forward recognition image captured by the front camera 12Fr on the touch panel 42. Furthermore, for example, if the vehicle 10's gear is set to reverse, the display control unit 56 displays a rear recognition image captured by the rear camera 12Rr on the touch panel 42. Moreover, if a top-down view is selected by the user, the display control unit 56 displays a top-down view on the touch panel 42.

[0084] Next, the obstacle detection unit 57 of the arithmetic unit 52 determines whether an obstacle has been detected around the vehicle 10 (step S13). As described above, obstacles around the vehicle 10 can be detected based on obstacle information acquired by the sonar arrays 32a to 32d. The obstacle information includes, for example, information related to the direction of the obstacle relative to the vehicle 10, information related to the distance from the vehicle 10 to the obstacle, and information related to which area of ​​a pre-defined detection area the obstacle was detected in. The pre-defined detection area will be... Figure 5 This will be discussed later.

[0085] In step S13, if no obstacle is detected around the vehicle 10 (step S13: No), the obstacle detection unit 57 repeatedly performs the processing of step S13 and stands by until an obstacle is detected.

[0086] In step S13, if an obstacle is detected around the vehicle 10 (step S13: Yes), the obstacle detection unit 57 outputs obstacle information related to the detected obstacle to the display control unit 56. The display control unit 56 overlays the obstacle information output from the obstacle detection unit 57 onto the external recognition image and displays it on the touch panel 42 (step S14). As described in step S12, the external recognition image includes a front recognition image, a rear recognition image, a top-down image, etc. The obstacle information displayed overlaid on the external recognition image will be... Figure 6 , 7 This will be discussed later.

[0087] Next, the display control unit 56 determines whether the external recognition image displayed on the touch panel 42 is hidden (step S15). For example, when the parking complete button (illustration omitted) displayed on the touch panel 42 is clicked, the external recognition image is hidden. Therefore, the display control unit 56 determines, for example, whether the user has clicked the parking complete button.

[0088] In step S15, without hiding the external recognition image, that is, without clicking the parking completion button (step S15: No), the display control unit 56 returns to step S13 and repeats each process.

[0089] In step S15, if the external recognition image is hidden, that is, if the parking completion button is clicked (step S15: Yes), the external recognition image displayed by the display control unit 56 on the touch panel 42 is hidden (step S16), and the process ends.

[0090] <Detection area and obstacle information>

[0091] Next, refer to Figure 5The detection area used to detect obstacles and the obstacle information representing the detection results are explained.

[0092] Figure 5 This is a diagram showing the detection area 60 formed around vehicle 10 when viewed from above. (See diagram below.) Figure 5 As shown, the detection area 60 is a region centered on the detection equipment (e.g., sonar groups 32a to 32d) mounted on the vehicle 10, and is formed to a range capable of detecting obstacles. For example, the detection area 60 includes a forward detection area 61 centered on the forward sonar group 32a, a rear detection area 62 centered on the rear sonar group 32b, a left-side detection area 63 centered on the left-side sonar group 32c, and a right-side detection area 64 centered on the right-side sonar group 32d.

[0093] The forward detection area 61 is a detection area composed of multiple fan-shaped areas centered on the forward sonar group 32a. The rear detection area 62 is a detection area composed of multiple fan-shaped areas centered on the rear sonar group 32b. The fan-shaped detection areas are included in the travel direction of the vehicle 10, that is, within the range of the front (Fr) and rear (Rr) of the vehicle 10. Furthermore, the left-side detection area 63 is a detection area composed of multiple rectangles centered on the left-side sonar group 32c. The right-side detection area 64 is a detection area composed of multiple rectangles centered on the right-side sonar group 32d. The rectangular detection areas are included in the width direction of the vehicle 10, that is, within the range of the left (L) and right (R) of the vehicle 10.

[0094] The front sonar group 32a and the rear sonar group 32b are capable of detecting obstacles within a detection distance LV in a fan-shaped detection range along the travel direction of the vehicle 10. The left sonar group 32c and the right sonar group 32d are capable of detecting obstacles within a rectangular detection range along the width direction of the vehicle 10, along a detection distance LH. The detection distance LV in the travel direction is configured to be longer than the detection distance LH in the width direction.

[0095] The front detection area 61, composed of multiple fan-shaped structures, includes an inner detection area group 61a disposed on the side closest to the vehicle 10 and an outer detection area group 61b disposed on the side farther from the vehicle 10 than the inner detection area group 61a. The inner detection area group 61a is an example of the first detection area group of the present invention. The outer detection area group 61b is an example of the second detection area group of the present invention. Furthermore, the front detection area 61 and the rear detection area 62 have the same structure; therefore, in the following description, the front detection area 61 will be the focus of the explanation.

[0096] The size of the central angle of the multiple sector-shaped detection areas is set according to the distance from the vehicle 10 to the detection area. Specifically, the size of the central angle of the sector in the detection area located away from the vehicle 10 is smaller than the size of the central angle of the sector in the detection area located closer to the vehicle 10. In the illustrated example, the inner detection area group 61a consists of one sector-shaped inner detection area portion 61a1, while the outer detection area group 61b is divided into two sectors, consisting of outer detection area portions 61b1 and 61b2. Therefore, the size of the central angle θ2 of the outer detection area portions 61b1 and 61b2 is smaller than the size of the central angle θ1 of the inner detection area portion 61a1. Furthermore, the number of outer detection area portions is greater than the number of inner detection area portions.

[0097] In the detection area configured in this way, the sonar array of vehicle 10 detects which sector or rectangle an obstacle around vehicle 10 is located within. Sonar arrays 32a to 32d detect the direction and distance of obstacles based on sound waves emitted towards the periphery of vehicle 10 and reflected sound received from obstacles, and detect the position of the sector or rectangle containing the obstacle based on the detected direction and distance. The position of the detected obstacle is displayed as obstacle information within the area where the obstacle was detected, representing the detection result. For example, when an obstacle is detected in the forward detection area 61 or the rear detection area 62, the obstacle information is displayed in a manner corresponding to the size of the central angle of the sector of the area where the obstacle was detected.

[0098] Specifically, in Figure 5 In the front right side of the vehicle 10, the obstacle 71 is detected within the fan-shaped area of ​​the outer detection area 61b2 in the front detection area 61. Therefore, the obstacle information for obstacle 71 is displayed as, for example, strip-shaped obstacle information 81 extending corresponding to the central angle θ2 of the fan-shaped area of ​​the outer detection area 61b2. Furthermore, the obstacle 72 located at the rear right side of the vehicle 10 is detected within the fan-shaped area of ​​the inner detection area 61a1 in the rear detection area 62. Therefore, the obstacle information for obstacle 72 is displayed as, for example, strip-shaped obstacle information 82 extending corresponding to the central angle θ1 of the fan-shaped area of ​​the inner detection area 61a1.

[0099] <Obstacle information displayed on the external recognition image>

[0100] Next, refer to Figure 6 and Figure 7 An example is given of obstacle information displayed overlaid on an external recognition image.

[0101] Figure 6This is a diagram illustrating an example of a forward recognition image captured by the front camera 12Fr of vehicle 10 and obstacle information displayed overlaid on the forward recognition image. Figure 6 For example, vehicle 10 is in a situation where it is looking for a parking space in a parking lot.

[0102] like Figure 6 As shown, a forward recognition image 91 captured by the front camera 12Fr is displayed on the touch panel 42. In the forward recognition image 91, a person (obstacle) 73 is captured in the front right of the vehicle 10. The direction and distance of the person 73 relative to the vehicle 10 are detected by the front sonar group 32a, and the position of the fan-shaped area in which the person 73 is located is also detected. The position of the detected fan-shaped area is then compared with the central angle of the fan-shaped area (reference). Figure 5 The corresponding obstacle information is displayed as strip obstacle information 83 overlaid on the position of person 73 in the forward recognition image 91. In addition, when the user of vehicle 10 notices person 73 and steps on the brake to stop, a message 92b such as "Please check the surroundings directly" is displayed on the forward recognition image 91 of touch panel 42.

[0103] Figure 7 It means based on inclusion Figure 6 An example diagram showing an overhead image generated from the data captured by each camera 12Fr, 12Rr, 12L, 12R in the frontal recognition image, and obstacle information superimposed on the overhead image. Figure 7 Vehicle 10 and Figure 6 They are in the same situation, searching for a parking space for vehicle 10 in a parking lot.

[0104] like Figure 7 As shown, a top-down image 92, generated based on the shooting data from each camera 12Fr, 12Rr, 12L, and 12R, is displayed on the touch panel 42. In the top-down image 92, [the image is displayed alongside...] Figure 6 Similarly, a person (obstacle) 73 was photographed from the right front of vehicle 10. Furthermore, based on the direction and distance of person 73 relative to vehicle 10 detected by the front sonar group 32a, the position of the fan-shaped area where person 73 is located is detected. The position of the detected fan-shaped area is then compared with the central angle of that fan-shaped area (refer to...). Figure 5 The corresponding strip obstacle information 83 is overlaid on the position of the character 73. Additionally, the overhead image 92 can also be displayed alongside... Figure 6 The front recognition images 91 shown are displayed side by side on the touch panel 42.

[0105] However, the conventional structure of the detection area formed around the vehicle 10 is, for example, in Figure 5In the detection area 60 shown, the central angle θ1 of the inner detection area portion 61a1 in the inner detection area group 61a of the front detection area 61 and the rear detection area 62 is the same as the central angle θ2 of the outer detection area portion 61b1 in the outer detection area group 61b (θ1=θ2). That is, the outer detection area portion in the outer detection area group 61b is formed as a single outer detection area portion. Therefore, when a detection is made... Figure 5 When there are 72 obstacles in the image, the image is overlaid with the image. Figure 5 The obstacle information 82 shown is the same as the strip obstacle information 81, but when obstacle 71 is detected, a strip obstacle information extending corresponding to the central angle θ1=θ2, such as the strip obstacle information 181 shown by the dashed line, is superimposed. Therefore, the more the strip obstacle information overlaps with distant obstacles, the longer the strip obstacle information is in the arc direction, and the larger it is displayed relative to the obstacles displayed as actual camera images, causing discomfort.

[0106] In contrast, when the obstacle information 83, representing the detection result of the person (obstacle) 73, is overlaid on the external recognition image (front recognition image 91, top-down image 92) for display, the display control unit 56 of the calculation unit 52 in this embodiment, for example, Figure 5 As shown, when viewed from above the vehicle 10, obstacle information 81 and 82 are displayed in the areas (inner detection area 61a1 and outer detection area 61b2) of multiple fan-shaped detection areas 60 centered on the vehicle 10 where obstacles 71 and 72 are detected. Furthermore, the size of the central angle of each fan-shaped detection area 60 is set according to its distance from the vehicle 10. With this structure, obstacle information 81 and 82 can be displayed within an accurate range regardless of the distance between the vehicle 10 and the obstacles 71 and 72. This suppresses the discomfort caused by the obstacle information being too large relative to the actual camera image, thus improving the safety of the vehicle 10.

[0107] Furthermore, the front detection area 61 and rear detection area 62, which are composed of multiple fan-shaped structures, include an inner detection area group 61a disposed on the side closer to the vehicle 10 and an outer detection area group 61b disposed on the side farther away from the vehicle 10 than the inner detection area group 61a. Moreover, the size of the central angle θ2 of the outer detection area portions 61b1 and 61b2 in the outer detection area group 61b is smaller than the size of the central angle θ1 of the inner detection area portion 61a1 in the inner detection area group 61a. Therefore, the outer detection area portions 61b1 and 61b2 of the outer detection area group 61b, which are farther from the vehicle 10, are configured more precisely than the inner detection area portion 61a1 of the inner detection area group 61a, which is closer to the vehicle 10. This prevents the obstacle range indicated by obstacle information from becoming too wide relative to the actual size of the obstacle, even when the obstacle is located at a distance, thereby improving the safety of the vehicle 10.

[0108] Furthermore, the detection range LV, which enables the detection of obstacles in the direction of travel of the vehicle 10 via the front sonar group 32a and the rear sonar group 32b, is greater than the detection range LH, which enables the detection of obstacles in the width direction of the vehicle 10 via the left sonar group 32c and the right sonar group 32d. Moreover, the fan-shaped detection area encompasses the range in the direction of travel of the vehicle 10, specifically the area within the front (Fr) and rear (Rr) of the vehicle 10. Therefore, by setting a longer detection range LV in the direction of travel of the vehicle 10, obstacles at a distance can be detected earlier. This reduces the risk of contact with obstacles while the vehicle 10 is moving, thus improving the safety of the vehicle 10.

[0109] Furthermore, when a user of vehicle 10 inputs information, for example, through the touch panel 42, the display control unit 56 overlays obstacle information onto the external recognition image and displays it on the touch panel 42. This allows obstacle information to be overlaid onto the external recognition image when needed, thus improving the safety of vehicle 10.

[0110] Furthermore, when the vehicle 10's moving speed is below a specified speed, or when it is detected that the vehicle 10 has moved into a designated parking area, the display control unit 56 overlays obstacle information onto the external recognition image and displays it on the touch panel 42. This allows obstacle information to be overlaid onto the external recognition image when needed, improving the safety of the moving vehicle.

[0111] The embodiments of the present invention have been described above, but the present invention is not limited to the above embodiments and can be appropriately modified or improved.

[0112] For example, in the above embodiment, the case of parking vehicle 10 was described, but the same control can be performed when vehicle 10 leaves the warehouse.

[0113] Furthermore, the control method described in the above embodiments can be implemented by a computer executing a pre-prepared control program. This control program is recorded in a computer-readable storage medium and executed by reading it from the storage medium. Additionally, this control program can be stored in a non-transitory storage medium such as flash memory and provided in this form, or it can be provided via a network such as the Internet. The computer executing this control program can be included in a control device, or in an electronic device such as a smartphone, tablet computer, or personal computer capable of communicating with the control device, or in a server device capable of communicating with these control devices and electronic devices.

[0114] In addition, at least the following items are described in this specification. Furthermore, the components corresponding to the above embodiments are shown in parentheses, but this is not a limitation.

[0115] (1) A control device, wherein the control device comprises: an external identification unit (external identification unit 55) which acquires an external identification image representing the identification result of the surroundings of a moving body (vehicle 10);

[0116] An obstacle detection unit (obstacle detection unit 57) acquires obstacle information (obstacle information 81, 82) representing the detection results of obstacles (obstacles 71, 72) around the moving body; and

[0117] The display control unit (display control unit 56) overlays the obstacle information onto the external recognition image and displays it on the display device (touch panel 42).

[0118] The obstacle information refers to the area in a plurality of fan-shaped detection areas (detection areas 60) centered on the moving body when viewed from above the moving body, where the obstacle was detected.

[0119] The size of the central angle of the sector-shaped detection area is set according to the distance of the sector-shaped detection area from the moving body.

[0120] According to (1), regardless of the distance between the moving body and the obstacle, obstacle information can be displayed within an accurate range, thereby improving the safety of the moving body.

[0121] (2) The control device according to (1), wherein,

[0122] The plurality of sector-shaped detection areas include: a first detection area group; and a second detection area group, which is farther from the moving body than the first detection area group and whose central angle of the sector is smaller than that of the first detection area group.

[0123] According to (2), the detection areas of the second detection area group that is far away from the moving body are configured more finely than the detection areas of the first detection area group that is close to the moving body. Therefore, even when the obstacle is far away, it can prevent the range of the obstacle represented by the obstacle information from becoming too wide relative to the actual size of the obstacle, thereby improving the safety of the moving body.

[0124] (3) The control device according to (1) or (2), wherein,

[0125] The obstacle detection unit has a longer detection distance in the direction of travel of the moving body than the obstacle detection unit has a longer detection distance in the width direction of the moving body.

[0126] The multiple sector-shaped detection areas are contained within the travel direction of the moving body.

[0127] According to (3), by setting the detection distance of the range of the moving direction of the mobile body to be longer, obstacles in the distance can be detected in advance, which can reduce the risk of contact with obstacles during the movement of the mobile body and improve the safety of the mobile body.

[0128] (4) The control device according to any one of (1) to (3), wherein,

[0129] When the user of the moving body makes a specified input, the display control unit overlays the obstacle information onto the external recognition image and displays it on the display device.

[0130] According to (4), obstacle information can be overlaid in scenarios where it is necessary to display obstacle information, thus improving the safety of moving objects.

[0131] (5) The control device according to any one of (1) to (4), wherein,

[0132] When the movement state of the moving body is in a predetermined state, the display control unit overlays the obstacle information onto the external recognition image and displays it on the display device.

[0133] According to (5), obstacle information can be overlaid in scenarios where it is necessary to display obstacle information, thereby improving the safety of moving objects.

[0134] (6) A control method, wherein the control method performs the following processing via a control device:

[0135] Acquire an external recognition image representing the recognition results of the surroundings of the moving object;

[0136] Obtain obstacle information representing the detection results of obstacles surrounding the moving body; and

[0137] The obstacle information is overlaid on the external recognition image and displayed on a display device.

[0138] The obstacle information refers to the area where the obstacle was detected among multiple fan-shaped detection areas centered on the moving body when viewed from above the moving body.

[0139] The size of the central angle of the sector-shaped detection area is set according to the distance of the sector-shaped detection area from the moving body.

[0140] According to (6), regardless of the distance between the moving body and the obstacle, obstacle information can be displayed within an accurate range, thereby improving the safety of the moving body.

[0141] (7) A storage medium storing a control program, wherein the control program causes a processor of a control device to perform the following processes:

[0142] Acquire an external recognition image representing the recognition results of the surroundings of the moving object;

[0143] Obtain obstacle information representing the detection results of obstacles surrounding the moving body; and

[0144] The obstacle information is overlaid on the external recognition image and displayed on a display device.

[0145] The obstacle information refers to the area where the obstacle was detected among multiple fan-shaped detection areas centered on the moving body when viewed from above the moving body.

[0146] The size of the central angle of the sector-shaped detection area is set according to the distance of the sector-shaped detection area from the moving body.

[0147] According to (7), regardless of the distance between the moving body and the obstacle, obstacle information can be displayed within an accurate range, thereby improving the safety of the moving body.

Claims

1. A control device, wherein, The control device includes: The external recognition unit acquires an external recognition image representing the recognition results of the surroundings of the moving object; An obstacle detection unit acquires obstacle information representing the detection results of obstacles around the moving body; as well as The display control unit displays the obstacle information on the external recognition image and displays the external recognition image on the display device. The obstacle information corresponds to a detection area in a plurality of detection areas that detect the obstacle when viewed from above the moving body, having a predetermined central angle centered on the moving body. The plurality of detection areas includes: a first detection area group; and a second detection area group, which is farther from the moving body than the first detection area group and has a smaller central angle than the first detection area group. The obstacle information is an image that extends in relation to the central corner of the detection area where the obstacle is detected.

2. The control device according to claim 1, wherein, The obstacle detection unit has a longer detection distance in the direction of travel of the moving body than the obstacle detection unit has a longer detection distance in the width direction of the moving body. The plurality of detection areas are contained within the travel direction of the moving body.

3. The control device according to claim 1 or 2, wherein, When the user of the moving body makes a specified input, the display control unit displays the obstacle information on the external recognition image and displays the external recognition image on the display device.

4. The control device according to claim 1 or 2, wherein, When the movement state of the moving body is in a predetermined state, the display control unit displays the obstacle information on the external recognition image and displays the external recognition image on the display device.

5. A control method, wherein, The control method performs the following processes through a control device: Acquire an external recognition image representing the recognition results of the surroundings of the moving object; Obtain obstacle information representing the detection results of obstacles surrounding the moving body; and The obstacle information is displayed on the external recognition image and the external recognition image is displayed on the display device. The obstacle information corresponds to a detection area in a plurality of detection areas that detect the obstacle when viewed from above the moving body, having a predetermined central angle centered on the moving body. The plurality of detection areas includes: a first detection area group; and a second detection area group, which is farther from the moving body than the first detection area group and has a smaller central angle than the first detection area group. The obstacle information is an image that extends in relation to the central corner of the detection area where the obstacle is detected.

6. A storage medium storing a control program, wherein, The control program causes the processor of the control device to perform the following processes: Acquire an external recognition image representing the recognition results of the surroundings of the moving object; Obtain obstacle information representing the detection results of obstacles surrounding the moving body; and The obstacle information is displayed on the external recognition image and the external recognition image is displayed on the display device. The obstacle information corresponds to a detection area in a plurality of detection areas that detect the obstacle when viewed from above the moving body, having a predetermined central angle centered on the moving body. The plurality of detection areas includes: a first detection area group; and a second detection area group, which is farther from the moving body than the first detection area group and has a smaller central angle than the first detection area group. The obstacle information is an image that extends in relation to the central corner of the detection area where the obstacle is detected.

Images