Information processing device, information processing method, program and projection device

Abstract

An information processing device related to one form of the present technology comprises an acquisition unit and a projection control unit. The acquisition unit acquires ambient environment information regarding the surrounding environment of a vehicle. Based on the ambient environment information, the projection control unit controls the display of a projection pattern projected onto the road surface surrounding the vehicle from a projection unit mounted on the vehicle. This makes it possible to enhance safety during driving.

Description

Technology Field

[0001] The present invention relates to an information processing device, an information processing method, a program, and a projection device applicable to the display control of a projection pattern projected from a vehicle onto a road surface. Background Technology

[0002] Patent Document 1 describes a predicted driving trajectory display device that displays the predicted driving trajectory of a vehicle on the ground surface. In this device, the predicted driving trajectory of a vehicle is calculated from the steering angle of the vehicle's steering wheel and forward / backward information indicating the vehicle's forward and backward movement. Then, the irradiation angle of a laser emitter mounted on the vehicle is controlled so that the predicted driving trajectory is drawn on the ground surface (paragraphs

[0021]

[0022]

[0023] of the specification of Patent Document 1, FIG. 7, etc.). Prior art literature

[0003] Japanese Patent Publication No. 2006-036005 The problem to be solved

[0004] In this way, by indicating the vehicle's direction of travel, it is possible to prompt attention both inside and outside the vehicle. In actual traffic environments, however, it is necessary to pay attention to various objects in addition to the vehicle's direction of travel, so technology capable of enhancing safety during driving is required.

[0005] Taking into account the circumstances described above, the objective of the present technology is to provide an information processing device, an information processing method, a program, and a projection device capable of enhancing safety during operation. means of solving the problem

[0006] To achieve the above objective, an information processing device related to one form of the present technology comprises an acquisition unit and a projection control unit.

[0007] The above acquisition unit acquires surrounding environment information regarding the environment surrounding the vehicle.

[0008] The projection control unit above controls the display of a projection pattern projected onto the road surface surrounding the vehicle from a projection unit mounted on the vehicle based on the surrounding environment information.

[0009] In this information processing device, a projection pattern is projected onto the surrounding road surface from a projection unit provided in the vehicle. The display of this projection pattern is controlled based on surrounding environment information regarding the environment surrounding the vehicle. As a result, it becomes possible to present, for example, the situation in which the vehicle is positioned to the inside and outside of the vehicle through the projection pattern, thereby enhancing safety during driving.

[0010] An information processing method related to one form of the present technology is an information processing method executed by a computer system and includes acquiring surrounding environment information regarding the surrounding environment of a vehicle.

[0011] Based on the above surrounding environment information, the display of a projection pattern projected onto the road surface surrounding the vehicle from a projection unit mounted on the vehicle is controlled.

[0012] A program related to one form of the present technology executes the following steps on a computer system.

[0013] A step for acquiring surrounding environment information regarding the environment surrounding the vehicle.

[0014] A step of controlling the display of a projection pattern projected onto the road surface surrounding the vehicle from a projection unit mounted on the vehicle, based on the surrounding environment information.

[0015] A projection device related to one form of the present technology comprises a projection unit, an acquisition unit, and a projection control unit.

[0016] The above projection unit is mounted on a vehicle and projects a projection pattern onto the road surface surrounding the vehicle.

[0017] The above acquisition unit acquires surrounding environment information regarding the surrounding environment of the above vehicle.

[0018] The projection control unit controls the display of the projection pattern projected from the projection unit based on the surrounding environment information. Brief explanation of the drawing

[0019] FIG. 1 is a schematic diagram illustrating the exterior of a vehicle equipped with a projection device related to the first embodiment of the present technology. Figure 2 is a schematic diagram illustrating an example of a projection pattern. Figure 3 is a schematic diagram illustrating an example of a scene in which a projection pattern is projected. FIG. 4 is a block diagram illustrating an example of the configuration of a projection device related to the first embodiment. FIG. 5 is a block diagram illustrating an example of the configuration of a surrounding environment recognition unit. Figure 6 is a flowchart illustrating a basic operation example of a projection device. Figure 7 is a schematic diagram illustrating the front view of a vehicle when changing lanes. FIG. 8 is a schematic diagram illustrating an example of a forward line projected when changing lanes. FIG. 9 is a schematic diagram illustrating an example of a lane change in a merging lane. FIG. 10 is a block diagram illustrating an example of the configuration of a projection device related to a second embodiment. FIG. 11 is a block diagram illustrating an example of the configuration of an environment recognition unit. FIG. 12 is a flowchart illustrating a basic operation example of a projection device. Figure 13 is a schematic diagram illustrating an example of a rear pattern according to collision risk. FIG. 14 is a block diagram illustrating an example of the configuration of a projection device related to a third embodiment. FIG. 15 is a block diagram illustrating an example of the configuration of an environment recognition unit. FIG. 16 is a flowchart illustrating a basic operation example of a projection device. FIG. 17 is a schematic diagram illustrating an example of a scene in which a vehicle width line is projected. FIG. 18 is a schematic diagram illustrating an example of a vehicle width line according to the approach distance. FIG. 19 is a schematic diagram illustrating an example of a vehicle width line being projected onto an obstacle. FIG. 20 is a schematic diagram illustrating an example of a vehicle width line projected when an obstacle is detected. Specific details for implementing the invention

[0020] Hereinafter, embodiments related to the present technology will be described with reference to the drawings.

[0021] <First Embodiment>

[0022] FIG. 1 is a schematic diagram illustrating the exterior of a vehicle equipped with a projection device related to a first embodiment of the present invention. FIG. 1A is a perspective view illustrating an example configuration of a vehicle (1), and FIG. 1B is a top view of the vehicle (1) seen from above. The vehicle (1) is equipped with a projection device (100) that projects an image onto the road surface surrounding the vehicle (1).

[0023] The projection device (100) has a plurality of projection units (10). A projection unit (10) is an element (projector) that projects an image by irradiating light onto a road surface. Hereinafter, the image projected onto the road surface surrounding the vehicle (1) from the projection unit (10) is described as a projection pattern. In the projection device (100), it is possible to individually control the projection patterns projected from each projection unit (10). The specific configuration of the projection unit (10) will be described later.

[0024] In FIG. 1A and FIG. 1B, eight projection sections (10a to 10h) provided in the vehicle (1) are schematically illustrated. In the example illustrated in FIG. 1, projection sections (10a to 10d) and projection sections (10e to 10h) are arranged on the left and right sides of the bottom of the vehicle (1) so as to be symmetrical with respect to each other.

[0025] Projection sections (10a and 10e) are positioned at the lower part of the front side of the vehicle (1) (e.g., the lower part of the front bumper) and project a projection pattern, for example, to the front of the vehicle (1). Projection sections (10b and 10f) are positioned at the lower part of the front side of the front door, and projection sections (10c and 10g) are positioned at the lower part of the rear side of the rear door. Projection sections (10b and 10f) and projection sections (10c and 10g) project a projection pattern, for example, to the side of the vehicle (1). Projection sections (10d and 10h) are positioned at the lower part of the rear side of the vehicle (1) (e.g., the lower part of the rear bumper) and project a projection pattern, for example, to the rear of the vehicle (1).

[0026] In addition, in FIG. 1B, for convenience, each projection part (10) is depicted as protruding from the vehicle body so that the position of each projection part (10) (projector) in the top view of the vehicle (1) can be seen. In the actual configuration, the projection part (10) is housed in the lower part of the vehicle body and installed so that it is not visible from the top. By doing so, it is possible to mount the projection device (100) without damaging the exterior of the vehicle (1).

[0027] Alternatively, as shown in Fig. 1B, each projection part (10) may be installed to protrude from the vehicle body. By doing so, it becomes possible to expand the range in which a projection pattern can be projected, for example.

[0028] The arrangement or number of projection parts (10) is not limited.

[0029] For example, a projection unit (10) (projection units (10a and 10e) in FIG. 1) that projects a pattern onto the front of a vehicle (1) may be installed on the front surface of the vehicle body. Specifically, the projection unit (10) may be provided around the headlights (upper, lower, left, and right sides of the headlights, etc.) or around the fog lamps (upper, lower, left, and right sides of the fog lamps, etc.). In addition, the projection unit (10) may be provided at the location of the front grille or in the central part of the entire surface of the vehicle body. Furthermore, the projection pattern of the left front and right front may be projected from a single projection unit (10).

[0030] In addition, for example, a projection part (10) that projects a pattern onto the side of the vehicle (1) (in FIG. 1, projection parts (10b, 10c, 10f and 10g)) may be provided at the bottom of the side mirror or at the lower side of the B-pillar that separates the front and rear doors (at the center of the vehicle's front and rear direction).

[0031] In addition, for example, a projection unit (10) (projection unit (10d and 10h) in FIG. 1) that projects a pattern onto the rear of a vehicle (1) may be installed on the rear surface of the vehicle body. Specifically, the projection unit (10) may be provided around the brake lamp (upper side, lower side, left and right side of the brake lamp, etc.), around the license plate (upper side, lower side, left and right side of the license plate, etc.), or in the central part of the rear surface of the vehicle body. In addition, the projection pattern of the left rear and right rear may be projected from a single projection unit (10).

[0032] In addition, the projection part (10) may be appropriately positioned at a location where a desired projection pattern can be projected.

[0033] FIG. 2 is a schematic diagram illustrating an example of a projection pattern. FIG. 3 is a schematic diagram illustrating an example of a scene in which the projection pattern is projected.

[0034] In this embodiment, depending on the situation in which the vehicle (1) is operating, a plurality of projection modes are selected, and a projection pattern (2) according to the projection mode is projected. In FIGS. 2A to FIGS. 2D, projection patterns (2) projected in each projection mode of normal driving mode, low-speed driving mode, reverse mode, and parking mode are schematically illustrated.

[0035] Among the multiple projection modes, the mode other than the parking mode is the mode in which the vehicle (1) actually drives. In the mode in which the vehicle (1) drives (Fig. 2A to Fig. 2C), a line pattern (3) that is elongated along a line is used as the projection pattern (2). Accordingly, the projection pattern (2) includes a line pattern (line pattern (3)). The line pattern (3) is, for example, a continuous strip pattern. Alternatively, the line pattern (3) may be formed by arranging small patterns along a line at regular intervals.

[0036] In this embodiment, the line pattern (3) includes a first pattern on the line projected forward in the direction of travel (4) of the vehicle (1) and a second pattern on the line projected backward in the direction of travel (4).

[0037] In FIGS. 2A to FIGS. 2C, an arrow indicating the direction of travel (4) of the vehicle (1) is schematically illustrated. The size of the arrow indicates the speed of the vehicle (1). The pattern projected forward with respect to this direction of travel (4) is the first pattern, and the pattern projected backward is the second pattern.

[0038] In addition, in this embodiment, a third pattern (central line (3b) described later) is used, which is projected onto the road surface surrounding the central part of the vehicle (1) (road surface extending from the bottom to the side of the vehicle (1)).

[0039] As described below, in the projection device (100), a predicted trajectory (5) that the vehicle (1) is predicted to pass through and a passing trajectory (6) that the vehicle (1) has passed through are each calculated. These predicted trajectory (5) and passing trajectory (6) are represented using a first pattern and a second pattern. That is, the first pattern is generated as a line pattern (3) representing the predicted trajectory (5) of the vehicle (1), and the second pattern is generated as a line pattern (3) representing the passing trajectory of the vehicle (1).

[0040] Below, the normal driving mode, low-speed driving mode, reverse mode, and parking mode in which a line pattern (3) is used are each described.

[0041] In FIG. 2A, an example of a projection pattern (2) projected in normal driving mode is shown. The upper and lower drawings of FIG. 2A are schematic diagrams of the vehicle (1) viewed from the side and above.

[0042] Here, the normal driving mode is a projection mode selected when, for example, the vehicle (1) is driving normally forward without slowing down. The normal driving mode is selected when, for example, the vehicle (1) is driving at a speed faster than a normal slow speed (for example, less than 10 km / h). Therefore, the normal driving mode is used when driving forward in a driving lane, rather than for driving that requires slow speed, such as stopping, turning right, turning left, or parking.

[0043] In normal driving mode, three types of line patterns (3), such as a front line (3a), a center line (3b), and a rear line (3c), are projected as projection patterns (2). Each line (3a to 3c) is configured as a pair of line patterns (3) projected on the left and right sides of the vehicle (1).

[0044] The front line (3a) is a line pattern (3) projected onto the road surface in front of the vehicle (1). The left and right front lines (3a) are projected from projection parts (10a and 10e) shown in FIG. 1, for example.

[0045] The center line (3b) is a line pattern (3) projected onto the road surface on the side from below the vehicle (1). The center line (3b) on the left is projected, for example, from the projection section (10b and 10c), and the center line (3b) on the right is projected, for example, from the projection section (10f and 10g).

[0046] The rear line (3c) is a line pattern (3) projected onto the rear road surface of the vehicle (1). The left and right rear lines (3c) are projected, for example, from the projection parts (10d and 10h).

[0047] In addition, the correspondence between each line (3a to 3c) and the projection unit (10) is not limited to the above-described example, and, for example, a configuration in which one line pattern (3) is projected using two projection units (10) is also possible.

[0048] As described above, the line pattern (3) shown in A of FIG. 2 is projected while the vehicle (1) is moving forward.

[0049] Accordingly, in normal driving mode, the front line (3a) becomes a first pattern on a line projected forward in the direction of travel of the vehicle (1) and is generated as a line pattern (3) representing the predicted trajectory (5) of the vehicle (1). Specifically, the line shape of the front line (3a) is set to represent the predicted trajectory (5) of the front tire of the vehicle (1).

[0050] Additionally, in normal driving mode, the rear line (3c) becomes a second pattern on the line projected to the rear of the vehicle (1) in the direction of travel, and is generated as a line pattern (3) representing the passing trajectory (6) of the vehicle (1). Specifically, the line shape of the rear line (3c) is set to represent the passing trajectory (6) of the rear tire of the vehicle (1).

[0051] In addition, in normal driving mode, the center line (3b) (third pattern) is used as a line light illuminating both sides of the center of the vehicle (1).

[0052] In FIG. 3A, a scene in which a normal driving mode is applied is schematically illustrated. Here, a front line (3a) representing a predicted trajectory (5), a rear line (3c) representing a passing trajectory (6), and a center line (3b) that serves as illumination on the line are each projected onto the road surface surrounding a vehicle (1) traveling at a relatively high speed in a driving lane.

[0053] In this way, a line pattern (3) showing a predicted trajectory (5) or a passing trajectory (6) is projected around the vehicle (1). By doing so, it becomes possible to explicitly convey the direction of travel or driving history of the vehicle (1) to pedestrians outside the vehicle or drivers of other vehicles, and it also becomes possible to effectively visualize the vehicle (1) while it is in motion.

[0054] In FIG. 2B, an example of a projection pattern (2) projected in low-speed driving mode is shown. The upper and lower drawings of FIG. 2B are schematic diagrams of the vehicle (1) viewed from the side and above.

[0055] Here, the low-speed driving mode is a projection mode selected, for example, when the vehicle (1) is moving forward while moving slowly. Therefore, the low-speed driving mode is used when performing driving that requires slow speed, such as stopping, turning right, turning left, or parking.

[0056] In low-speed driving mode, just like in the normal driving mode described above, three types of line patterns (3) of a front line (3a), a center line (3b), and a rear line (3c) are projected as projection patterns (2).

[0057] The line pattern (3) shown in Fig. 2B is projected while the vehicle (1) is moving forward, just like in Fig. 2A.

[0058] Accordingly, in low-speed driving mode, the front line (3a) becomes a first pattern on a line projected forward in the direction of travel of the vehicle (1) and is generated as a line pattern (3) representing the predicted trajectory (5) of the vehicle (1). Specifically, the line shape of the front line (3a) is set to represent the predicted trajectory (5) of the front tire of the vehicle (1).

[0059] Additionally, in low-speed driving mode, the rear line (3c) becomes a second pattern on the line projected to the rear of the vehicle (1) in the direction of travel, and is generated as a line pattern (3) representing the passing trajectory (6) of the vehicle (1). Specifically, the line shape of the rear line (3c) is set to represent the passing trajectory (6) of the rear tire of the vehicle (1).

[0060] Additionally, in low-speed driving mode, the center line (3b) (third pattern) is generated as a line pattern (3) representing the predicted trajectory (5) of the vehicle (1). Specifically, the center line (3b) is set so that the line shape represents the predicted trajectory (5) of the rear tire of the vehicle (1).

[0061] By this, when the vehicle (1) moves forward slowly, it becomes possible to specify to pedestrians, etc. the trajectory that the rear tire is expected to pass. By this, it becomes possible to suppress the risk of being entangled when turning right or left.

[0062] In FIG. 3B, the driving of a vehicle (1a) in a parking lot is schematically illustrated as an example of a scene in which a low-speed driving mode is applied. The vehicle (1a) is moving forward from the parking space on the right side of the drawing to exit. In this way, when a vehicle (1a) that was stopped moves forward, it becomes a slow-moving vehicle, so a low-speed driving mode is used.

[0063] Here, a front line (3a) representing the predicted trajectory (5) of the front tire and a center line (3b) representing the predicted trajectory (5) of the rear tire are respectively projected onto the road surface surrounding the vehicle (1a). Additionally, in FIG. 3B, the illustration of the rear line (3c) representing the passing trajectory (6) of the rear tire of the vehicle (1a) is omitted.

[0064] In this way, a line pattern (3) showing the predicted trajectory (5) of the front tire and rear tire is projected around the vehicle (1a). By doing so, it is possible to sufficiently avoid accidents such as entanglement or contact while urging pedestrians to pay attention.

[0065] Additionally, the front line (3a) may be set to indicate the width of the vehicle (1). For example, the spacing between the right and left lines is set to the maximum width of the vehicle (1) body. This makes it possible to encourage steering operation, etc., that is conscious of the vehicle width.

[0066] In Fig. 2C, an example of a projection pattern (2) projected in reverse mode is shown.

[0067] Here, reverse mode is a projection mode selected when the vehicle (1) is driving in reverse (back driving). Reverse mode is used, for example, when parking by reversing.

[0068] In reverse mode, just like each driving mode described above, three types of line patterns (3) of a front line (3a), a center line (3b), and a rear line (3c) are projected as projection patterns (2).

[0069] The line pattern (3) shown in Fig. 2C is projected while the vehicle (1) is driving in reverse.

[0070] Accordingly, in reverse mode, the front line (3a) becomes a second pattern on the line projected to the rear of the vehicle (1) in the direction of travel. In this case, the front line (3a) is generated as a line pattern (3) representing the passing trajectory (6) of the vehicle (1). Specifically, the line shape of the front line (3a) is set to represent the passing trajectory (6) of the front tire of the vehicle (1).

[0071] Additionally, in reverse mode, the rear line (3c) becomes a first pattern on a line projected forward in the direction of travel of the vehicle (1). In this case, the rear line (3c) is generated as a line pattern (3) representing the predicted trajectory (5) of the vehicle (1). Specifically, the line shape of the rear line (3c) is set to represent the predicted trajectory (5) of the rear tire of the vehicle (1).

[0072] Additionally, in reverse mode, the center line (3b) (third pattern) is generated as a line pattern (3) representing the predicted trajectory (5) of the vehicle (1). Specifically, the center line (3b) is set so that the line shape represents the predicted trajectory (5) of the front tire of the vehicle (1).

[0073] By this, when the vehicle (1) is reversing, it becomes possible to specify to pedestrians, etc. the predicted path of the front tires. By this, it becomes possible to suppress the risk of getting caught in the road when parking.

[0074] In FIG. 3B, as an example of a scene where reverse mode is applied, the driving of a vehicle (1b) in a parking lot is schematically illustrated. The vehicle (1b) is moving to reverse and park in the parking space on the left side of the drawing.

[0075] Here, a rear line (3c) representing the predicted trajectory (5) of the rear tire and a center line (3b) representing the predicted trajectory (5) of the front tire are respectively projected onto the road surface surrounding the vehicle (1b). Additionally, in FIG. 3B, the illustration of the front line (3a) representing the passing trajectory (6) of the front tire of the vehicle (1b) is omitted.

[0076] In this way, since a line pattern (3) showing the predicted trajectory (5) of the front tire and rear tire is projected around the vehicle (1b), it is possible to sufficiently avoid accidents such as entanglement or contact during back operation.

[0077] In addition, just as with the forward movement, the rear line (3c) may be set to indicate the width of the vehicle (1). By doing so, it becomes possible to perform back operations, etc., while checking the width of the vehicle.

[0078] In Fig. 2D, an example of a projection pattern (2) projected in parking mode is shown.

[0079] Here, the parking mode is a projection mode selected when the shift position of the vehicle (1) is parking ("P"), that is, when the vehicle (1) is in a stationary state.

[0080] In parking mode, the above-mentioned line pattern (3) is not displayed, and a pattern of lighting that surrounds the entire perimeter of the vehicle (1) (hereinafter referred to as the stopping pattern (7)) is projected as the projection pattern (2). Here, front lighting (7a), side lighting (7b), and rear lighting (7c) are used as the stopping pattern (7). Each of the lights (7a to 7c) is a gradient pattern in which the color becomes lighter as it moves away from the vehicle (1). In addition, the design of the stopping pattern (7) is not limited.

[0081] In this way, the projection pattern (2) includes a stopping pattern (7) different from the line pattern (3). In this embodiment, the stopping pattern (7) is an example of a different pattern.

[0082] By using the stopping pattern (7), it is possible to convey to pedestrians outside the vehicle that the vehicle (1) is stopped with the shift position set to parking, that is, that the vehicle (1) is not moving. As a result, for example, pedestrians or other vehicles can pass by the vehicle (1) safely.

[0083] In addition, the projection mode described above is not limited to other modes. For example, a Welcome Light mode that displays a predetermined lighting pattern when the driver unlocks the vehicle (1), opens the door, or starts the engine may be set.

[0084] FIG. 4 is a block diagram illustrating an example of the configuration of a projection device (100) related to the first embodiment.

[0085] The projection device (100) has the above-mentioned projection unit (10), a vehicle information sensor unit (11), a driver monitoring camera (12), an ambient environment sensor unit (13), a memory unit (15), and a controller (20).

[0086] The projection unit (10) is a device that projects a projection pattern (2) by irradiating light, and is configured to change the shape or color of the projection pattern (2).

[0087] For the projection unit (10), for example, a projector that emits laser light as an illumination light is used. By using laser light, it is possible to display a projection pattern (2) with high brightness over a long distance. In addition, in addition to the laser light source, an LED light source or a lamp light source may be used.

[0088] The method of modulating the illumination light is not limited, and, for example, a light modulation element using a transmissive liquid crystal panel or MEMS (Micro Electro Mechanical Systems) is used. In addition, by combining a phase modulation element using a reflective liquid crystal panel, etc., it becomes possible to project light that is concentrated within a predetermined range. By doing so, it becomes possible to significantly improve the brightness of the projection pattern (2).

[0089] In addition, the specific configuration of the projection unit (10) is not limited, and, for example, a projection light capable of modulating the illumination light or a laser light source may be used.

[0090] The vehicle information sensor unit (11) has a sensor that detects information regarding the state of each part of the vehicle (1).

[0091] Specifically, a steering angle sensor for detecting the steering angle, a speed sensor for detecting the driving speed of the vehicle (1), and an acceleration sensor for detecting the acceleration applied to the vehicle (1) are provided.

[0092] Additionally, the vehicle information sensor unit (11) has an accelerator opening degree sensor that detects the opening degree of the accelerator and a brake opening degree sensor that detects the opening degree of the brake (brake strength).

[0093] Additionally, the vehicle information sensor unit (11) has an accelerator pedal pressure sensor that detects the pressure of pressing on the accelerator pedal (operating force of the accelerator) and a brake pedal pressure sensor that detects the pressure of pressing on the brake pedal (operating force of the brake). These pressure sensors may be sensors that detect the total pressure value applied to the pedals, or sensors that detect the pressure distribution.

[0094] In addition, the vehicle information sensor unit (11) has a shift position sensor that detects the position of the shift lever (shift position) and a side brake sensor that detects the on / off of the side brake.

[0095] In addition, an ignition sensor that detects the on / off status of the ignition switch, a turn signal sensor that detects the on / off status of the turn signal (direction indicator), a hazard sensor that detects the on / off status of the hazard lamp, and a light sensor that detects the on / off status of the headlights or the switching between high beam and low beam (passing) may be provided.

[0096] In addition, any sensor that detects information about the vehicle (1) may be used as the vehicle information sensor unit (11).

[0097] The driver monitoring camera (12) is a camera that photographs the driver of the vehicle (1), and is provided inside the vehicle (1) so as to be able to photograph the driver from the front, for example. As the driver monitoring camera (12), a digital camera equipped with an imaging element such as CMOS or CCD is used, for example.

[0098] In this embodiment, as described below, an image captured by the driver monitoring camera (12) is used as data to detect the driver's line of sight.

[0099] The surrounding environment sensor unit (13) has a sensor that detects the state of the surrounding environment of the vehicle (1).

[0100] In this embodiment, an object detection sensor is provided as an ambient environment sensor unit (13) to detect objects around the vehicle (1).

[0101] As an object detection sensor, for example, a camera that captures the surroundings of a vehicle (1) is used. For example, a front camera, a rear camera, a left camera, and a right camera installed on the front, rear, left, and right sides of the vehicle (1) are used. The scenery reflected by vehicles driving around the vehicle (1) is captured by these cameras installed in the front, rear, left, and right directions. Alternatively, a full-surround camera capable of capturing the entire surroundings of the vehicle (1) using a fisheye lens or the like may be used.

[0102] In addition, as an object detection sensor, radar sensors, ultrasonic sensors, LiDAR sensors, etc. that detect front, back, left, and right may be used. By these sensors, the location of objects around the vehicle (1) is detected.

[0103] In addition, the types of object detection sensors are not limited, and, for example, a combination of a camera and other distance measuring sensors may be used.

[0104] The memory unit (15) is a non-volatile memory device. As the memory unit (15), for example, a recording medium using a solid-state device such as an SSD (Solid State Drive) or a magnetic recording medium such as an HDD (Hard Disk Drive) is used. In addition, the type of recording medium used as the memory unit (15) is not limited, and for example, any recording medium that records data non-temporarily may be used.

[0105] A control program for controlling the overall operation of the projection device (100) is stored in the memory unit (15). The control program corresponds to a program related to the present embodiment. Additionally, the memory unit (15) functions as a computer-readable recording medium on which the program is recorded.

[0106] Additionally, data specifying the shape or color of the projection pattern (2) is stored in the memory unit (15). Furthermore, the types of data stored in the memory unit (15) are not limited, and any data necessary for the operation of the projection device (100) may be stored.

[0107] The controller (20) controls the operation of each block of the projection device (100). The controller (20) has hardware configurations necessary for a computer, such as a CPU or memory (RAM, ROM). Various processes are executed by the CPU loading a control program stored in the memory unit (15) into the RAM and executing it. The controller (20) functions as an information processing device according to the present embodiment.

[0108] As the controller (20), devices such as a PLD (Programmable Logic Device) like an FPGA (Field Programmable Gate Array) or other ASICs (Application Specific Integrated Circuits) may be used. Also, a processor such as a GPU (Graphics Processing Unit) may be used as the controller (20).

[0109] In this embodiment, the CPU of the controller (20) executes a program related to this embodiment, thereby realizing a vehicle information acquisition unit (21), a trajectory calculation unit (22), a projection image determination unit (23), an image data generation unit (24), a gaze detection unit (25), and an surrounding environment recognition unit (26) as functional blocks. And through these functional blocks, an information processing method related to this embodiment is executed. In addition, dedicated hardware such as an IC (integrated circuit) may be appropriately used to realize each functional block. In addition, these functional blocks may be realized by other computers, etc., capable of communicating with the controller (20).

[0110] The vehicle information acquisition unit (21) acquires information (vehicle information) regarding the vehicle (1) detected by each sensor of the vehicle information sensor unit (11).

[0111] In this embodiment, speed-related information regarding the speed of the vehicle (1) is acquired by the vehicle information acquisition unit (21). The speed-related information includes information indicating the speed of the vehicle (1) or acceleration and deceleration that change along with the speed, and information regarding operations (accelerator operation and brake operation) that change the physical quantities thereof.

[0112] As speed-related information, speed information indicating the speed of the vehicle (1) and acceleration information indicating the acceleration of the vehicle (1) are obtained. For example, the detection results of the speed sensor and the acceleration sensor are read as speed information and acceleration information.

[0113] In addition, accelerator and brake information are acquired as speed-related information. Among these, the accelerator information includes the degree of accelerator opening and the pressure applied to the accelerator pedal (accelerator operating force). The brake information includes the degree of brake opening and the pressure applied to the brake pedal (brake operating force). For example, the detection results of the accelerator opening sensor and the accelerator pedal pressure sensor are read as the degree of accelerator opening and operating force. Additionally, the detection results of the brake opening sensor and the brake pedal pressure sensor are read as the degree of brake opening and operating force.

[0114] Additionally, driving state information regarding the driving state of the vehicle (1) is acquired by the vehicle information acquisition unit (21). As driving state information, shift lever information and side brake information are acquired. Among these, the shift lever information includes information indicating the state of the shift position of the vehicle (1). Additionally, the side brake information includes information indicating the side brake state of the vehicle (1). For example, the detection results of the shift position sensor and the side brake sensor are read as shift lever information and side brake information.

[0115] In addition, steering angle information indicating the steering angle is acquired by the vehicle information acquisition unit (21). For example, the detection result of the steering angle sensor is read as steering angle information.

[0116] In addition, turn signal information (turn indicator information) regarding the operation of the turn signal is acquired by the vehicle information acquisition unit (21). The turn signal information is information indicating the on / off status of the right and left indicators.

[0117] In addition, the detection results of each sensor constituting the vehicle information sensor unit (11) are appropriately obtained.

[0118] The trajectory calculation unit (22) calculates the predicted trajectory (5) and passing trajectory (6) of the vehicle (1) (see FIG. 2 and FIG. 3).

[0119] In this embodiment, the predicted trajectory (5) that the vehicle (1) is expected to pass through is estimated by the trajectory calculation unit (22) based on the steering angle information, speed information, and acceleration information of the vehicle (1). Here, for example, the trajectory of the front tire (or rear tire) predicted when the vehicle (1) moves at the current steering angle, speed, and acceleration is estimated. At this time, appropriate corrections based on centrifugal force or tire grip force may be performed.

[0120] In addition, instead of the predicted trajectory (5) of the tire, the center trajectory of the vehicle (1), for example, may be estimated.

[0121] The method for estimating the predicted trajectory (5) is not limited, and techniques such as trajectory prediction processing used in automatic driving, for example, can be applied.

[0122] In addition, in this embodiment, the trajectory (6) of the vehicle (1) is calculated by recording the movement of the vehicle (1). Here, the trajectory (6) of the front tire (or rear tire) is calculated from the recording of the steering angle, speed, and acceleration of the vehicle (1). For this processing, techniques such as dead reckoning may be used, for example. Alternatively, GPS positioning or WiFi positioning may be used.

[0123] In addition, for example, it is also possible to record the predicted trajectory (5) and use it as the passing trajectory (6).

[0124] The method for estimating the passing trajectory (6) is not limited, and any processing capable of reproducing the trajectory of the vehicle (1), for example, may be used.

[0125] In this embodiment, the trajectory calculation unit (22) functions as a predicted trajectory calculation unit and a passing trajectory calculation unit.

[0126] The projection image determining unit (23) determines the display content or display parameters of the projection pattern (2) and outputs the data of the projection pattern (2). This process is a process that controls the display of the projection pattern (2).

[0127] In the present disclosure, the processing for controlling the display of the projection pattern (2) is, for example, the processing for setting the display parameters of the projection pattern (2). Typically, parameters such as the color, width, length, and brightness of the line pattern (3) (see FIG. 2A to FIG. 2C) are set.

[0128] In the projection device (100), the display of a projection pattern (2) projected onto the road surface surrounding the vehicle (1) from a projection unit (10) mounted on the vehicle (1) is controlled by a projection image determination unit (23) based on surrounding environment information. The surrounding environment information is information acquired by a surrounding environment recognition unit (26) described later based on the output of a surrounding environment sensor unit (13), and is information indicating the condition of surrounding roads or other vehicles.

[0129] As described above, among the projection patterns (2), the first and second patterns (here, the front line (3a) and the rear line (3c)) include a pair of lines projected onto the left and right sides of the vehicle.

[0130] In this embodiment, the projection image determining unit (23) controls the display of the target line projected onto the target lane side of the vehicle (1) that is the target of lane change in the first pattern (front line (3a)) based on surrounding environment information.

[0131] Here, a lane change is a driving operation that involves moving from the lane in which the vehicle is currently driving to an adjacent lane by crossing the boundary line that defines the lane. Lane changes include driving operations such as changing lanes while driving in multiple lanes, entering the lane at the point of merging in a merging lane, or entering the shoulder. The lane to which the vehicle moves for such driving operations becomes the target lane.

[0132] In the projection image determination unit (23), the display parameters of the target line projected on the side of the front line (3a) that is close to the target lane are controlled using information (surrounding environment information) such as vehicles driving around the vehicle (1).

[0133] The details regarding surrounding environment information and the display control of the target line will be described in detail later.

[0134] In addition, in the projection image determination unit (23), the shape of the line pattern (3) is controlled to represent the trajectory of the vehicle (1) (predicted trajectory (5) and passing trajectory (6)) calculated by the trajectory calculation unit (22).

[0135] Specifically, a first pattern representing a predicted trajectory (5) is generated by the projection image determining unit (23). That is, the shape of the forward line (3a) is set to a shape that follows the predicted trajectory (5).

[0136] In addition, a second pattern representing the passing trajectory (6) is generated by the projection image determining unit (23). That is, the shape of the rear line (3c) is set to a shape that follows the passing trajectory (6).

[0137] In addition, the left and right line spacing in the front line (3a) (or rear line (3c)) may be set to represent the vehicle width.

[0138] In this embodiment, the projection image determining unit (23) corresponds to the projection control unit.

[0139] The image data generation unit (24) generates image data to be output to each projection unit (10) based on the data of the projection pattern (2) output from the projection image determination unit (23).

[0140] For example, a frame image is generated that represents the shape of the projection pattern (2) when viewed from above the vehicle (1). For this frame image, a process is performed to correct distortion or brightness bias associated with the projection according to the projection angle of each projection part (10). A series of frame images with such distortion corrected becomes image data.

[0141] In addition, any image processing to properly project the projection pattern (2) may be performed.

[0142] The gaze detection unit (25) detects the gaze direction of the driver based on an image of the driver captured by the driver monitoring camera (12). The method for detecting the gaze direction is not limited, and, for example, a pupil method that detects the gaze from the direction of the driver's pupil (the black part of the eye) is used. In addition, any method capable of detecting the gaze direction using an image of the driver may be used.

[0143] In addition, instead of the direction of gaze, the direction of the driver's face may be detected. In this case, the angle at which the driver rotated their face left or right is detected.

[0144] From the gaze detection unit (25), gaze information indicating the driver's gaze direction, or face direction information indicating the driver's face direction, is output to the projection image determination unit (23).

[0145] FIG. 5 is a block diagram illustrating an example configuration of a surrounding environment recognition unit (26).

[0146] The surrounding environment recognition unit (26) performs recognition processing regarding the surrounding environment of the vehicle (1) based on the output of the surrounding environment sensor unit (13). It detects objects (pedestrians, other vehicles, curbs, etc.) existing around the vehicle (1) and calculates various information regarding the surrounding environment (surrounding environment information). That is, the surrounding environment recognition unit (26) calculates and acquires surrounding environment information regarding the surrounding environment of the vehicle.

[0147] In this embodiment, the surrounding environment recognition unit (26) corresponds to the acquisition unit.

[0148] As illustrated in FIG. 5, the surrounding environment recognition unit (26) has an object detection unit (30), a relative speed calculation unit (31), a lane detection unit (32), and a space recognition unit (33).

[0149] The object detection unit (30) detects objects around the vehicle (1) based on the detection result of the surrounding environment sensor unit (13). As the object detection unit (30), a learning unit that performs image recognition processing using, for example, machine learning is used.

[0150] For example, in a configuration where a camera is provided as an ambient environment sensor unit (13), an image captured by the camera is input to an object detection unit (30).

[0151] In this case, using reference information generated by prior learning, objects in the input image are detected and their attributes are identified. The reference information is, for example, prior information stored by associating the type and characteristics of an object. Additionally, when a Deep Neural Network (DNN) or similar is used as a learning agent, recognition model information is used as reference information. In this process, as objects surrounding the vehicle (1) are detected, the attributes of objects such as cars (e.g., types of vehicles such as passenger cars, trucks, buses, etc.), motorcycles, bicycles, and people are identified. For each detected object, for example, a serial number ID is assigned.

[0152] In addition, for each detected object, its position within the image is determined. At this time, the distance (relative distance) from the position within the image to the object may be calculated.

[0153] From the object detection unit (30), ID information, attribute information, distance information, and location information regarding each object are output.

[0154] The relative speed calculation unit (31) calculates the relative speed of an object with respect to the vehicle (1) based on information regarding an object output from the object detection unit (30).

[0155] For example, for each object, the change in distance per unit time is calculated, and the relative speed of the vehicle (vehicle (1)) is calculated from the change in distance.

[0156] From the relative speed calculation unit (31), ID information, attribute information, distance information, location information, and speed information regarding each object are output.

[0157] The lane detection unit (32) detects a lane (lane) on the road where the vehicle (1) is traveling. For example, from an image of the road captured by cameras provided on the front, rear, left, and right sides of the vehicle (1), processing to recognize a white line (a boundary line separating the road lanes) or to detect a curb is executed using edge detection processing, etc. The area between this white line or curb is detected as the driving lane where the vehicle (1) is traveling. In addition, an adjacent lane adjacent to the driving lane (40) or an area of ​​the shoulder may also be detected.

[0158] From the lane detection unit (32), information indicating the area of ​​the lane is output.

[0159] The space recognition unit (33) recognizes the situation of the surrounding space of the vehicle (1) based on the detection results of the object detection unit (30) and the lane detection unit (32).

[0160] Specifically, adjacent lanes are identified from information such as the driving lane output from the lane detection unit (32). Additionally, information such as the location and size of other vehicles running on adjacent lanes is extracted from the output of the object detection unit (30). Then, using the extracted information, the location and size of the empty space are calculated. Here, the empty space is, for example, a space where no other vehicles exist.

[0161] In addition, using information on the location and size of the empty space, it is determined whether there is a space where the vehicle (1) can change lanes (i.e., a movable space where the vehicle (1) can enter from the target lane). The result of this determination is output as information on whether there is a movable space.

[0162] In addition, the presence or absence of an object (typically another vehicle) in the driving lane of the vehicle (1), a neighbor, or an adjacent lane is determined, and information about the object in each lane is calculated as lane situation information.

[0163] From the space recognition unit (33), empty space information (location and size), information indicating whether there is a movable space, and lane status information are output.

[0164] In this embodiment, as illustrated in FIG. 5, the output of the relative speed calculation unit (31) and the space recognition unit (33) becomes the final output of the surrounding environment recognition unit (26). Accordingly, from the surrounding environment recognition unit (26), information regarding ID information, attribute information, distance information (relative distance), location information, and speed information (relative speed) regarding an object is output. In addition, information regarding empty space, information indicating movable space, and lane condition information is output. All of these pieces of information correspond to surrounding environment information.

[0165] In addition, when a radar sensor, ultrasonic sensor, LiDAR sensor, etc. other than a camera is used as the surrounding environment sensor unit (13), the position and distance of an object are detected by the object detection unit (30), and an ID is assigned to the detected object. At this time, the attribute information of the object is not output.

[0166] Then, the relative speed of the detected object is calculated by the relative speed calculation unit (31). In this case, the output of the surrounding environment recognition unit (26) (relative speed calculation unit (31)) is the object's ID information, distance information, location information, and speed information. Additionally, if a radar sensor is used, not only object detection but also relative speed can be detected from the output.

[0167] FIG. 6 is a flowchart illustrating a basic operation example of a projection device (100). The processing shown in FIG. 6 is a loop processing that is repeatedly executed during the operation of the projection device (100), for example. Here, the vehicle (1) is driving forward, and a normal driving mode (or low-speed driving mode) is selected as the projection mode of the projection pattern (2).

[0168] First, speed-related information is obtained by the vehicle information acquisition unit (21) (step 101).

[0169] Specifically, speed information, acceleration information, accelerator information (opening degree and operating force of the accelerator) and brake information (opening degree and operating force of the brake) are read from sensors provided in each part of the vehicle (1) as speed-related information.

[0170] In addition, at this time, driving status information (shift lever information, handbrake information) and steering angle information are also read.

[0171] Next, the trajectory of the vehicle (1) (predicted trajectory (5) and passing trajectory (6)) is calculated by the trajectory calculation unit (22) (step 102).

[0172] Here, as shown in FIG. 2A and FIG. 2B, a projection mode used during forward driving (normal driving mode, low-speed driving mode) is selected, and the predicted trajectory (5) of the front tire and the passing trajectory (6) of the rear tire of the vehicle (1) are calculated.

[0173] Next, various processing for recognizing the surrounding environment is performed by the surrounding environment recognition unit (26), and surrounding environment information is produced (step 103).

[0174] In this embodiment, traffic situation information is calculated that indicates the surrounding traffic situation of the vehicle (1), including the target lane that is the target of lane change of the vehicle (1), as surrounding environment information.

[0175] The traffic condition information includes information indicating the empty space of the target lane. This is information indicating the aforementioned empty space information and movable space. Additionally, the traffic condition information includes placement information of other vehicles driving around the vehicle (1). This is, for example, distance information, location information, and lane condition information of an object determined to be a vehicle by attribute information. Additionally, the traffic condition information includes information regarding the relative speed of other vehicles to the vehicle (1). This is, for example, speed information of an object determined to be a vehicle. Furthermore, if attribute information is not detected, the location information or speed information of the object is simply detected.

[0176] FIG. 7 is a schematic diagram illustrating the front view of a vehicle (1) when changing lanes. Here, the driving lane (40a) in which the vehicle (1) is traveling merges with the adjacent lane (40b) to the right of the driving lane (40a). Therefore, in front of the vehicle (1), the driving lane (40a) gradually narrows, and the driver needs to change lanes (merge) to the adjacent lane (40b) to the right before the driving lane (40a) disappears.

[0177] In the surrounding environment recognition unit (26), other vehicles (50) traveling in adjacent lanes (40) are detected, for example, by the object detection unit (30). Here, a truck traveling to the right front of the vehicle (1) is detected as another vehicle (50). In addition, other vehicles (50) (not shown) to the side or rear of the vehicle (1) can also be detected. Then, location information, distance information, etc. regarding each other vehicle (50) are calculated.

[0178] In addition, the relative speed (speed information) for the vehicle (1) is calculated from the amount of movement of other vehicles (50), etc., by the relative speed calculation unit (31).

[0179] In addition, the lane detection unit (32) detects the boundary line (51) (solid line on the left or dotted line on the right) dividing the lane, and information indicating the area of ​​the driving lane (40a) is produced.

[0180] In addition, the adjacent lane (40b) adjacent to the area of ​​the driving lane (40a) is detected by the space recognition unit (33), and the location and size of the empty space (52) (the diagonal area in the drawing) are calculated based on the location information of other vehicles (50) in the adjacent lane (40b).

[0181] In addition, the space recognition unit (33) determines whether there is an empty space (52) of sufficient size in a position where the vehicle (1) can move. In the example shown in FIG. 7, for instance, the distance to another vehicle (50) in the right front is close, so it is determined that there is no space for the vehicle (1) to enter (no movable space).

[0182] In addition, the space recognition unit (33) calculates lane situation information indicating information (space arrangement, etc.) of other vehicles (50) in adjacent lanes (40b).

[0183] Returning to FIG. 6, a preliminary operation for lane change is detected by the projection image determining unit (23) (step 104). A preliminary operation for lane change is, for example, an operation performed immediately before a driver changes lanes, and is an operation of moving the gaze or head unit to look at the side where the lane change is being performed (right side in FIG. 7).

[0184] In this embodiment, the direction of the driver's gaze or the direction of the face is constantly monitored by the gaze detection unit (25) shown in FIG. 4. The projection image determination unit (23) tracks the driver's gaze movement based on the detection result of the gaze detection unit (25) and detects whether there is an action of checking the left or right direction (preliminary action of changing lanes) by looking over the side mirror or directly with the eyes.

[0185] In addition, if a preliminary operation is detected, information indicating the verification direction (left and left) simultaneously confirmed by the driver is generated.

[0186] Next, it is determined whether a preliminary operation for lane change has been detected (Step 105).

[0187] For example, if no preliminary operation is detected ("No" in Step 105), the projection image determining unit (23) performs normal display processing regarding the projection pattern (2) (Step 106).

[0188] Normal display processing is a display processing used, for example, when lane changes are not performed. In this processing, each of the left and right lines constituting the front line (3a) and the rear line (3c) is set to the same display parameters (color, width, length, blinking).

[0189] When a preliminary operation is detected ("Yes" in Step 105), the projection image determining unit (23) performs lane change indication processing regarding the projection pattern (2) (Step 107).

[0190] The lane change display processing is a display processing used when a lane change is performed. Specifically, the display parameters of the target line projected onto the target lane side of the forward line (3a) (first pattern) that is the target of the lane change are set according to the traffic condition information (surrounding environment information) mentioned above.

[0191] In the example illustrated in FIG. 7, the adjacent lane (40b) to the right of the driving lane (40a) becomes the target lane (41). Also, among the forward lines (3a), the right line closest to the target lane (41) becomes the target line (53).

[0192] In this way, in the present embodiment, whether to perform display control of the target line (53) is determined based on a change in the direction of gaze.

[0193] In addition, the method of detecting a preliminary action is not limited. For example, a turn signal operation performed at a speed greater than a certain level may be detected as a preliminary action. In addition, both a change in the direction of gaze and a turn signal operation may be detected as preliminary actions. For example, if a turn signal operation is not detected within a certain time after a change in the direction of gaze is detected, the lane change display processing may be switched to a normal display processing. By doing so, a situation in which the front line (3a) is unnecessarily changed is avoided.

[0194] The details of the lane change indication processing are explained in detail.

[0195] In this embodiment, the risk of change associated with changing lanes of the vehicle (1) is estimated based on traffic situation information. The risk of change is a parameter indicating the possibility of contact or collision with another vehicle (50) occurring, for example, when the vehicle (1) changes lanes.

[0196] For example, if there is an empty space (52) (movable space) where the vehicle (1) can change lanes, the risk of change is set low. On the other hand, the smaller the movable space, or the more separated the movable space is, the higher the risk of change is set. Also, if there is no movable space, it is set higher than when there is a movable space.

[0197] Also, for example, in an adjacent lane (40), if another vehicle (50) is detected, the risk of change is set higher as the distance to the other vehicle (50) is closer. Also, the risk of change is set higher as the relative speed to the other vehicle (50) located behind increases. Conversely, the risk of change is set lower as the relative speed to the other vehicle (50) located in front increases. Also, if another vehicle is located to the side, the risk is set higher compared to the case where the other vehicle (50) is located only behind or in front.

[0198] As for change risk, for example, the sum of the values ​​calculated for each item is used.

[0199] Alternatively, a cost map, etc., may be generated from the placement or relative speed of other vehicles (50), and the value of the cost in the movable space may be estimated as a change risk.

[0200] In addition, information such as the turn signals of other vehicles (50) or information on movement prediction may be used.

[0201] In addition, the risk of change may be estimated according to the situation of the target lane (41), and the situation of the lane adjacent to the inner side of the target lane (41) as viewed from the vehicle (1).

[0202] In addition, the method for estimating the risk of change is not limited, and any method capable of estimating the possibility of contact or collision with other vehicles (50) may be used.

[0203] In the projection image determination unit (23), a process is executed to control the color or blinking of the target line (53) based on the change risk. Specifically, a color corresponding to the change risk is set as the color of the target line (53). Alternatively, when the change risk exceeds a certain value, an instruction to blink the target line (53) is output. In this case, the speed of blinking, etc., may be set according to the change risk.

[0204] As a result, it becomes possible to sufficiently support the driving operation of lane changing, and the safety of lane changing can be significantly increased.

[0205] FIG. 8 is a schematic diagram illustrating an example of a forward line (3a) projected when a lane change is performed. The forward line (3a) shown in FIG. 8A to FIG. 8C is projected when a preliminary operation for a lane change is detected. Here, the line projected to the right side (upper side in the drawing) of the vehicle (1) among the forward lines (3a) is the target line (53).

[0206] FIG. 8A is the forward line (3a) projected when lane change is safe. In this case, the color of the target line (53) is set to, for example, green. Also, the opposite line of the target line (53) (the left line of the vehicle (1)) is set to, for example, a color used in normal display processing (white, etc.). This is also the case for FIG. 8B and FIG. 8C.

[0207] Fig. 8B is a forward line (3a) that is projected when attention is required to change lanes. In this case, the color of the target line (53) is set to, for example, orange.

[0208] Fig. 8C is a forward line (3a) projected when lane change is dangerous. In this case, the color of the target line (53) is set to, for example, red.

[0209] The safety level (safe, caution, danger, etc.) of a lane change is determined, for example, by performing threshold processing regarding the change risk mentioned above. For example, if the change risk is between 0% and 30% of its maximum value, the safety level is determined as "safe"; if the change risk is between 30% and 70%, the safety level is determined as "caution"; and if the change risk is between 70% and 100%, the safety level is determined as "danger". Of course, the range for determining the change risk, etc., can be set arbitrarily.

[0210] In this way, in the present embodiment, the risk of change is divided into multiple levels for determination, and the target line (53) is set to a different color for each determined level. As a result, the driver can easily recognize the safety level of the lane change while looking forward.

[0211] In addition to the control of changing the color of the target line (53), the blinking, etc., may also be controlled. In this case, if the change risk exceeds a certain value, a process of blinking the target line (53) is executed. By doing so, it becomes possible to clearly inform the driver of the safety level of the lane change.

[0212] In addition, depending on the change risk, processing such as changing the width or length of the target line (53) may be performed.

[0213] In addition to the target line (53) of the front line (3a), the display of the center line (3b) or the rear line (3c) may also be controlled. Specifically, the display of the center line (3b) and the rear line (3c) projected onto the same side as the target line (53), that is, the target lane (41) of the lane change, is controlled.

[0214] For example, when a preliminary action for changing lanes is detected, the lines on the target lane (41) side of the center line (3b) and rear line (3c) are displayed in orange. This functions as a sign similar to a direction signal, for example, by a turn signal, and makes it possible to convey to other vehicles (50) that the vehicle (1) is changing lanes.

[0215] In addition, control may be implemented to change the color of the central line (3b) and the rear line (3c) according to the risk of change.

[0216] Returning to FIG. 6, when normal display processing or lane change display processing is completed, image data to be output to each projection unit (10) is generated by the image data generation unit (24) based on the data of the line pattern (3) (front line (3a) and rear line (3c)) (step 108). For example, a frame image representing the line pattern (3) is generated. Image data is generated by performing correction processing on this frame image according to the projection angle of the projection unit (10), etc. The generated image data is output to each projection unit (10).

[0217] Then, a corresponding line pattern (3) is projected based on image data by each projection unit (10) provided in the vehicle (1) (step 109). For example, a front line (3a) is projected by the projection units (10a and 10e) shown in FIG. 1, a center line (3b) is projected by the projection units (10b, 10c, 10f, 10g), and a rear line (3c) is projected by the projection units (10d and 10h).

[0218] FIG. 9 is a schematic diagram illustrating an example of a lane change in a merging lane. FIG. 9A to FIG. 9D sequentially illustrate, in four steps, the process of a vehicle (1) changing lanes to merge into the main line from an entry lane connected to the main line. Additionally, the dotted lines in the drawings indicate, for example, the range illuminated by the headlights of the vehicle (1).

[0219] In FIG. 9A, a vehicle (1) traveling in an entry lane (driving lane (40a)) enters a merging section where the entry lane connects with the main line. The main line is a two-lane road, and here, the lane to the left of the main line (adjacent lane (40b)) becomes the target lane (41) for lane change.

[0220] In the stage where the vehicle enters the merging section, no preliminary lane change operation is detected, and the projection pattern (2) of the vehicle (1) is a pattern set by normal display processing. For example, the front line (3a) is set to white, the center line (3b) to blue, and the rear line (3c) to red.

[0221] In FIG. 9B, a preliminary operation for lane change is detected, and lane change display processing is initiated. At this time, an adjacent lane (40b) is set as the target lane (41) from the direction in which the driver is looking or the direction indicated by the turn signal.

[0222] Additionally, in the target lane (41), other vehicles (50a) traveling to the right front of the vehicle (1) and other vehicles (50b) traveling to the right rear are detected. At this time, although there is a movable space to the right of the vehicle (1), the distance to other vehicles (50b) is relatively close, so the risk of change is determined to be at the caution level. Accordingly, the target line (53) is set to orange (see B in FIG. 8).

[0223] Meanwhile, the opposite side line of the target line (53) of the front line (3a) is set to a normal display color (white).

[0224] In addition, since lane change display processing has started, the center line (3b) and the rear line (3c) are also changed to orange.

[0225] In Fig. 9C, another vehicle (50b) that was previously in the rear right is driving along the side of vehicle (1). Because of this, there is a high probability of a collision when vehicle (1) changes lanes. Because of this, the risk of changing lanes is determined to be at a dangerous level. Therefore, the target line (53) is set to red (see Fig. 8C).

[0226] In this way, since the red target line (53) is projected onto the side changing lanes, the driver can recognize that changing lanes is dangerous while keeping their gaze forward.

[0227] In Fig. 9D, another vehicle (50b) that was previously on the side is driving ahead of vehicle (1) and is moving to the right. Additionally, another vehicle (50c) is detected behind the other vehicle (50b). Here, the position of the other vehicle (50c) is sufficiently far and the relative speed is low, so the risk of change is determined to be at a safe level. Accordingly, the target line (53) is set to green (see Fig. 8C).

[0228] By doing so, the driver can confirm that the lane change is possible without changing the direction of their gaze.

[0229] In addition, in Fig. 9D, the color of the part of the target line (53) projected within the merging section (i.e., the part up to the end of the driving lane (40a)) is set to green, and the part ahead of it is set to a different color (e.g., orange). The part projected within the merging section is the part where the vehicle (1) can drive. Also, the part ahead of the merging section is the part where the vehicle (1) cannot drive.

[0230] For example, in an image of the forward line (3a), the drivable range is detected by detecting the position where the forward line (3a) is distorted by an obstacle (guardrail, etc.) (see FIG. 19). Alternatively, the drivable range may be detected from surrounding map information and GPS positioning information.

[0231] In addition, in Fig. 9D, the line opposite to the target line (53) is also separated in the same way.

[0232] In this way, in the present embodiment, the target line (53) is divided into a part where the vehicle (1) can travel and a part where the vehicle (1) cannot travel. And, control of the indication according to the change risk is applied to the part where the vehicle (1) can travel.

[0233] By doing so, it becomes possible to present the driver with the distance to the end of the merging section, and thus provide support for gauging the timing of lane changes.

[0234] In the above, in the controller (20) related to the present embodiment, a projection pattern (2) is projected onto the surrounding road surface from a projection unit (10) provided in the vehicle (1). The display of this projection pattern (2) is controlled based on surrounding environment information regarding the surrounding environment of the vehicle (1). By doing so, for example, the situation in which the vehicle (1) is placed can be presented to the inside and outside of the vehicle (1) through the projection pattern (2), thereby making it possible to increase safety during driving.

[0235] When driving, if you are changing lanes or merging, it is necessary to check the road ahead while assessing the distance and relative speed to vehicles to the side or behind you on an incline, and to determine the timing of the lane change. Furthermore, depending on traffic conditions, it may be necessary to perform these tasks instantaneously. For instance, at night, it can be difficult to instantaneously judge relative speeds. On the other hand, focusing on the side or behind you on an incline can lead to neglecting to check the road ahead, increasing the risk of traffic accidents or near-misses. While methods such as using electronic mirrors or displaying warnings or relative speeds on the instrument panel can be considered to mitigate these risks during lane changes, both involve shifting your gaze, which still raises concerns that safety checks ahead may be neglected.

[0236] In this embodiment, the display of the forward line (3a) projected onto the side targeted for lane change is controlled by using surrounding environment information in which the vehicle (1) is driving. Since the forward line (3a) is projected onto the front of the driver, the driver can check the presence or absence of a vehicle behind, the distance, relative speed, etc., while keeping their eyes on the front, thereby making it possible to provide safe lane changes or merging to many people.

[0237] <Second Embodiment>

[0238] A projection device of a second embodiment related to the present technology will be described. In the following description, parts that are similar to the configuration and operation of the projection device (100) described in the above embodiment will be omitted or simplified.

[0239] Road marking device (caution for following vehicles)

[0240] Solving the driver's problem by displaying the wheel trajectory on the road surface

[0241] FIG. 10 is a block diagram illustrating an example of the configuration of a projection device related to a second embodiment.

[0242] In this embodiment, as a display control of a projection pattern (2) using surrounding environment information, the display of a second pattern projected at the rear of the driving direction is controlled in stages according to the collision risk estimated from information regarding a subsequent vehicle of the vehicle (1) (subsequent vehicle information).

[0243] As illustrated in FIG. 10, the projection device (200) has a projection unit (210), a vehicle information sensor unit (211), an ambient environment sensor unit (213), a memory unit (215), and a controller (220). Among these, the projection unit (210), the vehicle information sensor unit (211), and the memory unit (215) are configured in the same way as, for example, the projection unit (10), the vehicle information sensor unit (11), and the memory unit (15) illustrated in FIG. 4.

[0244] The surrounding environment sensor unit (213) has a sensor that detects the state of the surrounding environment of the vehicle (1).

[0245] In this embodiment, as an ambient environment sensor unit (213), a rear sensor (such as a rear camera, radar sensor, ultrasonic sensor, LiDAR sensor, etc. installed at the rear of the vehicle (1)) is provided to detect objects at the rear of the vehicle (1).

[0246] In addition, a sensor (such as a front camera or a side camera) similar to the surrounding environment sensor unit (213) shown in FIG. 4 may be provided.

[0247] The controller (220) controls the operation of each block of the projection device (200). In this embodiment, the CPU of the controller (220) executes a program related to this embodiment stored in the memory unit (215), thereby realizing the vehicle information acquisition unit (221), trajectory calculation unit (222), projection image determination unit (223), image data generation unit (224), and surrounding environment recognition unit (226) as functional blocks.

[0248] Among these, the vehicle information acquisition unit (221), the trajectory calculation unit (222), and the image data generation unit (224) are configured in the same way as, for example, the vehicle information acquisition unit (21), the trajectory calculation unit (22), and the image data generation unit (24) shown in FIG. 4.

[0249] The projection image determining unit (223) determines the display content or display parameters of the projection pattern (2) and outputs the data of the projection pattern (2).

[0250] In this embodiment, the projection image determining unit (223) estimates the collision risk with the subsequent vehicle based on subsequent vehicle information and changes the second pattern discontinuously according to the collision risk.

[0251] Here, the subsequent vehicle information is information regarding a subsequent vehicle traveling behind the vehicle (1), and is surrounding environment information calculated by the surrounding environment recognition unit (226) described later.

[0252] Also, collision risk is a parameter indicating the possibility that a subsequent vehicle will collide with vehicle (1).

[0253] In the projection image determination unit (223), the display parameters of the second pattern are set to change stepwise, for example, according to the level of collision risk.

[0254] FIG. 11 is a block diagram illustrating an example configuration of a surrounding environment recognition unit (226).

[0255] The surrounding environment recognition unit (226) performs recognition processing regarding the surrounding environment of the vehicle (1) based on the output of the surrounding environment sensor unit (213). It detects objects (pedestrians, other vehicles, curbs, etc.) existing around the vehicle (1) and calculates various information regarding the objects.

[0256] The surrounding environment recognition unit (226) has an object detection unit (230) and a relative speed calculation unit (231). The object detection unit (230) detects distance information or location information of surrounding objects of the vehicle (1). The relative speed calculation unit (231) detects the relative speed of each object with respect to the vehicle (1).

[0257] The object detection unit (230) and the relative speed calculation unit (231) are configured similarly to the object detection unit (30) and the relative speed calculation unit (31) shown in FIG. 4, for example.

[0258] In this embodiment, among the objects surrounding the vehicle (1), a subsequent vehicle, particularly one behind the vehicle (1), is detected by the object detection unit (230), and its distance information or position information is calculated. Then, the relative speed of the subsequent vehicle is calculated by the relative speed calculation unit (231).

[0259] That is, the surrounding environment recognition unit (226) calculates subsequent vehicle information regarding the vehicle (1)'s subsequent vehicle based on detection results such as a rear camera. Accordingly, the surrounding environment recognition unit (226) calculates and acquires subsequent vehicle information. The subsequent vehicle information includes ID information, attribute information, distance information indicating the distance between vehicles (or location information indicating a relative position), and speed information indicating a relative speed. Additionally, the subsequent vehicle includes automobiles, motorcycles, bicycles, etc.

[0260] FIG. 12 is a flowchart illustrating a basic operation example of a projection device (200). FIG. 13 is a schematic diagram illustrating an example of a rear pattern according to collision risk.

[0261] The processing illustrated in FIG. 13 is a loop processing that is repeatedly executed during the operation of the projection device (200), for example. In addition, this processing may be executed when a subsequent vehicle is detected, when the relative speed of the subsequent vehicle is fast, or when the distance between the vehicle and the subsequent vehicle is short.

[0262] First, speed-related information is acquired by the vehicle information acquisition unit (221) (step 201), and then, the trajectory of the vehicle (1) (predicted trajectory (5) and passing trajectory (6)) is calculated by the trajectory calculation unit (222) (step 202). Then, a subsequent vehicle is detected by the surrounding environment recognition unit (226) and subsequent vehicle information is calculated (step 203).

[0263] Next, the collision risk is estimated by the projection image determination unit (223) (step 204).

[0264] The collision risk is set to a higher value as, for example, the relative distance to the following vehicle is closer and the relative speed to the following vehicle is faster. As an example, the relative speed / relative distance is calculated as the collision risk. Alternatively, the collision risk may be calculated from either the relative distance or the relative speed. Additionally, the collision risk may be calculated by adding the deceleration amount of the vehicle (1) by referring to brake information, etc.

[0265] Next, it is determined whether the collision risk is greater than or equal to a first threshold (step 205). The first threshold is, for example, a threshold for detecting cases where the collision risk is relatively high. The first threshold is, for example, set to a value of about 30% to 50% of the maximum value of the collision risk.

[0266] In addition, the first threshold can be set appropriately.

[0267] When the collision risk is less than the first threshold ("No" in Step 205), the projection image determining unit (223) determines that the second pattern is projected as a normal display (Step 206). The normal display is a display method (normal display processing) in normal brake operation.

[0268] In FIG. 13A, a schematic example of a second pattern (rear line (3c)) in a normal display is shown. Here, a rear line (3c) is projected with a relatively narrow width. Also, the rear line (3c) is set to a red color, for example, of the same color as the brake lamp (35).

[0269] Returning to Fig. 12, if the collision risk is greater than or equal to the first threshold ("Yes" in Step 205), it is determined whether the collision risk is greater than or equal to the second threshold (Step 207). The second threshold is, for example, a threshold that detects a state where the collision risk is sufficiently high. For example, a value greater than or equal to 50% of the maximum value of the collision risk is set as the second threshold.

[0270] In addition, the second threshold can be set appropriately.

[0271] When the collision risk is less than the second threshold ("No" in Step 207), the projection image determining unit (223) determines that the width of the second pattern is displayed as a relatively thick fixed width (Step 208). This method can be described as a method of statically highlighting and displaying the second pattern.

[0272] For example, the width of the second pattern is set to the maximum width. Alternatively, the width of the second pattern may be set to approximately 80% of the maximum width, or 90% of the maximum width.

[0273] In FIG. 13B, a schematic example of a second pattern (rear line (3c)) is shown with a thick width. Here, a rear line (3c) is projected with a fixed width that is thicker than a normal display. As a result, since a thick rear line (3c) is additionally projected onto the brake lamp (35), it is possible to emphasize to subsequent vehicles that the risk of collision with the vehicle (1) is high.

[0274] Returning to FIG. 12, if the collision risk is greater than or equal to the second threshold ("Yes") of Step 207, the projection image determining unit (223) determines that the second pattern is projected as a blinking display (Step 209).

[0275] A flickering display is a display method that displays the second pattern by making it flicker, and can be described as a method that displays the second pattern by dynamically highlighting it.

[0276] In Fig. 13C, a schematic example of a second pattern (rear line (3c)) display in the flashing display is illustrated. In the flashing display, for example, the width of the rear line (3c) is set to be equal to or greater than the fixed width of Fig. 13B. Then, at least a portion of the rear line (3c) is displayed to flash. In addition, when a portion of the rear line (3c) flashes, a display in which the flashing portion moves is also possible. That is, the rear line (3c) may be displayed as an animation. By doing so, since the rear line (3c) that additionally flashes is projected onto the brake lamp (35), it becomes possible to sufficiently emphasize and convey the risk of collision with the vehicle (1). As a result, it becomes possible to effectively alert subsequent vehicles, etc.

[0277] Returning to FIG. 12, in steps 206, 208, and 209, after the display of the second pattern is set, image data is generated by the image data generation unit (224) to be output to each projection unit (210) based on the set data (step 210). Then, a corresponding line pattern (3) is projected by each projection unit (210) provided in the vehicle (1) based on the image data (step 211).

[0278] In this way, in the present embodiment, the projection image determining unit (223) increases the width of the second pattern when the collision risk is greater than or equal to the first threshold and less than the second threshold which is greater than the first threshold, and turns the second pattern on and off when the collision risk is greater than or equal to the second threshold.

[0279] By doing so, it becomes possible to easily convey to the subsequent vehicle the possibility of a collision with the vehicle (1) ahead.

[0280] When calculating the collision risk, the degree of brake operation, etc., may be used. For example, as the braking strength increases, the deceleration of the vehicle (1) increases, so the collision risk is set high. By doing so, it becomes possible to sufficiently avoid collisions with subsequent vehicles.

[0281] In the operation example illustrated in FIG. 12, discontinuous display control of the second pattern according to collision risk was described. Instead of collision risk, it is also possible to control the second pattern discontinuously according to, for example, the degree of brake operation in the vehicle (1).

[0282] Specifically, the degree of brake operation performed by the driver is estimated by the projection image determination unit (223). Then, a process is executed to change the display of the rear pattern step by step according to the degree of brake operation. This process is, for example, a process illustrated in FIG. 12, which is executed using the degree of brake operation of the vehicle (1) instead of the collision risk.

[0283] In addition, in the processing described with reference to Fig. 12, a pattern (rear line) serving as a warning indicator to a subsequent vehicle is displayed by switching. Furthermore, it is also possible to emit a warning sound toward a subsequent vehicle, for example, using a directional speaker. In this case, since an auditory warning sound is transmitted along with warnings such as flashing indicators, it is possible to effectively convey the risk of brake operation or collision.

[0284] Accidents have been reported in which following vehicles collide with a stopped vehicle, such as at the rear of a traffic signal or in a traffic jam. As a measure to prevent such accidents, operations such as "flashing the brake lights by pumping the brake" or "turning on the hazard lights" are recommended as self-defense measures. As a method to automate these operations, for example, one approach is to monitor following vehicles and automatically flash the hazard lights if there is a risk of a collision. However, since these operations are not always sufficiently effective, there is a need for effective warning methods that allow the driver of the following vehicle to realize that their vehicle is stopped.

[0285] In this embodiment, a rear line (3c) (second pattern) displaying the trajectory of a wheel, etc., is projected onto the rear of a moving vehicle (1). Depending on the risk of collision with a following vehicle, this second pattern is switched to a display with a thicker line width (see B in FIG. 13) or a blinking display (see C in FIG. 13). In this way, by using dynamic displays in addition to static displays, it becomes possible to urge the driver of a following vehicle to pay attention to the road ahead. By doing so, it becomes possible to sufficiently avoid the occurrence of collisions with following vehicles.

[0286] <Third Embodiment>

[0287] In this embodiment, a pair of line patterns (3) representing the width of the vehicle (1) are projected as a first pattern (typically, a front line (3a)) projected in the forward direction of travel of the vehicle (1). Thus, the first pattern functions as an extension of the vehicle width. And the display of the line pattern (3) corresponding to the vehicle width is controlled based on the positional relationship between the vehicle (1) and objects around it.

[0288] In this way, by marking the extension line of the vehicle width on the road surface, it becomes possible to present the distance to surrounding objects in an easy-to-understand manner, thereby assisting the driver in recognizing the vehicle width.

[0289] FIG. 14 is a block diagram illustrating an example of the configuration of a projection device related to a third embodiment.

[0290] As illustrated in FIG. 14, the projection device (300) has a projection unit (310), a vehicle information sensor unit (311), an ambient environment sensor unit (313), a memory unit (315), and a controller (320). Among these, the projection unit (310), the vehicle information sensor unit (311), the ambient environment sensor unit (313), and the memory unit (315) are configured in the same way as, for example, the projection unit (10), the vehicle information sensor unit (11), the ambient environment sensor unit (13), and the memory unit (15) illustrated in FIG. 4.

[0291] The controller (320) controls the operation of each block of the projection device (300). In this embodiment, the CPU of the controller (320) executes a program related to this embodiment stored in the memory unit (315), thereby realizing the vehicle information acquisition unit (321), trajectory calculation unit (322), projection image determination unit (323), image data generation unit (324), and surrounding environment recognition unit (326) as functional blocks.

[0292] Among these, the vehicle information acquisition unit (321), the trajectory calculation unit (322), and the image data generation unit (324) are configured in the same way as, for example, the vehicle information acquisition unit (21), the trajectory calculation unit (22), and the image data generation unit (24) shown in FIG. 4.

[0293] The projection image determining unit (323) determines the display content or display parameters of the projection pattern (2) and outputs the data of the projection pattern (2).

[0294] In this embodiment, a pair of lines representing the width of the vehicle (1) as a front line (3a) (first pattern) is generated by the projection image determining unit (323).

[0295] The vehicle width of the vehicle (1) is, for example, the maximum width of the vehicle body. Alternatively, the width including the side mirrors, etc., may be used as the vehicle width.

[0296] For example, if the width of each line constituting the front line (3a) is sufficiently narrow (e.g., less than 10 cm), the spacing between the centers of each line (center width) is set as the width of the vehicle (1). Also, if the width of each line is relatively thick (e.g., 10 cm or more), the spacing between the outer edge of each line (outer width) is set as the width of the vehicle (1). In addition, the spacing between lines may be set according to the shape of each line, etc., so that the width of the vehicle (1) can be indicated.

[0297] In addition, the front line (3a) is set to a shape that represents the predicted trajectory (5) of the vehicle (1), just like in the above embodiment.

[0298] In the following, the front line (3a) indicating the vehicle width is referred to as the vehicle width line. Additionally, the lines projected onto the right and left sides of the vehicle (1) within the vehicle width line (front line (3a)) are referred to as the right line and the left line, respectively.

[0299] In addition, in the projection image determining unit (323), the display of a pair of lines is individually controlled based on position relationship information indicating the position relationship between the vehicle (1) and surrounding objects.

[0300] Position relationship information is information capable of indicating the position relationship between the vehicle (1) and objects existing in the surrounding environment of the vehicle (1) (other vehicles, pedestrians, curbs, guardrails, white lines of the road, etc.), and is surrounding environment information acquired by the surrounding environment recognition unit (326). For example, the distance between an object and the vehicle (1), and the relative position or relative orientation of an object viewed from the vehicle (1) are position relationship information. Additionally, the position information of the vehicle (1) and the object may be used as position relationship information.

[0301] In the projection image determination unit (323), for example, based on the positional relationship between the vehicle (1) and the object, the approach of the vehicle (1) to the object or the distance of the vehicle (1) from the object is detected. To display the results of such detection, display parameters (typically color and brightness) for the right line and the left line are each set. This point will be described in detail later.

[0302] In addition, in this embodiment, a projection pattern (2) (hereinafter referred to as a boundary line pattern) is generated to indicate a boundary line (51) that divides the lane in which the vehicle (1) travels, in addition to the vehicle width line. This boundary line pattern is generated using, for example, the detection result of a white line detected by a lane detection unit (332) described later. In addition, if, for example, a change in the attitude of the vehicle (1) can be detected, a process is performed to correct the boundary line pattern according to the change in the attitude of the vehicle (1).

[0303] FIG. 15 is a block diagram illustrating an example configuration of a surrounding environment recognition unit (326).

[0304] The surrounding environment recognition unit (326) detects objects (pedestrians, other vehicles, curbs, etc.) present around the vehicle (1) based on the output of the surrounding environment sensor unit (313), and calculates positional relationship information regarding each object as surrounding environment information.

[0305] As illustrated in FIG. 15, the surrounding environment recognition unit (326) has a projection situation determination unit (330), a lane detection unit (332), and a space recognition unit (333).

[0306] The projection situation determination unit (330) detects the vehicle width line from the front image captured by the front camera while the vehicle width line (projection pattern (2)) is projected.

[0307] In the process of detecting the vehicle width line, data (shape, color, width, length, brightness, etc.) of the vehicle width line (forward line (3a)) output from the projection image determining unit (323) is used. By doing so, it is possible to detect the vehicle width line projected onto the forward image with high precision.

[0308] Likewise, the projection situation determination unit (330) detects a boundary line pattern from the front image.

[0309] In addition, the projection situation determination unit (330) determines the situation in front of the vehicle (1) by using how the detected vehicle width line is projected onto the road surface. For example, if the vehicle width line is projected without distortion (such as refraction), the road surface in front is determined to be flat. Or, if the vehicle width line is distorted, it is determined that there is an obstacle in front.

[0310] If it is determined that there is an obstacle, information indicating the distorted part among the right line and left line constituting the vehicle width line is calculated. Specifically, the location of the bent part in each line (e.g., a position 1 / 4 from the leading edge) is calculated. This information becomes positional relationship information indicating the relative position of the obstacle to the vehicle (1).

[0311] In this way, the projection situation determination unit (330) detects the location of an obstacle on the line by detecting line distortion from an image (forward image) in which the line (right line and left line) is captured as position circular information. By utilizing line distortion within the image in which the line is captured, it is possible to easily detect an obstacle existing on the path of the vehicle (1).

[0312] In this embodiment, the projection situation determination unit (330) corresponds to the second detection unit.

[0313] From the projection situation determination unit (330), the detection result of the forward projection pattern (2) (vehicle width line and boundary line pattern) and the relative position of the obstacle are output.

[0314] The lane detection unit (332) is configured in the same way as the lane detection unit (32) shown in FIG. 5, for example, and detects the lane (lane) on the road where the vehicle (1) is traveling. More specifically, a process for detecting objects such as white lines or curbs on the road is executed, and their positions are detected. The white lines or curbs are boundary lines (51) indicating the range that the vehicle (1) must travel. Therefore, it can be said that the relative position of the boundary line (51) is detected as positional relationship information of the objects in the lane detection unit (332).

[0315] The space recognition unit (333) recognizes the situation of the surrounding space of the vehicle (1) based on the detection results of the projection situation determination unit (330) and the lane detection unit (332).

[0316] In this embodiment, the distance between the boundary line (51) and the vehicle width line is calculated based on the position information of the boundary line (51) of the driving lane (40) in which the vehicle (1) travels and the position information of the vehicle width line. For example, the distance between the right line and the right boundary line (51) of the vehicle (1) and the distance between the left line and the left boundary line (51) of the vehicle (1) are each calculated. This processing is performed based on a front image taken of the front of the vehicle (1).

[0317] In this way, the spatial recognition unit (333) detects the distance between the boundary line (51) and the line from an image in which the boundary line (51) and the line (right line and left line) are captured simultaneously. By using an image in which both the boundary line (51) and the line are captured, it is possible to easily calculate the distance of each line.

[0318] In this embodiment, the space recognition unit (333) corresponds to the first detection unit.

[0319] The distance of each line representing the boundary line (51) and the vehicle width represents the distance to the lane of the vehicle (1), that is, the distance between the vehicle (1) and the boundary line (51). In the space recognition unit (333), it is determined whether these distances are within a predetermined threshold.

[0320] Additionally, the spatial recognition unit (333) calculates the amount of deviation of the boundary line pattern relative to the actual boundary line (51) (white line or curb) from the location information of the boundary line (51) of the driving lane and the boundary line pattern. This amount of deviation is used for calibration of the position of the boundary line pattern in the projection image determination unit (323).

[0321] From the spatial recognition unit (333), the distance between the boundary line (51) and the line, the result of the determination thereof, and the amount of deviation of the boundary line pattern are output.

[0322] In this embodiment, as illustrated in FIG. 15, the output of the projection situation determination unit (330) and the space recognition unit (333) becomes the final output of the surrounding environment recognition unit (326). Accordingly, projection situation information (information indicating whether there is distortion of the vehicle width line and location information of the distortion location (obstacle)) is output from the surrounding environment recognition unit (326). In addition, the distance between the boundary line (51) and the line, the determination result, and the amount of misalignment of the boundary line pattern are output.

[0323] In addition, the method of detecting the location of objects around the vehicle (1) is not limited. For example, Simultaneous Localization and Mapping (SLAM) technology, which self-generates a three-dimensional map of the vehicle (1), may be used.

[0324] In addition, an object detection unit as shown in Fig. 5, etc. may be provided, and the position and distance of an object may be detected using a radar sensor, an ultrasonic sensor, a LiDAR sensor, etc.

[0325] FIG. 16 is a flowchart illustrating a basic operation example of a projection device (300).

[0326] The processing illustrated in FIG. 16 is a loop processing that is repeatedly executed during the operation of the projection device (300), for example, and is typically performed with the front line (3a), which becomes the vehicle width line, as the control target when the vehicle (1) is moving forward. Additionally, when the vehicle (1) is moving backward, processing may be performed with the rear line (3c), which is generated as the vehicle width line.

[0327] First, speed-related information is acquired by the vehicle information acquisition unit (321) (step 301), and then, the trajectory of the vehicle (1) (predicted trajectory (5) and passing trajectory (6)) is calculated by the trajectory calculation unit (222) (step 302).

[0328] Next, the surrounding environment recognition unit (326) recognizes the positional relationship between the vehicle (1) and surrounding objects and performs a process to calculate positional relationship information (step 303).

[0329] Specifically, the projection situation determination unit (330) calculates positional relationship information between the boundary line (51) and the vehicle width line (determination result of the distance between the boundary line (51) and the line).

[0330] In addition, if an obstacle exists on the vehicle width line by the spatial recognition unit (333), positional relationship information regarding the obstacle (information indicating whether there is a distortion of the vehicle width line and location information of the distortion location (obstacle)) is calculated.

[0331] Next, the display parameters of the vehicle width line (8) are set by the projection image determining unit (323) (step 304).

[0332] For example, if it is determined that the distance between the boundary line (51) and the vehicle width line (right line and left line) is smaller than a predetermined threshold, the settings such as color or flashing of the line that was determined are changed (see FIG. 17 and FIG. 18). By doing so, it becomes possible to warn the driver of the fact when the vehicle (1) is approaching the edge of the lane (approaching the boundary line (51)).

[0333] In addition, for example, when an obstacle is detected on the vehicle width lines (right line and left line), the settings such as color or flashing of the line where the obstacle is detected are changed so that the location of the detected obstacle can be known (see FIG. 19 and FIG. 20). By doing so, it is possible to warn the driver in advance of the existence of an obstacle that may be at risk of contact when the vehicle (1) proceeds as is.

[0334] In addition, if no approach to the boundary line (51) or obstacles are detected, the projection image determining unit (323) performs a normal display processing (normal display processing).

[0335] Next, image data is generated by the image data generation unit (324) based on the projected image data and output to each projection unit (310) (step 305). Then, a corresponding line pattern (3) is projected by each projection unit (210) provided in the vehicle (1) based on the image data (step 306).

[0336] Below, the handling of approach to the boundary line (51) and the case where an obstacle is detected will be explained in detail.

[0337] FIG. 17 is a schematic diagram illustrating an example of a scene in which a vehicle width line is projected.

[0338] In FIG. 17A, the scenery seen from the driver when the vehicle (1) is traveling straight on a single-lane road (driving lane (40a)) is schematically illustrated.

[0339] Here, a vehicle width line (8) (front line (3a)) representing the vehicle width is projected as a projection pattern (2) in front of the vehicle (1). Among these, the lines projected to the right and left sides of the vehicle (1) become the right line (8R) and the left line (8L).

[0340] In addition, a boundary line pattern (9) representing the boundary line (51) of the driving lane (40a) is projected in front of the vehicle (1). Among these, the boundary line pattern (9a) projected on the right side of the drawing is projected to overlap with the white line (center line) representing the boundary with the opposing lane. In addition, the boundary line pattern (9b) projected on the left side of the drawing is projected to overlap with the curb that forms the boundary with the sidewalk.

[0341] In this embodiment, the color or blinking of each line is controlled by the projection image determining unit (323) based on the positional relationship between the boundary line (51), the right line (8R), and the left line (8L). Specifically, the distance between the boundary line (51) and each line indicating the vehicle width (hereinafter referred to as the approach distance) is used as an indicator indicating the positional relationship.

[0342] In addition, in A of FIG. 17, arrows indicating the distance between the right line (8R) and the right boundary line (51) (white line) (right approach distance) and the distance between the left line (8L) and the left boundary line (51) (curb) (left approach distance) are respectively shown. Additionally, the arrows indicating the approach distance may be displayed as a projection pattern (2) or may not be displayed.

[0343] As described above, the spatial recognition unit (333) determines whether the approach distance to the right and left is lower than a predetermined threshold. Then, the projection image determination unit (323) determines the color or presence or absence of blinking of the right line (8R) and left line (8L) according to the determination result regarding the approach distance.

[0344] In FIG. 17A, the approach distance to the right and left is determined to be greater than a predetermined threshold. In this case, the colors of the right line (8R) and the left line (8L) are both set to colors typically used in display processing (e.g., white or blue). Additionally, the colors of the boundary line patterns (9a and 9b) are also set to colors typically used in display processing (e.g., green).

[0345] In FIG. 17B, the vehicle (1) is driven off to the right compared to the state shown in FIG. 17A, and the approach distance between the right line (8R) and the right boundary line (51) is determined to be smaller than a predetermined threshold. In this case, the color of the right line (8R) is set to a color different from the normal display processing color (e.g., orange) by the projection image determining unit (323). Also, in the example shown in FIG. 17B, along with the right line (8R), the right boundary line pattern (9a) is also set to a color different from the normal display processing color (e.g., red). Alternatively, the right line (8R) and the boundary line pattern (9a) may be set to blink.

[0346] In addition, the approach distance on the left is determined to be greater than a predetermined threshold. Because of this, the color of the left line (8L) and the left boundary line pattern (9b) does not change.

[0347] In this way, in the present embodiment, when the distance between the boundary line (51) and the right line (8R) or the left line (8L) is smaller than a predetermined threshold, a process is performed to change the color of the line or to make the line blink.

[0348] By doing so, it becomes possible to clearly convey to the driver that the vehicle width line (8) is approaching an opposing lane or sidewalk. As a result, it becomes possible to intuitively recognize whether the current driving position is appropriate, thereby enabling safe driving. Furthermore, it becomes possible to encourage driving while being conscious of the vehicle width, thereby supporting the driver in learning the appropriate vehicle width distance.

[0349] FIG. 18 is a schematic diagram illustrating an example of a vehicle width line (8) according to the approach distance. Here, the approach distance is determined in three stages, and at each stage, the color of the vehicle width line (8) (front line (3a)) is set to a different color. This determination process can be performed, for example, by threshold processing using two thresholds.

[0350] A of FIG. 18 is a vehicle width line (8) projected when the approach distances to the right and left are both within a safe range. In this case, the colors of the right line (8R) and the left line (8L) are set to, for example, the colors of the standard display (white or blue). This is the same setting as, for example, A of FIG. 17.

[0351] In Fig. 18B, the approach distance to the right is determined to be within the range requiring caution, and the approach distance to the left is determined to be within the safe range. In this case, the color of the right line (8R) of the vehicle width line (8) is set to, for example, orange. Also, the color of the left line (8L) is not changed.

[0352] In Fig. 18C, the approach distance to the right is determined to be within a dangerous range, and the approach distance to the left is determined to be within a safe range. In this case, the color of the right line (8R) of the vehicle width line (8) is set to, for example, red. Also, the color of the left line (8L) is not changed.

[0353] In addition to the control that changes the color of the target line (53), the blinking, etc., may also be controlled. In this case, for example, in the range that requires caution, the target line is set to orange and blinks, and in the dangerous range, the target line is set to red and the frequency of blinking is increased.

[0354] As a result, the driver can clearly recognize the safety level of the current driving position while keeping their eyes on the road ahead.

[0355] In addition to the method of determining the approach distance, the display of the vehicle width line (8) may also be controlled by using the output of a proximity sensor (a distance measuring sensor such as a radar sensor). For example, when approaching an object (such as a guardrail or another vehicle) in the vicinity of the vehicle (1), the color of the line on the side where the approach was detected changes and flashes. By doing so, it becomes possible to prevent contact accidents, such as those occurring not only in front of the vehicle (1) but also on the side or rear.

[0356] FIG. 19 is a schematic diagram illustrating an example of a vehicle width line (8) projected onto an obstacle. FIG. 19 schematically illustrates a scene in which the vehicle width line (8) intersects an obstacle (55) in front of the vehicle (1).

[0357] For example, among the vehicle width lines (8), the right line (8R) intersects with a rectangular obstacle (55a). The right line (8R), projected along the ground surface, is bent at the point where the side of the obstacle (55a) contacts the ground surface and is projected onto the side of the obstacle (55a).

[0358] In this case, the projection situation determination unit (330) detects a point (inversion point (P1)) where the right line (8R) is distorted based on the original shape of the right line (8R). This inversion point (P1) is used as location information for the obstacle (55a). Additionally, the part forward of the inversion point (P1) becomes an intersection part (56) that intersects with the obstacle (55a).

[0359] Also, for example, among the vehicle width lines (8), the middle part of the left line (8L) intersects with an obstacle (55b) placed on the ground surface with a cylindrical curved surface facing upward. Here, only the intersection part (56) where the left line (8L) intersects with the curved surface of the obstacle (55b) is distorted, and the front and rear parts of the intersection part are properly projected.

[0360] In this case, the projection situation determination unit (330) detects the front inflection point (P2) and the rear inflection point (P3) where the left line (8L) is distorted, based on the original shape of the left line (8L). These inflection points (P2 and P3) are used as location information for the obstacle (55b). Additionally, the part between the inflection points (P2 and P3) becomes the intersection part (56) where the obstacle (55b) is located.

[0361] In this way, when an intersection (56) is detected, the vehicle width line (8) is projected onto the obstacle (55), and the display of the vehicle width line (8) is controlled. Specifically, by the projection image determining unit (323), the right line (8R) and the left line (8L) are separated from the intersection (56) with the obstacle (55) and other parts based on the position (inflection point) of the obstacle (55).

[0362] For example, for the right line (8R), the color of the intersection part (56) ahead of the inflection point (P1) is set to red, and the part behind the inflection point (P1) is set to the normal display color. Also, for example, for the left line (8L), the color of the intersection part (56) sandwiched between the inflection points (P2 and P3) is set to red, and the part ahead of the inflection point (P2) and the part behind the inflection point (P3) are set to the normal display color.

[0363] By doing so, it becomes possible to easily convey to the driver the existence or location of an obstacle (55) that may be contacted when the vehicle (1) moves forward.

[0364] FIG. 20 is a schematic diagram illustrating an example of a vehicle width line (8) projected when an obstacle (55) is detected. Here, the safety level regarding the obstacle (55) is determined in three stages, and in each stage, the color of the vehicle width line (8) (front line (3a)) is set to a different color. The safety level of the obstacle (55) is set to be lower (more dangerous), for example, as the distance to the obstacle (55) is closer, or as the length of the obstacle (55) (intersection part (56)) is longer. Also, the higher the driving speed of the vehicle (1), the lower the safety level is set. In addition, the method of determining the safety level regarding the obstacle (55) is not limited.

[0365] A of FIG. 20 is a vehicle width line (8) projected when, for example, no obstacle (55) is detected and it is safe. In this case, the colors of the right line (8R) and the left line (8L) are set to the usual display colors (white or blue-based colors) throughout. Also, even when an obstacle (55) is detected, the colors of each line do not need to be changed, such as when the location is sufficiently separated.

[0366] Fig. 20B is a vehicle width line (8) projected when the safety level of an obstacle (55) is at a level requiring caution. Here, an obstacle is detected at a location relatively far from the vehicle (1). In this case, the color of the intersection part (56) of the right line (8R) and the left line (8L) is set, for example, to orange. Or the intersection part (56) is set to flash. Also, the color other than the intersection part (56) is set, for example, to a normal display color.

[0367] C of FIG. 20 is a vehicle width line (8) projected when the safety level of an obstacle (55) is at a dangerous level. Here, an obstacle is detected near the vehicle (1). In this case, the color of the intersection part (56) is set to, for example, red. Or the flashing speed of the intersection part (56) is set high.

[0368] The color other than the intersection part (56) is set to, for example, the normal display color. It is not limited to this, and, for example, the color other than the intersection part (56) may be changed to indicate the presence of dangerous obstacles (55).

[0369] A sense of vehicle width is a fundamental skill of driving. Vehicle width is important in various situations, such as determining whether to enter narrow roads, brushing past oncoming vehicles, maintaining a driving position within a lane, and parking close to guardrails or walls. Auxiliary means for accurately perceiving vehicle width include, for example, corner poles and fender mirrors, but these auxiliary components have rarely been used in recent years because they detract from the vehicle's appearance.

[0370] In addition, while the number of vehicles with a wide forward view is increasing to make it easier to get a sense of the vehicle width, the perception of the vehicle width relies heavily on the driver's "skill."

[0371] In this embodiment, a vehicle width line (8) indicating the vehicle width is projected by projecting light. The display of the right line (8R) and left line (8L) of the vehicle width line (8) is controlled according to the positional relationship between the vehicle (1) and objects in the surrounding environment. By doing so, it becomes possible to intuitively present information to the driver, for example, that the current driving position is skewed toward the white line or that contact with an obstacle is likely. This enhances the driver's driving safety and provides support for acquiring a correct sense of vehicle width.

[0372] Additionally, the projection part (310) for projecting the projection pattern (2) can be configured to converge with the main body of the vehicle (1). Because of this, it is possible to provide a vehicle width sense aid that does not affect the appearance by using the projection device (300).

[0373] <Other forms of implementation>

[0374] The present technology is not limited to the embodiments described above and can realize various other embodiments.

[0375] In the above embodiment, a pattern representing the vehicle's trajectory (passage trajectory or predicted trajectory) is set primarily as a line pattern. It is not limited to this, and, for example, a straight line pattern extending along the front-rear direction of the vehicle (1) can be fixed and used. Alternatively, a pattern with closed ends of left and right lines may be used. In addition, line patterns of any shape, size, or color may be used as projection patterns.

[0376] In the foregoing, a controller was exemplified as one embodiment of an information processing device related to the present technology. However, the controller may be configured separately, and the information processing device related to the present technology may be realized by any computer connected to the controller via wired or wireless means. For example, the information processing method related to the present technology may be executed by a cloud server. Alternatively, the information processing method related to the present technology may be executed by linking the controller with another computer.

[0377] That is, the information processing method and program related to the present technology can be executed not only in a computer system composed of a single computer, but also in a computer system in which multiple computers operate in conjunction. Furthermore, in the present disclosure, the term "system" refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all components are housed in the same housing. Accordingly, multiple devices housed in separate housings and connected via a network, and a single device housing multiple modules within a single housing, are all considered systems.

[0378] The execution of information processing methods and programs related to the present invention by a computer system includes both cases where, for example, the acquisition of surrounding environment information and the control of the display of projection patterns are executed by a single computer, and cases where each processing is executed by a different computer. Furthermore, the execution of each processing by a specific computer includes executing part or all of the said processing on another computer and obtaining the result.

[0379] In other words, the information processing method and program related to this technology can also be applied to cloud computing configurations where a single function is distributed and jointly processed by multiple devices through a network.

[0380] It is also possible to combine at least two of the feature parts related to the technology described above. That is, the various feature parts described in each embodiment may be combined arbitrarily without distinction between each embodiment. Furthermore, the various effects described above are merely examples and are not limited thereto, and other effects may also be exhibited.

[0381] In the present disclosure, "identical," "equivalent," "orthogonal," etc., are concepts that include "substantially identical," "substantially equivalent," "substantially orthogonal," etc. For example, states that fall within a predetermined range (e.g., a range of ±10%) based on criteria such as "completely identical," "completely equivalent," "completely orthogonal," etc. are also included.

[0382] In addition, the present technology may also adopt the following configuration.

[0383] (1) A unit for acquiring surrounding environment information regarding the surrounding environment of a vehicle, and

[0384] A projection control unit that controls the display of a projection pattern projected onto the road surface surrounding the vehicle from a projection unit mounted on the vehicle, based on the above surrounding environment information.

[0385] An information processing device equipped with

[0386] (2) It is an information processing device described in (1),

[0387] The above projection pattern includes a first pattern on a line projected forward in the direction of travel of the vehicle and a second pattern on a line projected backward in the direction of travel.

[0388] Information processing device.

[0389] (3) It is an information processing device described in (2),

[0390] The first pattern above includes a pair of lines projected onto the left and right sides of the vehicle, and

[0391] The projection control unit controls the display of a target line projected onto the target lane side, which is the target of the vehicle's lane change among the first patterns, based on the surrounding environment information.

[0392] Information processing device.

[0393] (4) It is an information processing device described in (3),

[0394] The above surrounding environment information includes traffic situation information indicating the surrounding traffic conditions of the vehicle including the target lane, and

[0395] The projection control unit estimates a change risk associated with a lane change of the vehicle based on the traffic condition information, and controls the color or flashing of the target line based on the change risk.

[0396] Information processing device.

[0397] (5) It is an information processing device described in (4),

[0398] The above traffic condition information includes information indicating the empty space of the target lane, information on the placement of other vehicles traveling around the vehicle, and information regarding the relative speed of the other vehicles with respect to the vehicle.

[0399] Information processing device.

[0400] (6) It is an information processing device described in (4) or (5), and

[0401] The projection control unit determines the change risk by dividing it into multiple levels and sets the target line to a different color for each determined level.

[0402] Information processing device.

[0403] (7) An information processing device described in any one of (3) to (6),

[0404] The projection control unit above divides the target line into a part where the vehicle can travel and a part where the vehicle cannot travel.

[0405] Information processing device.

[0406] (8) An information processing device described in any one of (3) to (7), and also,

[0407] The vehicle is equipped with a gaze detection unit that detects the gaze direction of a driver operating the vehicle, and

[0408] The projection control unit determines whether to execute display control of the target line based on the change in the line of sight.

[0409] Information processing device.

[0410] (9) An information processing device described in any one of (2) to (8),

[0411] The above surrounding environment information includes subsequent vehicle information regarding the subsequent vehicle of the above vehicle, and

[0412] The projection control unit estimates the collision risk with the subsequent vehicle based on the subsequent vehicle information and discontinuously changes the second pattern according to the collision risk.

[0413] Information processing device.

[0414] (10) It is an information processing device described in (9),

[0415] The above subsequent vehicle information includes information on at least one of the relative distance and relative speed between the vehicle and the subsequent vehicle.

[0416] Information processing device.

[0417] (11) It is an information processing device described in (9) or (10), and

[0418] The projection control unit increases the width of the second pattern when the collision risk is greater than or equal to a first threshold and less than a second threshold which is greater than the first threshold, and blinks the second pattern when the collision risk is greater than or equal to the second threshold.

[0419] Information processing device.

[0420] (12) An information processing device described in any one of (2) to (11), and

[0421] The above surrounding environment information includes position relationship information indicating the positional relationship between the surrounding objects of the vehicle and the vehicle, and

[0422] The projection control unit generates a pair of lines representing the vehicle width as the first pattern and individually controls the display of the pair of lines based on the positional relationship information.

[0423] Information processing device.

[0424] (13) It is an information processing device described in (12),

[0425] The above object is a boundary line indicating the range in which the vehicle must travel, and

[0426] The projection control unit above controls the color or flickering of the line based on the positional relationship between the boundary line and the line.

[0427] Information processing device.

[0428] (14) It is an information processing device described in (13),

[0429] The above acquisition unit has a first detection unit that detects the distance between the boundary line and the line from an image in which the boundary line and the line are captured simultaneously, as the above positional relationship information.

[0430] The projection control unit above changes the color of the line or makes the line blink when the distance between the boundary line and the line is smaller than a predetermined threshold.

[0431] Information processing device.

[0432] (15) An information processing device described in any one of (12) to (14), and

[0433] The above acquisition unit has a second detection unit that detects the location of an obstacle on the line by detecting distortion of the line from an image of the line captured as the above positional relationship information.

[0434] The projection control unit above divides the line into an intersection with the obstacle and a different part based on the location of the obstacle.

[0435] Information processing device.

[0436] (16) An information processing device described in any one of (2) to (15), and also,

[0437] It is equipped with a predicted trajectory calculation unit that calculates a predicted trajectory predicted to pass through the above vehicle, and

[0438] The projection control unit generates the first pattern representing the predicted trajectory.

[0439] Information processing device.

[0440] (17) An information processing device described in any one of (2) to (16), and also,

[0441] It is equipped with a passage trajectory calculation unit that calculates the passage trajectory passed by the above vehicle, and

[0442] The projection control unit generates the second pattern representing the passage trajectory.

[0443] Information processing device.

[0444] (18) Obtain surrounding environment information regarding the surrounding environment of the vehicle, and

[0445] Based on the above surrounding environment information, controlling the display of a projection pattern projected onto the road surface surrounding the vehicle from a projection unit mounted on the vehicle.

[0446] A method of information processing that a computer system executes.

[0447] (19) A step for obtaining surrounding environment information regarding the surrounding environment of a vehicle, and

[0448] A step for controlling the display of a projection pattern projected onto the road surface surrounding the vehicle from a projection unit mounted on the vehicle, based on the surrounding environment information.

[0449] A program that runs on a computer system.

[0450] (20) A projection unit mounted on a vehicle and projecting a projection pattern onto the road surface surrounding the vehicle, and

[0451] An acquisition unit for acquiring surrounding environment information regarding the surrounding environment of the above-mentioned vehicle, and

[0452] A projection control unit that controls the display of the projection pattern projected from the projection unit based on the above surrounding environment information.

[0453] A projection device equipped with Explanation of the symbols

[0454] 1, 1a, 1b: Vehicle 2: Projection Pattern 3: Line Pattern 3a: Forward line 3c: Rear line 3b: Center line 4: Direction of travel 5: Predicted trajectory 6: Passage trajectory 8L: Left line 8R: Right Line 8: Vehicle width line 10, 10a∼10h, 210, 310: Projection section 12: Driver surveillance camera 13, 213, 313: Ambient environment sensor unit 20, 220, 320: Controller 22, 222, 322: Trajectory Calculation Unit 23, 223, 323: Projection image determining unit 26, 226, 326: Surrounding environment recognition unit 41: Target Lane 51: Boundary line 53: Target line 55, 55a, 55b: Obstructions 100, 200, 300: Projection device

Claims

Claim 1 An information processing device comprising: an acquisition unit for acquiring surrounding environment information regarding the surrounding environment of a vehicle; a projection control unit for controlling the display of a projection pattern projected onto a road surface surrounding the vehicle from a projection unit mounted on the vehicle based on the surrounding environment information; wherein the projection pattern includes a first pattern in the shape of a line projected forward in the direction of travel of the vehicle; wherein the surrounding environment information includes positional relationship information indicating the positional relationship between an object surrounding the vehicle and the vehicle; wherein the projection control unit generates a pair of vehicle width lines representing the vehicle width as the first pattern and individually controls the display of the pair of vehicle width lines based on the positional relationship information; wherein the acquisition unit includes an obstacle detection unit that detects the location of an obstacle on the vehicle width line by detecting distortion of the vehicle width line from an image captured of the vehicle width line as the positional relationship information; and wherein the projection control unit changes the display content of a part that intersects the obstacle and a part that is different from the vehicle width line based on the location of the obstacle. Claim 2 An information processing device according to claim 1, wherein the projection pattern includes a second pattern of line shape projected behind the direction of travel. Claim 3 In claim 1, the first pattern includes a pair of lines projected onto the left and right sides of the vehicle, and the projection control unit generates a predetermined pattern as the pair of lines when the lane change of the vehicle is not performed, and when the lane change is performed, controls the display of the target line projected onto the target lane side of the pair of lines that is the target of the lane change to have a display content different from the predetermined pattern based on the surrounding environment information. Claim 4 In paragraph 3, the surrounding environment information includes traffic situation information indicating the surrounding traffic conditions of the vehicle including the target lane, and the projection control unit estimates a change risk associated with a lane change of the vehicle based on the traffic situation information, and sets the color of the target line to a color set corresponding to the value of the change risk, or flashes the target line when the value of the change risk exceeds a predetermined value. Claim 5 In paragraph 4, the traffic condition information comprises information indicating the empty space of the target lane, information on the placement of other vehicles driving around the vehicle, and information regarding the relative speed of the other vehicles with respect to the vehicle. Claim 6 In paragraph 4, the projection control unit determines the change risk by dividing it into multiple levels and sets the target line to a different color for each determined level. Claim 7 In paragraph 3, the surrounding environment information includes obstacle information indicating the location of obstacles around the vehicle, and the projection control unit detects the range in which the vehicle can travel based on the obstacle information, and the information processing device distinguishes the target line by color into a part in which the vehicle can travel and a part in which the vehicle cannot travel based on the detection result. Claim 8 In paragraph 3, the information processing device further comprises a gaze detection unit for detecting the gaze direction of a driver operating the vehicle, and the projection control unit detects a preliminary operation for changing the lane based on a change in the gaze direction, and executes display control of the target line when the preliminary operation for changing the lane is detected. Claim 9 In paragraph 2, the surrounding environment information includes subsequent vehicle information regarding a subsequent vehicle of the vehicle, and the projection control unit estimates the collision risk with the subsequent vehicle based on the subsequent vehicle information, and an information processing device that discontinuously changes the second pattern according to the value of the collision risk so that the second pattern becomes one of a plurality of different patterns set corresponding to the value of the collision risk. Claim 10 In claim 9, the subsequent vehicle information comprises information of at least one of the relative distance and relative speed of the vehicle and the subsequent vehicle. Claim 11 In claim 9, the projection control unit is an information processing device that increases the width of the second pattern when the collision risk is greater than or equal to a first threshold and less than a second threshold which is greater than the first threshold, and blinks the second pattern when the collision risk is greater than or equal to the second threshold. Claim 12 In claim 1, the object is a boundary line indicating the range in which the vehicle must travel, and the projection control unit is an information processing device that controls the display of the vehicle width line according to the distance between the boundary line and the vehicle width line so that it becomes one of a plurality of different patterns set corresponding to the distance between the boundary line and the vehicle width line. Claim 13 In claim 12, the acquisition unit has a distance detection unit that detects the distance between the boundary line and the vehicle width line from an image in which the boundary line and the vehicle width line are simultaneously captured as positional relationship information, and the projection control unit changes the color of the vehicle width line or makes the vehicle width line blink when the distance between the boundary line and the vehicle width line is smaller than a predetermined threshold. Claim 14 In claim 1, the projection control unit is an information processing device that colors the line into a section intersecting the obstacle and a different section based on the position of the obstacle. Claim 15 In claim 1, the information processing device further comprises a prediction trajectory calculation unit that calculates a prediction trajectory predicted to be passed by the vehicle, and the projection control unit generates the first pattern representing the prediction trajectory. Claim 16 In paragraph 2, the information processing device further comprises a passage trajectory calculation unit that calculates a passage trajectory passed by the vehicle, and the projection control unit generates the second pattern representing the passage trajectory. Claim 17 An information processing method in which a computer system executes steps for acquiring surrounding environment information regarding the surrounding environment of a vehicle and, based on the surrounding environment information, controlling the display of a projection pattern projected onto a road surface surrounding the vehicle from a projection unit mounted on the vehicle, wherein the projection pattern includes a first pattern in the shape of a line projected forward in the direction of travel of the vehicle, and the surrounding environment information includes positional relationship information indicating the positional relationship between an object surrounding the vehicle and the vehicle, and the step of controlling the display of the projection pattern generates a pair of vehicle width lines representing the vehicle width as the first pattern and individually controls the display of the pair of vehicle width lines based on the positional relationship information, the step of acquiring the surrounding environment information detects the location of an obstacle on the vehicle width line by detecting distortion of the vehicle width line from an image captured of the vehicle width line as the positional relationship information, and the step of controlling the display of the projection pattern changes the display content of a part that intersects the obstacle and a part different from the vehicle width line based on the location of the obstacle. Claim 18 A program stored on a computer-readable recording medium, comprising executing on a computer system a step of acquiring surrounding environment information regarding the surrounding environment of a vehicle, and a step of controlling the display of a projection pattern projected onto a road surface surrounding the vehicle from a projection unit mounted on the vehicle based on the surrounding environment information, wherein the projection pattern includes a first pattern in the shape of a line projected forward in the direction of travel of the vehicle, and the surrounding environment information includes positional relationship information indicating the positional relationship between objects surrounding the vehicle and the vehicle, and the step of controlling the display of the projection pattern generates a pair of vehicle width lines representing the vehicle width as the first pattern, and individually controls the display of the pair of vehicle width lines based on the positional relationship information, wherein the step of acquiring the surrounding environment information detects the location of an obstacle on the vehicle width line by detecting distortion of the vehicle width line from an image captured of the vehicle width line as the positional relationship information, and the step of controlling the display of the projection pattern determines the display content of the part of the vehicle width line intersecting the obstacle and the part other than the obstacle based on the location of the obstacle. A program stored on a computer-readable recording medium that changes. Claim 19 A projection device mounted on a vehicle and comprising: a projection unit that projects a projection pattern onto a road surface surrounding the vehicle; an acquisition unit that acquires surrounding environment information regarding the surrounding environment of the vehicle; and a projection control unit that controls the display of the projection pattern projected from the projection unit based on the surrounding environment information. The projection pattern includes a first pattern in the shape of a line projected in front of the vehicle's direction of travel, and the surrounding environment information includes positional relationship information indicating the positional relationship between an object surrounding the vehicle and the vehicle. The projection control unit generates a pair of vehicle width lines representing the vehicle width as the first pattern and individually controls the display of the pair of vehicle width lines based on the positional relationship information. The acquisition unit includes an obstacle detection unit that detects the location of an obstacle on the vehicle width line by detecting distortion of the vehicle width line from an image captured of the vehicle width line as the positional relationship information. The projection control unit changes the display content of the part intersecting the obstacle and the part different from the part on the vehicle width line based on the location of the obstacle. Claim 20 delete

Images