Information processing apparatus

Abstract

To support safe driving.SOLUTION: Displaying, on a display device mounted on a moving body, first information for supporting driving operation of the moving body based on driving operation information including at least one of information indicating a steering wheel operation, information indicating a brake operation, and information indicating an accelerator operation detected in the moving body and environment around the moving body, and second information indicating a driving state of a driver of the moving body, the first information corresponding to the second information; And a control unit configured to perform control for driving operation support.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to driving assistance. [Background technology]

[0002] A driving assistance device has been disclosed that outputs a message on a display urging the user to reduce or increase the accelerator operation based on the amount of access operation that brings the operating point of a hybrid vehicle's engine within an acceptable range in which a decrease in efficiency is allowed depending on the target vehicle speed (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-138152 Summary of the Invention [Problem to be solved by the invention]

[0004] An object of one aspect of the present disclosure is to provide an information processing device that supports safe driving. [Means for solving the problem]

[0005] One aspect of the present disclosure is displaying, on a display device mounted on the moving body, first information for assisting the driving operation of the moving body, based on driving operation information detected in the moving body, the first information including at least one of information indicating a steering operation, information indicating a braking operation, and information indicating an accelerator operation, and the environment around the moving body, and second information indicating a driving state of a driver of the moving body, superimposed on real space; performing a control to assist driving operation on at least one of a steering wheel, a brake, or an accelerator of the moving object, the control corresponding to the first information; A control unit that executes It is an information processing device. [Effects of the Invention]

[0006] According to one aspect of the present disclosure, safe driving can be supported. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram showing an example of the system configuration of a driving operation assistance system according to the first embodiment, and the configurations of an in-vehicle device and a server. [Figure 2] FIG. 2 is an example of a flowchart of the driving operation assistance process of the server according to the first embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] One aspect of the present disclosure is an information processing device that executes the following operations: displaying, on a display device mounted on the moving body, first information for assisting the driving operation of the moving body and second information indicating the driving state of the driver of the moving body, superimposed on a real space, the first information being based on driving operation information detected in the moving body and including at least one of information indicating a steering operation, information indicating a braking operation, and information indicating an accelerator operation, and an environment around the moving body; and controlling at least one of the steering, braking, and accelerator of the moving body corresponding to the first information to provide driving operation assistance. It is equipped with a control unit.

[0009] The information processing device is, for example, a data communication module mounted on a mobile body, a navigation device, a drive recorder, etc. The device is not limited to these, and may be, for example, a server located in a different location from the mobile body. The mobile body is, for example, a vehicle such as an automobile, a motorcycle, a bicycle, or a train. The mobile body is not limited to a vehicle that runs on the ground, and may include, for example, a mobile body that moves on the sea, such as a ship, and a mobile body that moves in the air, such as an aircraft or a drone. Alternatively, the information processing device may be a computer mounted on a mobile body other than a vehicle, or a mobile terminal such as a smartphone, a tablet terminal, a PC (Personal Computer), or a game terminal. Good too.

[0010] The control unit is, for example, a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a DSP (Digital Signal Processor). However, the control unit is not limited to a processor, and may be a circuit such as an FPGA (Field Programmable Gate Array), a semiconductor integrated circuit (IC: Integrated Circuit), or a CPLD (Complex Programmable Logic Device).

[0011] The display device provided in the vehicle is, for example, an organic EL display attached to the windshield, a display of a car navigation system, etc. The image data indicating the warning includes, for example, a message, an illustration, a mark, etc. indicating the warning. The image data indicating the warning may also be moving image data. Displaying information superimposed on a real space image is what is known as AR (Augmented Reality) display.

[0012] The first information for assisting the driving operation of the mobile body is, for example, a message encouraging improvement in driving operation, a message giving advice on driving operation, etc. Furthermore, the first information for assisting the driving operation of the mobile body is not limited to a message, and may be an illustration, a moving image, etc.

[0013] The second information indicating the driving state of the driver of the mobile body is, for example, a message indicating that the brake or accelerator pedal is pressed hard, appropriate, or light, a message indicating that the timing of the brake or accelerator pedal press is early, appropriate, or late, a message indicating that the amount of steering wheel rotation is large, appropriate, or small, etc. However, the second information indicating the driving state of the driver of the mobile body is not limited to these. Furthermore, the second information indicating the driving state of the driver of the mobile body is not limited to a message, and may be an illustration, a moving image, etc.

[0014] Control that assists in driving operations of a moving body involves, for example, controlling the brake, accelerator, or steering wheel, or issuing instructions to the brake, accelerator, or steering control device so that the amount of brake depression, accelerator depression, or change in steering angle corresponding to the amount of steering wheel rotation becomes a target value corresponding to the surrounding environment.

[0015] According to one aspect of the present disclosure, it is possible to allow the driver of the vehicle to intuitively recognize the driving state and to provide driving operation assistance, thereby providing assistance for safe driving.

[0016] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. The configurations of the following embodiments are examples, and the present disclosure is not limited to the configurations of the embodiments.

[0017] First Embodiment FIG. 1 shows the system configuration of a driving operation assistance system 100 according to the first embodiment and an in-vehicle device. 1 is a diagram showing an example of the configuration of a server 1 and a server 2. The driving operation assistance system 100 includes a server 2 and a vehicle 10. In the first embodiment, the vehicle 10 is a vehicle equipped with a communication function. The vehicle 10 periodically transmits detection values ​​of various sensors mounted on the vehicle 10 and driving operation information to the server 2 at predetermined intervals. The detection values ​​of the sensors include camera images, speed, and traveling direction. The driving operation information includes, for example, the amount of brake depression, amount of accelerator depression, and amount of change in steering angle of the vehicle 10.

[0018] The server 2 detects the driving environment around the vehicle 10 based on the sensor detection values ​​received from the vehicle 10. The driving environment around the vehicle 10 includes road signs, road markings, pedestrians, other vehicles, buildings, etc. that exist around the vehicle 10. The server 2 determines a scene based on the driving environment of the vehicle 10 and evaluates the driving operation for that scene based on the driving operation information. In the first embodiment, the scene is a traffic-related scene. The determined scenes include, for example, a scene of stopping at a stop line, a scene of going straight, turning right, or turning left at an intersection, a scene of driving on a highway, etc. However, the scenes determined in the first embodiment are not limited to these.

[0019] Based on the evaluation results, the server 2 acquires first information for assisting driving operation and second information indicating the driving state for making the driver aware of the driving state. The server 2 also generates driving operation assistance information for the vehicle 10 according to the driving operation assistance content indicated by the first information. The driving operation assistance information includes, for example, the amount of operation of the brake, accelerator, steering wheel, etc. up to a target value. The server 2 transmits to the vehicle 10 the first information, the second information, an instruction to superimpose the first information and the second information on real space, and the driving operation assistance information. Hereinafter, superimposing and displaying image data on real space will be referred to as AR display.

[0020] The vehicle 10 displays the first information and the second information received from the server 2 on a display in AR. For example, if the windshield of the vehicle 10 serves as a display device, the first information and the second information are displayed on the windshield so as to be superimposed on the real space in front of the vehicle 10. The vehicle 10 also operates the brake, accelerator, or steering wheel in accordance with the driving operation assistance information received from the server 2. The first information and the second information are, for example, text, image data, or video data.

[0021] For example, in a scene where the vehicle 10 is to stop at a stop line, if it is evaluated that it is difficult to stop at the stop line at the current speed of the vehicle 10, a message such as "Please press the brakes a little harder" is acquired as the first information, and a message such as "Your braking is too light" is acquired as the second information. In addition, the amount of brake application required for the vehicle 10 to stop at the stop line is acquired as driving operation assistance information.

[0022] Next, the configurations of the in-vehicle device 1 and the server 2 will be described. The server 2 includes, as hardware components, a processor 201, a memory 202, an auxiliary storage device 203, and a communication unit 204. The memory 202 and the auxiliary storage device 203 are computer-readable recording media. The auxiliary storage device 203 is, for example, an erasable programmable read only memory (EPROM), a hard disk drive, or a solid state drive (SSD). ) The programs stored in the auxiliary storage device 203 include, for example, an operating system (OS) and a driving operation assistance control program. The memory 202 includes, for example, semiconductor memory such as a ROM (Read Only Memory) and a RAM (Random Access Memory).

[0023] The processor 201 is, for example, a CPU, a GPU, or a DSP (Digital Signal Processor). The number of processors 201 is not limited to one, and multiple processors may be provided. The controller 201 is an example of a “controller.” The communication unit 204 is connected to a public line network such as the Internet via a wired or wireless connection.

[0024] The processor 201 executes a control program for driving operation assistance stored in the auxiliary storage device 203, thereby performing processing as the control unit 21, which is a functional component. The control unit 21 acquires sensor detection values ​​and driving operation information from the vehicle 10 at predetermined intervals, evaluates the driving operation according to the scene, and acquires first information, second information, and driving operation assistance information based on the evaluation results, and transmits them to the vehicle 10. Details of the processing by the control unit 21 will be described later.

[0025] Next, the vehicle 10 includes an in-vehicle device 1, a display 110, a camera 120, an obstacle detection sensor 130, an accelerator control unit 140, a brake control unit 150, and a steering control unit 160. Note that in Fig. 1, only elements according to the first embodiment are extracted from the hardware configuration of the vehicle 10 and are shown, and the hardware configuration of the vehicle 10 is not limited to that shown in Fig. 1.

[0026] The camera 120 and the obstacle detection sensor 130 are examples of sensors mounted on the vehicle 10 and used to detect the driving environment. The camera 120 is installed, for example, above the windshield, facing outward from the vehicle 10. However, without being limited to this, a plurality of cameras 120 may be provided and installed on the outside of the vehicle 10 so that each camera faces in a different direction. The obstacle detection sensor 130 is, for example, a sensor that detects obstacles, such as a millimeter-wave radar or a LiDAR. The sensors provided on the vehicle 10 are not limited to the camera 120 and the obstacle detection sensor 130, but may also include sensors such as a speedometer and a direction sensor.

[0027] The accelerator control unit 140, the brake control unit 150, and the steering control unit 160 are, for example, ECUs (Electronic Control Units) that control the accelerator, the brake, and the steering, respectively.

[0028] The display 110 is, for example, an organic EL display installed on the windshield of the vehicle 10, with the entire windshield serving as a display screen. However, the display 110 is not limited to being installed on a window, and may be, for example, a stand-alone display. Furthermore, the display 110 may be, for example, a display used in a car navigation system.

[0029] The in-vehicle device 1 is, for example, a data communication module, a car navigation system, The in-vehicle device 1 is an information system, a drive recorder, etc. The in-vehicle device 1 includes, as a hardware configuration, a processor, a memory, an auxiliary storage device, a wireless communication unit, etc.

[0030] The in-vehicle device 1 periodically acquires detection values ​​from sensors such as the camera 120 and the obstacle detection sensor 130. The in-vehicle device 1 also periodically acquires the accelerator depression amount from the accelerator control unit 140, the brake depression amount from the brake control unit 150, and the steering angle and the amount of change in the steering angle from the steering control unit 160, thereby acquiring driving operation information. The in-vehicle device 1 transmits the sensor detection values ​​and driving operation information to the server 2 periodically.

[0031] Furthermore, when the in-vehicle device 1 receives from the server 2 first information for assisting driving operation and second information indicating the driving state for making the driver aware of the driving state, the in-vehicle device 1 displays the first information and the second information in AR on the display 110. Furthermore, when the in-vehicle device 1 receives the driving operation assistance information from the server 2, the in-vehicle device 1 outputs the driving operation assistance information to the corresponding control unit among the accelerator control unit 140, the brake control unit 150, and the steering control unit 160, and adjusts the accelerator, brake, and steering speeds so as to reach the target values ​​included in the driving operation assistance information. Control the steering.

[0032] Fig. 2 is an example of a flowchart of the driving operation assistance process of the server 2 according to the first embodiment. The process shown in Fig. 2 is repeatedly executed at a predetermined cycle. The process shown in Fig. 2 is executed mainly by the processor 201 of the server 2, but for convenience, the process will be described mainly with respect to the functional components.

[0033] In OP101, the control unit 21 performs scene discrimination. The scene discrimination may be performed, for example, by image recognition processing of images captured by the camera 120 in the latest or most recent predetermined period, or by inputting the images captured by the camera 120 into a machine learning model trained for scene discrimination. In addition to the images captured by the camera 120, the scene discrimination may also use, as one of the sensor detection values, the type, size, and distance from the vehicle 10 of an obstacle detected by the obstacle detection sensor 130. Furthermore, in addition to the sensor detection values, information such as the time of day, weather, and sunlight may also be used in the scene discrimination.

[0034] In OP102, the control unit 21 determines whether the scene determined in OP101 has changed to a different type of scene. If there has been a change in the scene (OP102: YES), the process proceeds to OP103. If there has not been a change in the scene (OP102: NO), the process shown in FIG. 2 ends.

[0035] In OP103, the control unit 21 evaluates the driving operation, for example, based on the driving operation information. The evaluation of the driving operation may be performed based on the insufficiency or excess of the operation amount relative to an evaluation criterion prepared in advance for each scene, or may be performed using a machine learning model that has already learned the evaluation of the driving operation prepared for each scene. For example, in a scene where the driver has to stop at a stop line, the amount of brake application necessary to temporarily stop the vehicle is determined as a target value (or evaluation criterion) based on the current speed, and an evaluation is made as to whether the current amount of brake application is too large (strong), appropriate, or small (weak) relative to the target value.

[0036] In OP104, the control unit 21 acquires first information for assisting the driving operation based on the evaluation result in OP103. The first information may be prepared in advance for each scene and evaluation result, for example. For example, if the evaluation result of the driving operation in a scene of a temporary stop at a stop line is "the brakes are not applied enough," the first information may be a message such as "please apply the brakes a little more."

[0037] In OP105, the control unit 21 acquires driving operation assistance information corresponding to the assistance content for the driving operation indicated by the first information acquired in OP104, based on the evaluation result in OP103. For example, if the evaluation result of the driving operation in a scene where the driver has to stop at a stop line is "the brake application amount is weak," the driving operation assistance information includes the brake application amount up to the target value.

[0038] In OP106, the control unit 21 acquires second information indicating the driving state for making the driver aware of the driving state based on the evaluation result in OP103. The second information may be prepared in advance for each scene and evaluation result, for example. For example, if the evaluation result of the driving operation in a scene of a temporary stop at a stop line is "the brakes are not applied enough," the second information is a message such as "the brakes are not applied enough."

[0039] In OP107, the control unit 21 transmits the first information, the second information, an instruction to display the first information and the second information in AR, and the driving operation assistance information to the vehicle 10. After that, the process shown in Fig. 2 ends. The vehicle 10 displays the first information and the second information received from the server 2 in AR on the display 110, and controls the vehicle 10 based on the driving operation assistance information.

[0040] <Effects of the First Embodiment> According to the first embodiment, first information for assisting driving operation and second information indicating the driving state are displayed in AR on the display of the vehicle 10, so that the driver of the vehicle 10 can be assisted in driving operation and be aware of the driving state. This can reduce the driver's fear of driving and support safe driving. Furthermore, in the vehicle 10, driving operation assistance based on the first information for assisting driving operation is provided, so safe driving can be supported.

[0041] <Other embodiments> The above-described embodiment is merely an example, and the present disclosure can be modified and implemented as appropriate within the scope that does not deviate from the gist of the disclosure.

[0042] In the first embodiment, the server 2 acquires the first information, the second information, and the driving operation assistance information. Alternatively, the in-vehicle device 1 may acquire the first information, the second information, and the driving operation assistance information in the same manner as the server 2 in the first embodiment.

[0043] The processes and means described in this disclosure can be freely combined and implemented as long as no technical contradiction occurs.

[0044] Furthermore, a process described as being performed by one device may be shared and executed by multiple devices. Alternatively, a process described as being performed by different devices may be executed by a single device. In a computer system, the hardware configuration (server configuration) by which each function is realized can be flexibly changed.

[0045] The present disclosure can also be realized by providing a computer program implementing the functions described in the above embodiments to a computer, and having one or more processors in the computer read and execute the program. Such a computer program may be provided to the computer via a non-transitory computer-readable storage medium connectable to the computer's system bus or via a network. Non-transitory computer-readable storage media include, for example, any type of disk, such as a magnetic disk (e.g., a floppy disk, a hard disk drive (HDD), etc.), an optical disk (e.g., a CD-ROM, a DVD disk, a Blu-ray disk), a read-only memory (ROM), a random-access memory (RAM), an EPROM, an EEPROM, a magnetic card, a flash memory, an optical card, or any type of medium suitable for storing electronic instructions. [Explanation of symbols]

[0046] 1...In-vehicle device 2. Server 10. Vehicle 21 Control unit 100··Driving operation assistance system 201 Processor 202 Memory 203...Auxiliary storage device 204··Communications Department

Claims

[Claim 1] displaying, on a display device mounted on the moving body, first information for assisting the driving operation of the moving body, based on driving operation information detected in the moving body, the first information including at least one of information indicating a steering operation, information indicating a braking operation, and information indicating an accelerator operation, and the environment around the moving body, and second information indicating a driving state of a driver of the moving body, superimposed on real space; performing a control to assist driving operation on at least one of a steering wheel, a brake, or an accelerator of the moving object, the control corresponding to the first information; A control unit that executes Information processing device.

Images