Information processing apparatus and method executed by computer

Abstract

The invention relates to an information processing apparatus and a method executed by a computer. And safe driving is assisted. An information processing device is provided with a control unit that executes the following operations: detecting, from one or more sensors provided in a vehicle, a traveling environment around the vehicle; and causing a display device provided in the vehicle to display image data indicating a warning by superimposing the image data on a real space when the occurrence of an event relating to the traveling of the vehicle and another vehicle in the vicinity of the vehicle is sensed on the basis of the traveling environment in the vicinity of the vehicle. The control unit detects, as one of the events, a feature on the traveling road or in the vicinity of the traveling road on the basis of the traveling environment around the vehicle. Image data representing a warning relating to a feature is transmitted to a subsequent vehicle of the vehicle together with an instruction to display the image data by superimposing the image data on a real space.

Description

Technical Field

[0001] This disclosure relates to driving assistance, and more specifically, to methods performed by information processing devices and computers. Background Technology

[0002] An advanced driver assistance system is disclosed that determines features and hazardous areas of features within the vehicle's environment as sensed by sensors and displays them on a display unit (e.g., Patent Document 1).

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2022-159094 Summary of the Invention

[0006] The problem that the invention aims to solve

[0007] One of the objectives of this disclosure is to provide an information processing device and a method executed by a computer to assist in safe driving.

[0008] Solution for solving the problem

[0009] One aspect of this disclosure is an information processing apparatus comprising a control unit that performs the following actions: detecting the driving environment surrounding the vehicle from one or more sensors of the vehicle; and, in the event that an event related to the driving of the vehicle and other vehicles in the vicinity of the vehicle is sensed based on the driving environment surrounding the vehicle, causing a display device of the vehicle to display image data indicating a warning overlaid on real space, wherein, in the event that a feature is detected on or around the driving road based on the driving environment surrounding the vehicle, the control unit sends image data indicating a warning related to the feature, along with an instruction to display the image data overlaid on real space, to subsequent vehicles of the vehicle.

[0010] Another aspect of this disclosure is a method performed by a computer that performs the following actions: detecting the driving environment surrounding the vehicle from one or more sensors of the vehicle; if, based on the driving environment surrounding the vehicle, an event related to the driving of the vehicle and other vehicles in the vicinity of the vehicle is sensed, causing a display device of the vehicle to display image data indicating a warning overlaid in real space; and if, as one of the events, a feature is detected on or around the driving road based on the driving environment surrounding the vehicle, image data indicating a warning related to the feature, along with an instruction to display the image data overlaid in real space, is sent to subsequent vehicles of the vehicle.

[0011] Invention Effects

[0012] According to one of the methods disclosed herein, safe driving can be assisted. Attached Figure Description

[0013] Figure 1 This is a diagram illustrating an example of vehicle processing according to the first embodiment.

[0014] Figure 2 This is a diagram illustrating an example of the hardware configuration of a vehicle.

[0015] Figure 3 This is a diagram illustrating an example of the functional configuration of an onboard device.

[0016] Figure 4 This is a diagram representing an example of a warning event and a warning message.

[0017] Figure 5 This is a flowchart of the vehicle's safe driving assistance process.

[0018] Explanation of reference numerals in the attached figures

[0019] 1: Onboard device; 10: Vehicle; 11: Control unit; 101: Processor; 102: Memory; 103: Auxiliary storage device; 104: Wireless communication unit; 105: Workshop communication unit. Detailed Implementation

[0020] One aspect of this disclosure is an information processing apparatus. This apparatus includes a control unit that performs the following actions: detecting the driving environment surrounding the vehicle from one or more sensors equipped with the vehicle; and, upon sensing the occurrence of an event related to the driving of the vehicle and other vehicles in the vicinity based on the driving environment surrounding the vehicle, causing a display device equipped with the vehicle to overlay warning image data onto real space for display. When an event occurs in which a feature is detected on or around the driving road based on the driving environment surrounding the vehicle, the control unit sends warning image data related to the feature, along with an instruction to overlay the image data onto real space for display, to following vehicles.

[0021] Information processing devices include, for example, data communication modules, navigation devices, and dashcams mounted on mobile bodies. However, the information processing device is not limited to these; it can also be a computer mounted on a mobile body other than a vehicle, or a mobile terminal such as a smartphone, tablet, PC (Personal Computer), or gaming console. The information processing device is not limited to being mounted on a vehicle; it can also be mounted on a mobile body other than a vehicle. Mobile bodies include, for example, cars, motorcycles, bicycles, and trams. Furthermore, mobile bodies are not limited to vehicles traveling on land; they can also include mobile bodies moving at sea, such as ships, and mobile bodies moving in the air, such as aircraft and drones. Alternatively, the information processing device can also be a server located in a different location than the vehicle.

[0022] The control unit can be a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a DSP (Digital Signal Processor). However, the control unit is not limited to a processor; it can also be a circuit such as an FPGA (Field Programmable Gate Array), an IC (Integrated Circuit), or a CPLD (Complex Programmable Logic Device).

[0023] The vehicle is equipped with one or more sensors, such as cameras, speedometers, millimeter-wave radar, and LiDAR (Light Detection and Ranging). The driving environment surrounding the vehicle, detected by these sensors, includes road signs, road markings, pedestrians, other vehicles, and buildings. Events related to the vehicle and other vehicles in its vicinity include traffic violations, predicted traffic violations, and predicted accidents. Examples of traffic violations include driving against traffic, speeding, and entering prohibited areas. Predicted traffic violations include entering intersections where right or left turns are prohibited and approaching temporary stop lines at speeds exceeding the prescribed limits. Predicted accidents include the presence of features on or around the road. Characteristic objects include pedestrians, moving bicycles and motorcycles, parked vehicles, accident vehicles, notices such as those announcing construction, traffic control workers and other items and personnel at the construction site, tires, corrugated cardboard and other debris from trucks, etc.

[0024] Vehicle-mounted display devices include, for example, organic EL displays attached to the windshield and displays in car navigation systems. Warning image data includes warning messages, illustrations, and markings. Warning image data can also be dynamic image data. Overlaying it onto real space for display is known as AR (Augmented Reality) display.

[0025] According to one of the solutions disclosed herein, when an event related to the driving of the vehicle and other vehicles in the vicinity is sensed, a warning is displayed on the vehicle's display device (AR). This allows the driver to intuitively recognize the warning and provide safe driving assistance.

[0026] Furthermore, in one of the solutions disclosed herein, when a feature is detected on or around the driving road based on the vehicle's surrounding driving environment, the control unit sends image data indicating a warning related to the feature, along with an instruction to overlay the image data onto real space for display, to the vehicle's successor. The image data indicating the warning related to the feature can also be sent via inter-vehicle device communication, for example. Inter-device communication includes, for example, vehicle-to-vehicle wireless LAN (Local Area Network) communication, DSRC (Dedicated Short-Range Communications), 5G sidelink communication, device-to-device (D2D) communication, and ad hoc communication, among other terminal-to-terminal communication methods. This inter-device communication can also be multi-hop. However, inter-device communication is not limited to this. Furthermore, the image data indicating the warning related to the feature is not limited to inter-vehicle device communication; it can also be sent via road-to-road communication or a server. Road-to-road communication includes, for example, Wi-Fi (Wireless Fidelity) and DSRC. In the case of Wi-Fi, the roadside device is, for example, an access point set up in a designated location.

[0027] Therefore, following vehicles can quickly display AR image data indicating a warning related to the feature on the display device, which can help the driver of the following vehicle to notice the feature.

[0028] As another option, this disclosure can define a method for a computer to perform the processing performed by the aforementioned information processing apparatus, a program for causing the computer to perform the method, and a non-transitory and computer-readable recording medium on which the program is recorded.

[0029] Hereinafter, embodiments of the present disclosure will be described based on the accompanying drawings. The configurations of the following embodiments are illustrative, and the present disclosure is not limited to the configurations of the embodiments.

[0030] <First Implementation Method>

[0031] Figure 1This diagram illustrates an example of the processing of the vehicle 10 according to the first embodiment. In the first embodiment, the vehicle 10 is a vehicle equipped with communication capabilities. The vehicle 10 detects the driving environment around it using various sensors. The driving environment around the vehicle 10 includes road signs, road markings, pedestrians, other vehicles, and buildings present around the vehicle 10. When the vehicle 10 senses the occurrence of an event related to the driving of the vehicle 10 and other surrounding vehicles based on the detected driving environment, it causes a display device mounted on the vehicle 10 to display image data indicating a warning overlaid on real space. For example, if the windshield of the vehicle 10 is used as a display device, the image data indicating a warning is displayed on the windshield in a manner that overlays the real space in front of the vehicle 10. Hereinafter, displaying image data overlaid on real space is referred to as AR display.

[0032] Events related to the movement of vehicle 10 and other surrounding vehicles include, for example, events where vehicle 10 violates traffic regulations, events where vehicle 10 is predicted to violate traffic regulations, and events where an accident is predicted. Events where vehicle 10 violates traffic regulations include, for example, driving against traffic, speeding, or entering a prohibited area. Events where vehicle 10 is predicted to violate traffic regulations include, for example, entering an intersection where right or left turns are prohibited, or approaching a temporary stop line at a speed exceeding the prescribed limit. Events where an accident is predicted include, for example, the presence of features on or around the road. Features include, for example, pedestrians, moving bicycles and motorcycles, parked vehicles, accident vehicles, notices such as those for construction, items and personnel at construction sites such as traffic control workers, tires, corrugated cardboard, etc., dropped from trucks, etc. Features are also referred to as hazards. Hereinafter, events related to the movement of vehicle 10 and other surrounding vehicles will be referred to as warning events. In the first embodiment, when a warning event is detected, the warning is displayed in AR on the vehicle 10, thus enabling the driver of the vehicle 10 to visually recognize the warning and assisting in safe driving.

[0033] Furthermore, upon detecting a feature, vehicle 10 sends image data indicating a warning related to the feature and an instruction for an AR display of the image data to the following vehicle 20. This image data and the AR display instruction are transmitted, for example, via inter-vehicle communication. Thus, in the following vehicle 20, the image data indicating a warning related to the feature is rapidly AR-displayed on its onboard display device.

[0034] Figure 2 This is a diagram illustrating an example of the hardware configuration of vehicle 10. Figure 2The hardware components of the processing in the first embodiment are extracted and illustrated. The vehicle 10 includes an on-board unit 1. The on-board unit 1 is, for example, a data communication module, a car navigation system, or a dashcam.

[0035] The vehicle-mounted device 1 comprises a processor 101, a memory 102, an auxiliary storage device 103, a wireless communication unit 104, a vehicle-to-vehicle communication unit 105, and an interface 106 as hardware components. The memory 102 and the auxiliary storage device 103 are computer-readable recording media. The auxiliary storage device 103 is, for example, an EPROM (Erasable Programmable ROM), a hard disk drive, or an SSD (Solid State Drive). The program stored in the auxiliary storage device 103 includes, for example, an operating system (OS) and a communication mode selection control program. The memory 102 includes, for example, semiconductor memories such as ROM (Read Only Memory) and RAM (Random Access Memory).

[0036] Processor 101 may be, for example, a CPU, GPU, or DSP (Digital Signal Processor). There may be multiple processors 101, not just one. Processor 101 is an example of a "control unit".

[0037] Wireless communication unit 104 is, for example, a wireless communication line corresponding to cellular communication methods such as 5G, 4G, or 6G. Workshop communication unit 105 is a wireless communication line corresponding to a specified workshop communication method. The workshop communication methods used by workshop communication unit 105 include, for example, workshop wireless LAN communication, DSRC, 5G sidelink communication, device-to-device communication, and point-to-point mode, which enable communication between terminals. This workshop communication can also be multi-hop. However, the communication methods used by workshop communication unit 105 are not limited to these.

[0038] Interface 106 is an interface for connecting to an in-vehicle network that connects to other devices. The in-vehicle network may be, for example, an Ethernet-based network or a CAN (Controller Area Network).

[0039] Furthermore, the vehicle 10 includes a camera 120, a speedometer 130, an obstacle sensing sensor 140, and a direction sensor 150 as sensors for detecting the driving environment and driving status. These sensors are connected to an in-vehicle network via an interface 106 of the vehicle-mounted device 1. The camera 120 is positioned above the windshield, for example, facing outwards from the vehicle 10. However, it is not limited to this; multiple cameras 120 may be provided, each facing a different direction, positioned on the outside of the vehicle 10. The speedometer 130 is a sensor that senses the speed of the vehicle 10. The obstacle sensing sensor 140 is, for example, a millimeter-wave radar or LiDAR sensor that senses obstacles. The direction sensor 150 is, for example, an acceleration sensor that senses the driving direction of the vehicle 10. Hereinafter, the camera 120, speedometer 130, obstacle sensing sensor 140, and direction sensor 150 will be collectively referred to as the sensor group.

[0040] In addition, vehicle 10 includes a display 110. The display 110 is connected to an in-vehicle network via an interface 106 of the in-vehicle device 1. The display 110 is, for example, an organic EL display installed on the windshield of vehicle 10, displaying the entire surface of the windshield as the screen. However, the display 110 is not limited to being installed on a windshield; it can also be a stand-alone display. Furthermore, the display 110 can also be, for example, a display used in a car navigation system. It should be noted that the hardware configuration of the in-vehicle device 1 is not limited to... Figure 2 The structure shown.

[0041] Figure 3 This diagram illustrates an example of the functional configuration of the vehicle-mounted device 1. The vehicle-mounted device 1 includes a control unit 11 as a functional component. The control unit 11 is a functional component that is implemented, for example, by executing a predetermined program through the processor 101 of the vehicle-mounted device 1.

[0042] The control unit 11 acquires sensor values ​​from the sensor array at predetermined intervals while the vehicle 10 is in motion. For example, it acquires captured images from the camera 120 as sensor values. It acquires the speed of the vehicle 10 from the speedometer 130. It acquires information such as the presence or absence of obstacles, the size of obstacles, the type of obstacles, and the distance of obstacles from the vehicle 10 from the obstacle sensing sensor 140. It acquires the direction of travel of the vehicle 10 from the direction sensor 150.

[0043] The control unit 11 analyzes the sensor readings and determines whether a warning event has occurred based on the analysis results. Details regarding warning events will be described later. If a warning event is determined to have occurred, the control unit 11 generates image data representing the warning based on the type of warning event and displays it in AR on the display 110.

[0044] When the warning event is the presence of a feature on or around the driving road, the control unit 11 generates image data indicating a warning related to the feature and sends the image data and an AR display instruction for the image data to the following vehicle 20 via the inter-vehicle communication unit 105. For example, the image data and the AR display instruction for the image data can also be sent via multicast or broadcast. Alternatively, if prior consultation has been made with the following vehicle 20, the image data and the AR display instruction for the image data can also be sent to the following vehicle 20 via unicast. When the following vehicle 20 receives the image data and the AR display instruction for the image data from the vehicle 10 via inter-vehicle communication, it displays the image data indicating a warning related to the feature on its AR display. It should be noted that the functional configuration of the vehicle-mounted device 1 is not limited to... Figure 3 The structure shown.

[0045] Figure 4 This is a diagram representing an example of a warning event and a warning message. Furthermore, in Figure 4 The diagram also illustrates the sensing conditions for the occurrence of a warning event. Specifically, when the control unit 11 determines that the sensing conditions are met, it senses the occurrence of the corresponding warning event and generates image data containing a warning message corresponding to the sensed warning event.

[0046] For example, for a temporary stop warning event, the sensing condition is the detection of a temporary stop road sign or a stop line along the driving road from the image captured by camera 120. However, the sensing condition for a temporary stop warning event is not limited to this; for example, it could also be the detection of a temporary stop road sign or stop line from the image captured by camera 120, and the vehicle 10's speed being above a predetermined speed that makes it difficult to stop at the stop line. The warning message when a temporary stop warning event occurs may use, for example, "Immediately Temporarily Stop".

[0047] For example, in the event of a warning about a vehicle entering a prohibited road, the vehicle's entry into the prohibited road sign will be detected from the image captured by camera 120 as a sensing condition. The warning message when the warning event occurs may use phrases such as "Entry into a prohibited road has occurred."

[0048] For example, in the event of a warning about a restriction on travel direction, a prohibition sign outside the travel direction is detected from the image captured by camera 120 as a sensing condition. The prohibition sign outside the travel direction is, for example, a sign indicating no left turn or no right turn. The warning message when a warning event of a restriction on travel direction occurs may use phrases such as "Travel direction is restricted."

[0049] For example, for a warning event of driving against traffic, the sensing condition could be that the direction of travel of vehicle 10 is the same as the direction of travel of other vehicles traveling across the center line as detected in the image captured by camera 120. However, the sensing condition for a warning event of driving against traffic is not limited to this. Sensing conditions can be set based on all road conditions, including signs and road markings that can be detected in the image captured by camera 120, the number of lanes, etc. The warning message when a warning event of driving against traffic occurs could use, for example, "Driving against traffic!" Furthermore, since driving against traffic increases the likelihood of vehicle 10 violating traffic rules and posing a danger to surrounding vehicles, the display of the warning message, such as color and font, can be made more prominent compared to other warning messages.

[0050] For example, for a hazard detection warning event, the obstacle detection sensor 140 detects a hazard on and around the driving road as a sensing condition. The warning message when a hazard detection warning event occurs uses, for example, "Hazard detected!" It should be noted that... Figure 4 The warning event, the sensing conditions for the occurrence of the warning event, and the warning message shown are just examples and are not limited to... Figure 4 The example shown.

[0051] Figure 5 This is a flowchart of the safe driving assistance process for vehicle 10. Figure 5 The process shown is repeatedly executed while the vehicle 10 is being driven or in motion. The vehicle 10 is in motion, for example, by detecting that the vehicle 10's driving system is enabled. Figure 5 The processor 101 of the vehicle-mounted device 1 is shown to be the main body executing the processing, but for convenience, the description will focus on the functional components.

[0052] In OP101, the control unit 11 acquires sensed values ​​from the sensor group. In OP102, the control unit 11 analyzes the acquired sensed values. For example, it performs image recognition processing on the images captured by the camera 120 to detect road signs, road markings, and the driving directions of surrounding vehicles.

[0053] In OP103, the control unit 11 determines whether a warning event has occurred based on the analysis results of the sensor's sensed values. If a warning event has occurred (OP103: Yes), the process proceeds to OP104. If no warning event has occurred (OP103: No), Figure 5 The processing shown is now complete.

[0054] In OP104, controller 11 acquires image data containing warning messages corresponding to the sensed warning events. For the image data containing warning messages, for example, image data pre-stored in auxiliary storage device 103 according to each type of warning event can be read, or it can be created using machine learning models, etc.

[0055] In OP105, the control unit 11 determines whether the sensed warning event is a hazard sensing warning event. If the sensed warning event is a hazard sensing warning event (OP105: Yes), the process proceeds to OP106. In OP106, the control unit 11 sends image data containing the warning message corresponding to the hazard sensing and an instruction for AR display of the image data to the following vehicle 20 via inter-vehicle communication.

[0056] If the sensed warning event is not a hazard detection warning event (OP105: No), the process proceeds to OP107. In OP107, the control unit 11 displays the image data AR acquired in OP104 on the display 110. Then, Figure 5 The processing shown is now complete.

[0057] It should be noted that, Figure 5 The processing shown is an example; safe driving assistance processing is not limited to... Figure 5 The processing is shown below. Warning events can also be pre-prioritized, and when multiple warning events are detected, the warning message with the highest priority is displayed. The priority of warning events can be set, for example, in ascending order of events predicting a traffic violation by vehicle 10, events predicting an accident, and events predicting the vehicle's traffic violation. However, this is not a limitation; the priority can also be set at the granularity of the warning event unit.

[0058] <Effects of the First Embodiment>

[0059] In the first embodiment, when a warning event is detected by a sensor, the vehicle 10 displays image data containing a warning message corresponding to the warning event on the display 110AR. This assists in safe driving of the vehicle 10. Furthermore, in the first embodiment, when the vehicle 10 detects a hazard on or around the road, it sends image data indicating a warning related to the hazard and an instruction to display the image data AR to the following vehicle 20. This allows the following vehicle 20 to be notified of the presence of the hazard earlier, thus improving the driver's awareness of the hazard.

[0060] <Other Implementation Methods>

[0061] The above-described implementation is merely an example, and this disclosure may be implemented with appropriate modifications without departing from its spirit.

[0062] In the first embodiment, vehicle 10 senses a warning event, acquires image data containing a warning message, and performs an AR display. Alternatively, vehicle 10 may send the sensor readings to a server, which then senses the occurrence of the warning event, acquires image data containing a warning message, and sends the image data back to vehicle 10 for AR display. Furthermore, if the warning event is a hazard detection, the server may also identify a subsequent vehicle 20 and send image data containing a warning message related to the hazard and an AR display instruction to the subsequent vehicle 20.

[0063] The processes and components described in this disclosure can be freely combined and implemented as long as they do not create technical contradictions.

[0064] Furthermore, a process described as being performed by one device can also be executed by multiple devices. Alternatively, a process described as being performed by different devices can also be executed by a single device. In a computer system, the hardware configuration (server configuration) used to implement each function can be flexibly changed.

[0065] This disclosure can also be implemented by supplying a computer program that implements the functions described in the above embodiments to a computer, which has one or more processors that read and execute the program. Such a computer program can be provided to the computer either via a non-transitory computer-readable storage medium connectable to the computer's system bus or via a network. Non-transitory computer-readable storage media include, for example, any type of disk such as a hard disk (floppy disk, hard disk drive (HDD), etc.), an optical disk (CD-ROM (CompactDisk-ReadOnly Memory), DVD (Digital Video Disc), Blu-ray disc, etc.), read-only memory (ROM), random access memory (RAM), EPROM, EEPROM (Electrically Erasable Programmable Read-Only Memory), magnetic cards, flash memory, optical cards, and any type of medium suitable for storing electronic instructions.

Claims

1. An information processing device comprising a control unit, The control unit performs the following actions: The vehicle detects its surrounding driving environment from one or more sensors; and When an event related to the driving of the vehicle and other vehicles in the vicinity is sensed based on the driving environment surrounding the vehicle, the vehicle's display device displays image data indicating a warning overlaid on the real space. When an event occurs in which a feature is detected on or around the driving road based on the driving environment surrounding the vehicle, the control unit sends image data indicating a warning related to the feature, along with an instruction to overlay the image data onto real space for display, to the vehicle's successor.

2. The information processing apparatus according to claim 1, wherein, The control unit performs the following actions: Images captured by the vehicle's camera device are used to detect at least road signs and markings to define the driving environment surrounding the vehicle. At least the vehicle's speed and direction of travel are detected to determine the vehicle's driving status. If, upon sensing or predicting an event of a vehicle violating traffic regulations as an event from at least one of the detected vehicle's driving direction, the road sign, and the road marking, the display device displays the image data representing the warning overlaid on real space.

3. The information processing apparatus according to claim 2, wherein, The control unit performs the following actions: The driving direction of the other vehicles can be further detected from the captured images. If, upon sensing that the vehicle is traveling in the wrong direction from at least one of the detected driving direction of the vehicle, the driving direction of other vehicles, the center line of the road the vehicle is traveling on, and the road sign, the display device overlays image data indicating that the vehicle is traveling in the wrong direction onto real space as image data representing the warning.

4. A method executed by a computer, the computer performing the following actions: The vehicle detects the surrounding driving environment from one or more sensors it is equipped with; When an event related to the driving of the vehicle and other vehicles in the vicinity is sensed based on the driving environment surrounding the vehicle, the display device of the vehicle displays image data indicating a warning overlaid on the real space; and In the event that a feature is detected on or around the driving road based on the driving environment surrounding the vehicle, as one of the events described above, image data indicating a warning related to the feature is sent to the vehicle's follow-up vehicles along with an instruction to overlay the image data onto real space for display.

Images