Information processing device and method

Abstract

Supports safe driving. [Solution] An information processing device includes a control unit that detects the driving environment around the vehicle using one or more sensors equipped in the vehicle, and when an event related to the driving of the vehicle and other vehicles around the vehicle is detected based on the driving environment around the vehicle, causes a display device equipped in the vehicle to display image data indicating a warning superimposed on real space. When the control unit detects a characteristic object on or near the road as one of the events based on the driving environment around the vehicle, the control unit transmits image data indicating a warning related to the characteristic object to a following vehicle, along with an instruction to display the image data superimposed on real space.

Description

[Technical Field]

[0001] The present disclosure relates to driving assistance. [Background technology]

[0002] An advanced driver assistance system has been disclosed that determines features and danger zones of features in the vehicle's environment detected by sensors and displays them on a display unit (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-159094 Summary of the Invention [Problem to be solved by the invention]

[0004] An object of one aspect of the present disclosure is to provide an information processing device and method that support safe driving. [Means for solving the problem]

[0005] One aspect of the present disclosure is Detecting a driving environment around the vehicle from one or more sensors provided in the vehicle; When an occurrence of an event related to the traveling of the vehicle and other vehicles around the vehicle is detected based on the traveling environment around the vehicle, displaying image data indicating a warning on a display device provided in the vehicle by superimposing the image data on real space; a control unit that executes the When the control unit detects a characteristic object on or around a road as one of the events based on the driving environment around the vehicle, the control unit transmits image data showing a warning about the characteristic object to a following vehicle of the vehicle, together with an instruction to display the image data superimposed on real space. It is an information processing device.

[0006] Another aspect of the present disclosure is The computer Detecting a driving environment around the vehicle from one or more sensors provided in the vehicle; When an occurrence of an event related to the traveling of the vehicle and other vehicles around the vehicle is detected based on the traveling environment around the vehicle, displaying image data indicating a warning on a display device provided in the vehicle by superimposing the image data on real space; When a characteristic object is detected on or around a road as one of the events based on the driving environment around the vehicle, transmitting image data showing a warning about the characteristic object to a following vehicle of the vehicle together with an instruction to display the image data superimposed on real space; This is a method for doing this. [Effects of the Invention]

[0007] According to one aspect of the present disclosure, safe driving can be supported. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of processing in a vehicle according to the first embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of a vehicle. [Figure 3] FIG. 3 is a diagram illustrating an example of a functional configuration of the in-vehicle device. [Figure 4] FIG. 4 is a diagram showing an example of a warning event and a warning message. [Figure 5] FIG. 5 is a flowchart of the safe driving support process for a vehicle. DETAILED DESCRIPTION OF THE INVENTION

[0009] One aspect of the present disclosure is an information processing device. The information processing device includes a control unit that detects a driving environment around the vehicle using one or more sensors provided in the vehicle, and, when an event related to the driving of the vehicle and other vehicles around the vehicle is detected based on the driving environment around the vehicle, causes a display device provided in the vehicle to display image data indicating a warning superimposed on real space. When the control unit detects a characteristic object on or near the road as one of the events based on the driving environment around the vehicle, the control unit transmits image data indicating a warning related to the characteristic object to a following vehicle along with an instruction to display the image data superimposed on real space.

[0010] The information processing device is, for example, a data communication module mounted on a mobile body, a navigation device, a drive recorder, etc. The devices are not limited to these, and may include, for example, computers mounted on moving bodies other than vehicles, smartphones, tablet terminals, PCs (Personal Computers), game terminals, etc. Any mobile terminal may be used. The information processing device is not limited to being mounted on a vehicle, but may also be mounted on a mobile object other than a vehicle. Examples of the mobile object include automobiles, motorcycles, bicycles, and trains. The mobile object is not limited to vehicles that run on the ground, but may include, for example, mobile objects that move on the sea, such as ships, and mobile objects that move in the air, such as aircraft and drones. Alternatively, the information processing device may be a server located in a location separate from the vehicle.

[0011] The control unit is, for example, a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a DSP (Digital Signal Processor). However, the control unit is not limited to a processor, and may be a circuit such as an FPGA (Field Programmable Gate Array), a semiconductor integrated circuit (IC: Integrated Circuit), or a CPLD (Complex Programmable Logic Device).

[0012] The one or more sensors provided in the vehicle are, for example, a camera, a speedometer, and a radar such as a millimeter wave radar and a LiDAR (Lider). The driving environment around the vehicle detected from the vehicle detection system includes, for example, road signs, road markings, pedestrians, other vehicles, and buildings. Events related to the driving of the vehicle and other vehicles around the vehicle include, for example, an event of a traffic law violation by the vehicle, an event in which a traffic law violation by the vehicle is predicted, and an event in which an accident is predicted. Events of a traffic law violation by the vehicle include, for example, driving the wrong way, speeding, and entering a road where entry is prohibited. Events in which a traffic law violation by the vehicle is predicted include, for example, entering an intersection where a right or left turn is prohibited, and approaching a stop line at a speed above a predetermined value. Events in which an accident is predicted include, for example, the presence of features on or around the road. Examples of features include pedestrians, bicycles and motorcycles in motion, parked vehicles, vehicles involved in accidents, construction notices and traffic control workers and other objects and personnel related to construction sites, tires, cardboard boxes, and other objects falling from trucks, etc.

[0013] The display device provided in the vehicle is, for example, an organic EL display attached to the windshield. Examples of image data that may be used include a display of a car navigation system, a display screen, a screen display, etc. Image data that indicates a warning may include, for example, a message, an illustration, a mark, etc. that indicates a warning. Image data that indicates a warning may also be video image data. Displaying the image data superimposed on real space is what is known as AR (Augmented Reality) display.

[0014] According to one aspect of the present disclosure, when an event related to the driving of the vehicle or other vehicles in the vicinity of the vehicle is detected, a warning is displayed in AR on a display device provided in the vehicle, which allows the driver of the vehicle to intuitively recognize the warning and assists safe driving.

[0015] In one aspect of the present disclosure, when a feature is detected on or near a roadway based on the driving environment around the vehicle, the control unit may transmit image data indicating a warning about the feature to a following vehicle of the vehicle, along with an instruction to display the image data superimposed on real space. The image data indicating the warning about the feature may be transmitted, for example, via device-to-device communication between vehicles. The device-to-device communication may be, for example, a communication method capable of terminal-to-terminal communication, such as vehicle-to-vehicle wireless LAN communication, dedicated short-range communications (DSRC), 5G sidelink communication, device-to-device (D2D) communication, or ad hoc mode. These device-to-device communications may be multi-hop communications. However, the device-to-device communications are not limited to these. The image data indicating the warning about the feature may also be transmitted, for example, via road-to-vehicle communication or via a server, without being limited to device-to-vehicle communication between vehicles. The road-to-vehicle communication may be, for example, Wi-Fi, DSRC, or the like. In the case of Wi-Fi, the roadside device may be, for example, an access point installed at a predetermined location.

[0016] This allows the following vehicle to quickly display image data indicating a warning about the characteristic object on its display device in AR, thereby encouraging the driver of the following vehicle to become aware of the characteristic object.

[0017] As another aspect, the present disclosure can be identified as a method for a computer to execute the processing performed by the information processing device, a program for causing a computer to execute the method, and a non-transitory computer-readable recording medium on which the program is recorded.

[0018] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. The configurations of the following embodiments are examples, and the present disclosure is not limited to the configurations of the embodiments.

[0019] First Embodiment FIG. 1 is a diagram illustrating an example of processing of a vehicle 10 according to the first embodiment. In the first embodiment, the vehicle 10 is a vehicle equipped with a communication function. The vehicle 10 detects the driving environment around the vehicle 10 using various sensors mounted thereon. The driving environment around the vehicle 10 includes road signs, road markings, pedestrians, other vehicles, buildings, and the like present around the vehicle 10. When the vehicle 10 detects the occurrence of an event related to the driving of the vehicle 10 and other vehicles in the vicinity based on the detected driving environment, the vehicle 10 displays image data indicating a warning superimposed on real space on a display device mounted on the vehicle 10. For example, if the windshield of the vehicle 10 is a display device, image data indicating the warning is displayed on the windshield so as to be superimposed on the real space in front of the vehicle 10. Hereinafter, the superimposed display of image data on real space will be referred to as AR display.

[0020] Events related to the traveling of the vehicle 10 and other vehicles in the vicinity include, for example, an event of a violation of a traffic law by the vehicle, an event in which a violation of a traffic law by the vehicle 10 is predicted, and an event in which an accident is predicted. Events related to a violation of a traffic law by the vehicle 10 include, for example, Examples of events that may result in a violation of traffic laws by the vehicle 10 include driving the wrong way, speeding, and entering a road where entry is prohibited. Examples of events that may result in a violation of traffic laws by the vehicle 10 include entering an intersection where right or left turns are prohibited, and approaching a stop line at a speed above a predetermined value. Examples of events that may result in an accident include the presence of a characteristic object on or near the road. Examples of characteristic objects include pedestrians, bicycles and motorcycles in motion, parked vehicles, accident vehicles, construction notices and traffic control workers, and objects and personnel related to construction sites, such as tires and cardboard boxes, that have fallen from trucks. Characteristic objects are also referred to as hazardous objects. Hereinafter, an event related to the travel of the vehicle 10 and other nearby vehicles is referred to as a warning event. In the first embodiment, when a warning event is detected, a warning is displayed in AR on the vehicle 10, allowing the driver of the vehicle 10 to intuitively recognize the warning and supporting safe driving.

[0021] Furthermore, when the vehicle 10 detects a characteristic object, it transmits image data showing a warning about the characteristic object and an instruction to display the image data in AR to the following vehicle 20. The image data and the instruction to display the AR are transmitted, for example, via vehicle-to-vehicle communication. As a result, the image data showing the warning about the characteristic object is quickly displayed in AR on the display device installed in the following vehicle 20.

[0022] FIG. 2 is a diagram showing an example of the hardware configuration of the vehicle 10. In FIG. 2, hardware components related to the processing of the first embodiment are extracted and shown. The vehicle 10 is equipped with an in-vehicle device 1. The in-vehicle device 1 includes, for example, a data communication module, a camera, a - Navigation systems, drive recorders, etc.

[0023] The in-vehicle device 1 includes, as its hardware configuration, a processor 101, a memory 102, an auxiliary storage device 103, a wireless communication unit 104, an inter-vehicle communication unit 105, and an interface 106. The memory 102 and the auxiliary storage device 103 are computer-readable recording media. The auxiliary storage device 103 is, for example, an erasable programmable read only memory (EPROM). , a hard disk drive, or a solid state drive (SSD). Programs stored in the auxiliary storage device 103 include, for example, an operating system (OS) and a communication method selection control program. The memory 102 includes, for example, semiconductor memory such as a read only memory (ROM) and a random access memory (RAM).

[0024] The processor 101 is, for example, a CPU, a GPU, or a DSP (Digital Signal Processor). The number of processors 101 is not limited to one, and multiple processors 101 may be provided. The processor 101 is an example of a "control unit."

[0025] The wireless communication unit 104 is a wireless communication circuit that supports a cellular communication method such as 5G, 4G, or 6G. The vehicle-to-vehicle communication unit 105 is a wireless communication circuit that supports a predetermined vehicle-to-vehicle communication method. The vehicle-to-vehicle communication method employed by the vehicle-to-vehicle communication unit 105 includes, for example, vehicle-to-vehicle wireless LAN communication, DSRC, 5G side link communication, device-to-device communication, and a communication method that enables terminal-to-terminal communication, such as ad hoc mode. These vehicle-to-vehicle communications may be multi-hop. However, the communication methods employed by the vehicle-to-vehicle communication unit 105 are not limited to these.

[0026] The interface 106 is an interface for connecting to an in-vehicle network to which other devices are connected, such as an in-vehicle Ethernet network or a CAN (Controller Area Network).

[0027] Furthermore, the vehicle 10 includes a camera 120 as a sensor for detecting the driving environment and driving conditions. The vehicle 10 includes a speedometer 130, an obstacle detection sensor 140, and a direction sensor 150. These sensors are connected to an in-vehicle network to which the interface 106 of the in-vehicle device 1 connects. The camera 120 is installed, for example, above the windshield, facing the outside of the vehicle 10. However, this is not limited thereto, and multiple cameras 120 may be installed outside the vehicle 10 so that each camera faces a different direction. The speedometer 130 is a sensor that detects the speed of the vehicle 10. The obstacle detection sensor 140 is a sensor that detects obstacles, such as a millimeter-wave radar or LiDAR. The direction sensor 150 is a sensor that detects the traveling direction of the vehicle 10, such as an acceleration sensor. Hereinafter, the camera 120, the speedometer 130, the obstacle detection sensor 140, and the direction sensor 150 will be collectively referred to as a sensor group.

[0028] The vehicle 10 also includes a display 110. The display 110 is connected to an in-vehicle network to which the interface 106 of the in-vehicle device 1 is connected. The display 110 is, for example, an organic EL display installed on the windshield of the vehicle 10, with the entire windshield serving as a display screen. However, the display 110 is not limited to being installed on a window, and may be, for example, a stand-alone display. The display 110 may also be, for example, a display used in a car navigation system. The hardware configuration of the in-vehicle device 1 is not limited to that shown in FIG. 2.

[0029] 3 is a diagram showing an example of the functional configuration of the in-vehicle device 1. The in-vehicle device 1 includes, as its functional configuration, a control unit 11. The control unit 11 is a functional component that is achieved, for example, by the processor 101 of the in-vehicle device 1 executing a predetermined program.

[0030] The control unit 11 acquires sensor detection values ​​from the sensor group at predetermined intervals while the vehicle 10 is traveling. For example, a captured image is acquired from the camera 120 as a detection value. The speedometer 130 acquires the speed of the vehicle 10. The obstacle detection sensor 140 acquires, for example, the presence or absence of an obstacle, the size of the obstacle, the type of the obstacle, and the distance of the obstacle from the vehicle 10. The direction sensor 150 acquires the traveling direction of the vehicle 10.

[0031] The control unit 11 analyzes the detection value of the sensor and determines whether or not a warning event has occurred based on the analysis result. Details of warning events will be described later. If it is determined that a warning event has occurred, the control unit 11 generates image data indicating a warning according to the type of the warning event that has occurred and displays the image data in AR on the display 110.

[0032] If the warning event that has occurred is the presence of a feature on or near the road, the control unit 11 generates image data indicating a warning about the feature and transmits the image data and an instruction to display the image data in AR to the following vehicle 20 via the vehicle-to-vehicle communication unit 105. For example, the image data and the instruction to display the image data in AR may be transmitted by multicast or broadcast. Alternatively, if negotiation with the following vehicle 20 has been performed in advance, the image data and the instruction to display the image data in AR may be transmitted to the following vehicle 20 by unicast. When the following vehicle 20 receives the image data and the instruction to display the image data in AR from the vehicle 10 via vehicle-to-vehicle communication, the following vehicle 20 displays the image data indicating the warning about the feature in AR on the display. Note that the functional configuration of the in-vehicle device 1 is not limited to that shown in FIG. 3.

[0033] Fig. 4 is a diagram showing an example of a warning event and a warning message. Fig. 4 also shows detection conditions for the occurrence of a warning event. That is, when the control unit 11 determines that the detection conditions are satisfied, it detects the occurrence of the corresponding warning event and generates image data including a warning message corresponding to the detected warning event.

[0034] For example, the detection condition for a stop warning event is that a stop road sign installed along the road or a stop line is detected in the image captured by camera 120. However, the detection condition for the occurrence of a stop warning event is not limited to this, and may be, for example, that a stop road sign or a stop line is detected in the image captured by camera 120 and the speed of vehicle 10 is equal to or greater than a predetermined speed at which it is difficult to stop before the stop line. The warning message used when a stop warning event occurs may be, for example, "Stop soon."

[0035] For example, the detection condition for a no entry warning event is that a road sign prohibiting vehicle entry is detected in the image captured by the camera 120. When a no entry warning event occurs, for example, a warning message such as "Road prohibited from entry exists" is used.

[0036] For example, the detection condition for a warning event of a travel direction restriction is that a no-travel sign outside the travel direction is detected in the image captured by the camera 120. A no-travel sign outside the travel direction is, for example, a sign indicating no left turns, no right turns, etc. The warning message when a warning event of a travel direction restriction occurs may be, for example, "Restrictions on travel direction."

[0037] For example, a detection condition for a wrong-way driving warning event is that the traveling direction of vehicle 10 and the traveling direction of another vehicle traveling on either side of the center line detected from the image captured by camera 120 are the same. However, the detection condition for the occurrence of a wrong-way driving warning event is not limited to this, and detection conditions are set based on signs, road markings, the number of lanes, and the like that can be detected from the image captured by camera 120 according to various road conditions. A warning message used when a wrong-way driving warning event occurs may be, for example, "Wrong-way driving!!" Furthermore, because wrong-way driving indicates that vehicle 10 is violating traffic laws and is likely to pose a danger to surrounding vehicles, the display mode of the warning message, such as color or font, may be made more prominent than other warning messages.

[0038] For example, the detection condition for a warning event of dangerous object detection is that a dangerous object has been detected on or around the road by the obstacle detection sensor 140. The warning message when a warning event of dangerous object detection occurs may be, for example, "Dangerous object present!" Note that the warning events, detection conditions for the occurrence of warning events, and warning messages shown in FIG. 4 are merely examples, and are not limited to the examples shown in FIG. 4.

[0039] Fig. 5 is a flowchart of a safe driving support process for the vehicle 10. The process shown in Fig. 5 is repeatedly executed while the vehicle 10 is in operation or traveling. The fact that the vehicle 10 is traveling is detected, for example, by the traveling system of the vehicle 10 being turned on. The process shown in Fig. 5 is executed mainly by the CPU 101 of the in-vehicle device 1, but for convenience, the process will be described mainly focusing on the functional components.

[0040] In OP101, the control unit 11 acquires detection values ​​from the group of sensors. In OP102, the control unit 11 analyzes the acquired detection values. For example, the control unit 11 performs image recognition processing on the image captured by the camera 120 to detect road signs, road markings, the traveling direction of surrounding vehicles, and the like.

[0041] In OP103, the control unit 11 determines whether or not a warning event has occurred based on the analysis result of the detection value of the sensor. If a warning event has occurred (OP103: YES), the process proceeds to OP104. If a warning event has not occurred (OP103: NO), the process shown in FIG. 5 ends.

[0042] In OP104, the control unit 11 acquires image data including a warning message corresponding to the detected warning event. The image data including the warning message may be read from the auxiliary storage device 103 in advance for each type of warning event, or may be read from the auxiliary storage device 103 by machine learning. It may be created using a model or the like.

[0043] In OP105, the control unit 11 determines whether the detected warning event is a warning event for the detection of a hazardous object. If the detected warning event is a warning event for the detection of a hazardous object (OP105: YES), the process proceeds to OP106. In OP106, the control unit 11 transmits, via vehicle-to-vehicle communication, to the following vehicle 20, image data including a warning message corresponding to the detection of a hazardous object and an instruction to display the image data in AR.

[0044] If the detected warning event is not a warning event for the detection of a dangerous object (OP105: NO), the process proceeds to OP107. In OP107, the control unit 11 displays the image data acquired in OP104 in AR on the display 110. Then, the process shown in FIG. 5 ends.

[0045] Note that the process shown in Fig. 5 is an example, and the safe driving support process is not limited to the process shown in Fig. 5. Priorities may be set in advance for warning events, and when the occurrence of multiple warning events is detected, the warning message with the highest priority may be displayed. The priority of warning events may be set in the following order, for example: an event in which a violation of a traffic law by the vehicle 10 is predicted, an event in which an accident is predicted, and an event in which the vehicle violates a traffic law. However, the priority is not limited to this, and may be set at a granularity of a warning event unit.

[0046] <Effects of the First Embodiment> In the first embodiment, when the vehicle 10 detects the occurrence of a warning event using a sensor, it displays image data including a warning message corresponding to the warning event in AR on the display 110. This can assist safe driving of the vehicle 10. Furthermore, in the first embodiment, when the vehicle 10 detects a hazardous object on or around the road, it transmits image data indicating a warning about the hazardous object and an instruction to display the image data in AR to the following vehicle 20. This can quickly notify the following vehicle 20 of the presence of the hazardous object, and can encourage the driver of the following vehicle 20 to notice the hazardous object.

[0047] <Other embodiments> The above-described embodiment is merely an example, and the present disclosure can be modified and implemented as appropriate within the scope that does not deviate from the gist of the disclosure.

[0048] In the first embodiment, the vehicle 10 detects a warning event, acquires image data including a warning message, and displays it in AR. Alternatively, the vehicle 10 may transmit a detection value of a sensor to a server, and the server may detect the occurrence of a warning event, acquire image data including a warning message, and transmit the image data to the vehicle 10 for AR display. Furthermore, if the warning event is the detection of a dangerous object, the server may identify the vehicle 20 following the vehicle 10 and transmit image data including a warning message regarding the dangerous object and an instruction to display AR to the following vehicle 20.

[0049] The processes and means described in this disclosure can be freely combined and implemented as long as no technical contradiction occurs.

[0050] Furthermore, a process described as being performed by one device may be shared and executed by multiple devices. Alternatively, a process described as being performed by different devices may be executed by a single device. In a computer system, the hardware configuration (server configuration) by which each function is realized can be flexibly changed.

[0051] The present disclosure relates to a computer program that implements the functions described in the above embodiments. The computer program may also be implemented by supplying the program to a computer, which then reads and executes the program on one or more processors of the computer. Such a computer program may be provided to the computer on a non-transitory computer-readable storage medium connectable to the computer's system bus, or may be provided to the computer via a network. Non-transitory computer-readable storage media include, for example, any type of disk, such as a magnetic disk (e.g., a floppy disk, a hard disk drive (HDD), etc.), an optical disk (e.g., a CD-ROM, a DVD disk, a Blu-ray disk), a read-only memory (ROM), a random access memory (RAM), an EPROM, an EEPROM, a magnetic card, a flash memory, an optical card, or any type of medium suitable for storing electronic instructions. [Explanation of symbols]

[0052] 1...In-vehicle device 10. Vehicle 11 Control unit 101 Processor 102 Memory 103...Auxiliary storage device 104 Wireless Communication Unit 105 Vehicle-to-vehicle communication unit

Claims

1. Detecting a driving environment around the vehicle from one or more sensors provided in the vehicle; When an occurrence of an event related to the traveling of the vehicle and other vehicles around the vehicle is detected based on the traveling environment around the vehicle, displaying image data indicating a warning on a display device provided in the vehicle by superimposing the image data on real space; a control unit that executes the When the control unit detects a characteristic object on or around a road as one of the events based on the driving environment around the vehicle, the control unit transmits image data showing a warning about the characteristic object to a following vehicle of the vehicle, together with an instruction to display the image data superimposed on real space. Information processing device.

2. The control unit detecting at least road signs and road markings as the driving environment around the vehicle from an image captured by an imaging device provided in the vehicle; Detecting at least the speed and the direction of travel of the vehicle as the traveling state of the vehicle; when detecting, as the event, a violation of a traffic law by the vehicle or an occurrence of an event in which a violation of a traffic law by the vehicle is predicted from at least one of the detected traveling direction of the vehicle, the road sign, and the road marking, causing the display device to superimpose the image data indicating the warning on real space and display it; The information processing device according to claim 1 .

3. The control unit Further, a traveling direction of the other vehicle is detected from the captured image. when it is detected that the vehicle is traveling in the wrong direction from at least one of the detected traveling direction of the vehicle, the traveling direction of the other vehicle, the center line of the road on which the vehicle is traveling, and the road sign, displaying, on the display device, image data indicating that the vehicle is traveling in the wrong direction as image data indicating the warning, by superimposing the image data on the real space; The information processing device according to claim 2 .

4. The computer Detecting a driving environment around the vehicle from one or more sensors provided in the vehicle; When an occurrence of an event related to the traveling of the vehicle and other vehicles around the vehicle is detected based on the traveling environment around the vehicle, displaying image data indicating a warning on a display device provided in the vehicle by superimposing the image data on real space; When a characteristic object is detected on or around a road as one of the events based on the driving environment around the vehicle, transmitting image data showing a warning about the characteristic object to a following vehicle of the vehicle together with an instruction to display the image data superimposed on real space; How to do it.

Images