Data processing device and data processing method

Abstract

To provide suitable feedback to an occupant of a vehicle.SOLUTION: A data processing device 20 installed in a vehicle includes an occupant state recognition unit 22 that recognizes an action of an occupant of the vehicle pointing outside the vehicle; a surroundings acquisition unit 21 that acquires surroundings of the vehicle; an object recognition unit 24 that recognizes an object pointed at by the occupant based on the recognition results obtained by the occupant state recognition unit 22 and the acquisition results obtained by the surroundings acquisition unit 21; a relative direction determination unit 25 that determines a relative direction of the object with respect to the vehicle; and a lighting control unit 27 that controls a lighting mode of a long light-emitting unit 15 installed above or below a windshield of the vehicle in a vehicle-width direction. The lighting control unit 27 controls the lighting mode such that the relative direction can be identified based on the recognition results of the object recognition unit 24, and changes the lighting mode in accordance with the change in relative direction as the vehicle moves.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a data processing device and a data processing method.

Background Art

[0002] Conventionally, there is a technique for identifying a target object pointed to by a user who is an occupant of a vehicle. For example, Patent Document 1 describes: "To provide a target object identification device that accurately identifies a target object present in the direction pointed to by the user's hand or finger. The positioning unit 13 detects the current position of the vehicle and the azimuth of the vehicle. The imaging unit 18 captures an image of the surroundings of the vehicle. The pointing direction detection unit 16 detects the pointing direction indicated by the user in the vehicle using his or her own hand. The target object extraction unit extracts, from the image captured by the imaging unit 18, a target object present in the pointing direction detected by the pointing direction detection unit 16. The target object position identification unit identifies the position of the target object extracted by the target object extraction unit relative to the vehicle."

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of Invention

Problems to be Solved by the Invention

[0004] In the conventional technology, it is difficult to provide feedback on how the in-vehicle device recognizes the pointing action by the vehicle occupant. For example, when a driver points to a target through the windshield, it is desirable that the in-vehicle device establish an interaction that transmits to the driver as feedback that the pointed direction has been recognized, or that the pointed target has been recognized. This feedback is required to be performed without obstructing the driver's visual check of the surroundings.

[0005] The present invention aims to provide a data processing device and a data processing method that provide feedback suitable for vehicle occupants. [Means for solving the problem]

[0006] To achieve the above objective, one representative data processing device of the present invention is a data processing device installed in a vehicle, comprising: an occupant state recognition unit that recognizes an action by an occupant of the vehicle pointing to the outside of the vehicle; an ambient condition acquisition unit that acquires the conditions around the vehicle; an object recognition unit that recognizes an object pointed to by the occupant based on the recognition result by the occupant state recognition unit and the acquisition result by the ambient condition acquisition unit; a relative direction determination unit that determines the relative direction of the object to the vehicle; and a light emission control unit that controls the light emission mode of a long light-emitting unit installed above or below the windshield of the vehicle in the direction of the width of the vehicle body, wherein the light emission control unit controls the light emission mode so as to be able to identify the relative direction based on the recognition result by the object recognition unit, and changes the light emission mode in accordance with the change when the relative direction changes due to the movement of the vehicle. [Effects of the Invention]

[0007] According to the present invention, suitable feedback can be provided to vehicle occupants. Other issues, configurations, and effects will be clarified by the following description of embodiments. [Brief explanation of the drawing]

[0008] [Figure 1] This diagram illustrates the overview of operations and behavior in the data processing of Example 1. [Figure 2] Configuration diagram of the entire in-vehicle system including the data processing device of Example 1 [Figure 3] Flowchart explaining light emission control [Figure 4] Diagram illustrating the control of light emission patterns. [Modes for carrying out the invention]

[0009] The following examples will be described with reference to the drawings. [Examples]

[0010] Figure 1 is an explanatory diagram illustrating the overview of operations and behavior in data processing in Embodiment 1. Figure 2 is a configuration diagram of the entire in-vehicle system 10, including the data processing device 20 of Embodiment 1. The in-vehicle system 10 mounted on the vehicle comprises the data processing device 20, an in-vehicle camera 12 for imaging the interior of the vehicle, and an exterior camera 11 for imaging the area around the vehicle. The in-vehicle system 10 includes a long, elongated light-emitting unit 15 installed below the vehicle's windshield, extending in the direction of the vehicle's width. The light-emitting unit 15 is, for example, a light-emitting device in which multiple LEDs (Light Emitting Diodes) are arranged in the direction of the vehicle's width, and a light-emitting device is used that deliberately emits some of its LEDs at a low resolution that can be recognized even in the peripheral field of vision of the occupants (hereinafter referred to as "low-resolution LED").

[0011] The data processing device 20 acquires images from the in-vehicle camera 12 and recognizes the state of the vehicle's occupants, who are the users. For example, if an occupant points outside the vehicle, the data processing device 20 analyzes the images from the in-vehicle camera 12 to identify the position of the occupant's eyes and fingers, and determines that the line connecting the eye position and the fingertips is the direction of the pointing. The term "fingers" is used to mean at least one of the hand and fingers.

[0012] The data processing device 20 acquires information about the surroundings of the vehicle from the output of the external camera 11. The external camera 11 is, for example, a stereo camera, and the data processing device 20 can acquire a spatial map, which is three-dimensional map data of the area around the vehicle. LiDAR (Light Detection and Ranging) may be used instead of the stereo camera, or a combination of stereo camera and LiDAR may be used.

[0013] The data processing device 20 recognizes the object pointed to by the occupant as the pointed object using the pointing direction and a spatial map. The data processing device 20 determines the relative direction of the pointed object to its own vehicle. The data processing device 20 illuminates the light-emitting unit 15 in a light-emitting pattern that indicates the relative direction. The data processing device 20 continues the light-emitting control that indicates the relative direction even after the occupant stops pointing. When the relative direction changes due to the movement of the vehicle, the data processing device 20 changes the light-emitting pattern in accordance with the change.

[0014] In Figure 1, the data processing device 20 determines the direction pointed to by the occupant's hand H1 (A1). The data processing device 20 recognizes store T1 as the pointing target and displays a display L1 indicating the relative direction of store T1 on the light-emitting unit 15 (A2). The data processing device 20 continues to display the relative direction even after the occupant stops pointing. When the vehicle moves and the relative direction changes, the data processing device 20 changes the display L2 to follow the change in relative direction (A3).

[0015] The data processing device 20 can provide feedback on the results of its recognition of the occupant's pointing action by indicating the relative direction. The data processing device 20 continues to emit light while tracking changes in the relative direction, so it can clearly notify the occupant of the recognition result. The data processing device 20 displays the recognition result as a change in the pattern of light emission below the windshield, so the occupant can check the feedback from the data processing device 20 in their peripheral vision while directly visually inspecting the area around the vehicle.

[0016] In Figure 2, the data processing unit 20 is connected to the drive control unit 30 and the navigation unit 40. The drive control unit 30 is a unit that controls the acceleration, deceleration, and steering of the vehicle. The vehicle's navigation unit 40 is a unit that identifies the vehicle's position and searches for and guides the vehicle's route by referring to a map database. This map database can be used as an object database that shows the locations of objects that the occupants may point to.

[0017] The data processing device 20 is connected to the exterior camera 11, interior camera 12, microphone 13, speaker 14, and light emitting unit 15 included in the in-vehicle system 10. The exterior camera 11 is an imaging device that captures images of the surroundings of the vehicle. The interior camera 12 is an imaging device that captures images of the vehicle interior, and functions as a sensor that acquires the state of the occupant in the vehicle interior. The microphone 13 collects sounds inside the vehicle interior. The speaker 14 is a device installed in the vehicle interior that outputs sound to the occupant. The light emitting unit 15 is an elongated low-resolution LED installed below the windshield of the vehicle across the width direction of the vehicle body.

[0018] The exterior camera 11 is installed, for example, at a position that includes the occupant's field of view in its imaging range. The interior camera 12 is installed at a position where it can capture images of the occupant's eyes and hands. For example, it may be installed near the rearview mirror or near the interior light.

[0019] The data processing device 20 includes a surrounding situation acquisition unit 21, an occupant state recognition unit 22, a voice recognition unit 23, a target recognition unit 24, a relative direction determination unit 25, a voice output unit 26, and a light emission control unit 27. When the data processing device 20 is implemented by a computer, a CPU (Central Processing Unit) executes a program to implement the functions corresponding to the surrounding situation acquisition unit 21, occupant state recognition unit 22, voice recognition unit 23, target recognition unit 24, relative direction determination unit 25, voice output unit 26, and light emission control unit 27.

[0020] The surrounding situation acquisition unit 21 acquires the situation around the own vehicle. The surrounding situation acquisition unit 21 obtains a spatial map from the imaging result of the exterior camera 11 that captures the surroundings of the own vehicle. The surrounding situation acquisition unit 21 obtains the position data of the own vehicle and the surrounding map data specified by the navigation unit 40.

[0021] The occupant state recognition unit 22 acquires an image captured by the in-vehicle camera 12 and recognizes the state of the occupant of the vehicle. The state of the occupant recognized by the occupant state recognition unit 22 includes an action of pointing to the outside of the vehicle. The occupant state recognition unit 22 recognizes the position of the occupant's eyes and the position of the occupant's fingers. The occupant state recognition unit 22 recognizes the pointing direction from the positions of the occupant's eyes and fingers, and outputs the pointing direction to the target recognition unit 24.

[0022] The voice recognition unit 23 recognizes the utterance by the occupant from the voice collected by the microphone 13 installed in the vehicle. For example, a word used by the occupant together with a pointing action, such as "What is that?", may be recognized.

[0023] The target recognition unit 24 recognizes the target pointed to by the occupant using the surrounding situation and the pointing direction. The target recognition unit 24 superimposes the pointing direction on the spatial map, and recognizes an object on the spatial map located in the pointing direction as the target pointed to by the occupant. The timing at which the target recognition unit 24 recognizes the target may be the time when the occupant state recognition unit 22 recognizes the pointing, or may be the timing at which the voice recognition unit 23 recognizes a predetermined utterance.

[0024] The relative direction determination unit 25 determines the relative direction of the target recognized by the target recognition unit 24 with respect to the vehicle. The relative direction determination unit 25 uses the position of the target recognized by the target recognition unit 24 in the spatial map for the determination of the relative direction. The relative direction determination unit 25 can use the position data of the own vehicle specified by the navigation unit 40 and the surrounding map data for the determination of the relative direction. The relative direction determination unit 25 can use the data related to the speed and steering state of the own vehicle acquired from the drive control unit 30 for the determination of the relative direction. When the vehicle moves after the target is recognized, the relative direction determination unit 25 changes the relative direction in accordance with the movement.

[0025] The voice output unit 26 outputs a voice signal from the speaker 14. The voice output unit 26 causes the speaker 14 to output a predetermined voice signal based on the recognition result of the target recognition unit 24.

[0026] The light emission control unit 27 controls the light emission mode of the light emission unit 15. Based on the recognition result by the object recognition unit 24, the light emission control unit 27 controls the light emission mode so as to be able to identify the relative direction, and changes the light emission mode in accordance with the change when the relative direction changes due to the movement of the vehicle. The light emission control unit 27 stops the light emission from the light emission unit 15 when a predetermined termination condition is met. Applicable termination conditions include "a certain amount of time has elapsed since the object recognition unit 24 recognized the object" and "when the relative direction becomes the rear of the vehicle".

[0027] Figure 3 is a flowchart showing the processing procedure of the data processing device 20. The data processing device 20 repeatedly executes the processes from step S101 to step S107 shown in Figure 3. Step S101 (Occupant State Recognition Step) The occupant state recognition unit 22 recognizes the occupant's state based on the image captured by the in-vehicle camera 12 and determines whether or not the occupant is performing a pointing motion. If it determines that the occupant is not performing a pointing motion, the occupant state recognition unit 22 repeats step S101. If it determines that the occupant is performing a pointing motion, the occupant state recognition unit 22 proceeds to step S102.

[0028] Step S102 (Occupant State Recognition Step) The occupant state recognition unit 22 determines the pointing direction and proceeds to step S103. Step S103 (Surroundings acquisition step): The surroundings acquisition unit 21 acquires a spatial map from the imaging results of the external camera 11 and proceeds to step S104. Step S104 (Object Recognition Step): The object recognition unit 24 recognizes the object pointed to by the occupant using the spatial map and the pointing direction, and proceeds to step S105. Step S105 (Relative Direction Determination Step): The relative direction determination unit 25 determines the relative direction to the vehicle of the target recognized by the target recognition unit 24, and proceeds to step S106. Step S106 (Light emission control step): The light emission control unit 27 performs light emission control on the light emission unit 15 to indicate the relative direction and determines whether the light emission termination condition has been met. If the termination condition is not met, the process returns to step S105. If the termination condition is met, the process proceeds to step S107. In step S107, the light emission control unit 27 terminates the light emission from the light emission unit 15 and returns to step S101.

[0029] Figure 4 is an explanatory diagram regarding the control of the light emission mode. The conditions for initiating object recognition can include "pointing recognition" and "speech recognition of pre-registered words." The termination conditions for controlling the light emission mode can include "a certain amount of time has elapsed since object recognition," "a certain amount of time has elapsed since pointing ended," "the relative direction has become the rear of the vehicle," and "the use of the object recognition result has ended." The object recognition results can be used for purposes such as "providing information about the object," "registering locations and setting destinations," and "automatic control and automatic parking."

[0030] In the "Provide Target Information" mode, the audio output unit 26 outputs information about the target from the speaker 14. If the recognized target is a store, the audio output unit 26 outputs audio from the speaker 14 indicating the name, type, business hours, event information, etc. This information may be stored as a predetermined audio signal, or it may be converted from text information to audio information as appropriate. Display output is also possible, not just audio output.

[0031] In "location registration and destination setting," the object recognition unit 24 requests the navigation unit 40 to register the recognized object as a location or set a destination. In "automatic control and automatic parking," the object recognition unit 24 requests the drive control unit 30 to control automatic driving or automatic parking with the recognized object's location as the destination.

[0032] As described above, the disclosed data processing device 20 includes an occupant state recognition unit 22 installed in the vehicle that recognizes an action by an occupant of the vehicle pointing to the outside of the vehicle, an ambient condition acquisition unit 21 that acquires the conditions around the vehicle, an object recognition unit 24 that recognizes an object pointed to by the occupant based on the recognition result by the occupant state recognition unit 22 and the acquisition result by the ambient condition acquisition unit 21, a relative direction determination unit 25 that determines the relative direction of the object with respect to the vehicle, and a light emission control unit 27 that controls the light emission mode of a long light-emitting unit 15 installed above or below the windshield of the vehicle in the direction of the width of the vehicle body, wherein the light emission control unit 27 controls the light emission mode so as to be able to identify the relative direction based on the recognition result by the object recognition unit 24, and changes the light emission mode in accordance with the change when the relative direction changes due to the movement of the vehicle. This configuration and operation enable the data processing device 20 to provide appropriate feedback to the vehicle occupants.

[0033] The disclosed data processing device 20 further includes a speech recognition unit 23 that recognizes speech made by the occupant from the sound picked up by the microphone 13 installed inside the vehicle, and the object recognition unit 24 recognizes the object pointed to by the occupant at the timing when the speech recognition unit 23 recognizes a predetermined speech. Therefore, the data processing device 20 can accurately grasp the intention of the crew member pointing to an object and recognize the object.

[0034] The disclosed data processing device 20 further includes an audio output unit 26 that outputs an audio signal from a speaker 14 installed inside the vehicle, and the audio output unit 26 outputs a predetermined audio signal from the speaker 14 based on the recognition result of the target recognition unit 24. Therefore, the data processing device 20 can provide the crew with information regarding the recognition results.

[0035] As an example, the light emission control unit 27 stops the light emission from the light emission unit after a certain period of time has elapsed since the recognition of the object. As an example, the light emission control unit 27 stops the light emission from the light emission unit when the relative direction becomes the rear of the vehicle. In this way, the data processing device 20 can stop emitting light appropriately depending on the state.

[0036] The surrounding conditions acquisition unit 21 generates a spatial map based, for example, on information obtained from a LiDAR (Light Detection and Ranging) and / or stereo camera installed on the outside of the vehicle. Therefore, the data processing device 20 can accurately grasp the surrounding situation and recognize the target.

[0037] The light-emitting unit 15 is, for example, a low-resolution LED (Light Emitting Diode). Since the feedback from the light-emitting unit 15 does not require the output of detailed information, a low-resolution LED is suitable. Furthermore, a low-resolution LED can also be used as an interior decoration for vehicles. Instead of such low-resolution LEDs, the light-emitting unit 15 may use a high-resolution display device such as a liquid crystal display (LCD) or an organic light-emitting diode (OLED) display, and display the feedback at low resolution on all or part of it. Alternatively, a projector that projects light rays onto a part of the vehicle interior installed across the width of the vehicle, such as the dashboard, may be used, or a head-up display (HUD) that illuminates a part of the windshield may be used. Regardless of the form of the light-emitting unit 15 described above, it is intentionally emitted with a low resolution that is recognizable even in the occupant's peripheral vision, so that the occupant's gaze is not drawn to the light-emitting unit 15. The occupant can keep their gaze focused on the front windshield and recognize the status of the light-emitting unit 15 while paying attention to safety.

[0038] It should be noted that the present invention is not limited to the embodiments described above, and various modifications are included. For example, the embodiments described above are explained in detail to make the present invention easier to understand, and are not necessarily limited to those having all the configurations described. Furthermore, it is possible to replace or add configurations, not just delete them.

[0039] For example, if there are other passengers besides the driver, the present invention may be applied to those other passengers. Alternatively, a specific passenger may be designated as the target of the present invention from among multiple passengers.

[0040] Furthermore, the occupant's status may be acquired from any sensor other than the in-car camera, such as LIDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) or a stationary smartphone. In addition, it is possible to use a combination of multiple sensors and the outputs of multiple types of sensors.

[0041] Furthermore, in the above embodiment, the direction indicated by the occupant was determined to be along the extension of the line between the eye and the fingertip. However, it is also possible to recognize two points on the fingers and determine the direction indicated by the occupant along the extension of the line between these two points. [Explanation of Symbols]

[0042] 10: In-vehicle system, 11: Exterior camera, 12: Interior camera, 13: Microphone, 14: Speaker, 15: Light-emitting unit, 20: Data processing unit, 21: Surroundings acquisition unit, 22: Occupant status recognition unit, 23: Voice recognition unit, 24: Target recognition unit, 25: Relative direction determination unit, 26: Voice output unit, 27: Light emission control unit, 30: Drive control unit, 40: Navigation unit

Claims

1. A data processing device installed in a vehicle, An occupant state recognition unit that recognizes an action by an occupant of the vehicle that points to the outside of the vehicle, The surrounding conditions acquisition unit acquires the conditions around the vehicle, Based on the recognition results from the occupant state recognition unit and the acquisition results from the surrounding situation acquisition unit, an object recognition unit recognizes the object pointed to by the occupant, A relative direction determination unit that determines the relative direction of the target with respect to the vehicle, The vehicle comprises a light emission control unit that controls the light emission mode of a long, elongated light-emitting unit installed above or below the windshield of the vehicle, extending in the direction of the vehicle's width, The light emission control unit controls the light emission pattern so as to be able to identify the relative direction based on the recognition result by the object recognition unit, and changes the light emission pattern in accordance with the change when the relative direction changes due to the movement of the vehicle. A data processing device characterized by the following features.

2. The vehicle further includes a voice recognition unit that recognizes speech uttered by the occupant from the sound picked up by a microphone installed inside the vehicle. The data processing device according to claim 1, characterized in that the object recognition unit recognizes the object pointed to by the occupant at the timing when the speech recognition unit recognizes a predetermined utterance.

3. The data processing device according to claim 1, characterized in that the light emission control unit stops emitting light from the light emission unit after a certain period of time has elapsed since the recognition of the target.

4. The data processing apparatus according to claim 1, characterized in that the light emission control unit stops emitting light from the light emission unit when the relative direction becomes the rear of the vehicle.

5. The data processing device according to claim 1, characterized in that the light emission mode is such that a part of the light-emitting unit is illuminated with a low resolution that can be recognized even in the peripheral field of vision of the occupant.

6. A data processing device installed in a vehicle, which has a long, luminescent unit mounted above or below the windshield, extending in the direction of the vehicle's width, A passenger state recognition step in which the passenger of the vehicle points to the outside of the vehicle, A surrounding conditions acquisition step to acquire the conditions around the vehicle, Based on the recognition result from the occupant state recognition step and the acquisition result from the surrounding situation acquisition step, an object recognition step is performed to recognize the object pointed to by the occupant, A relative direction determination step for determining the relative direction of the target with respect to the vehicle, The system includes a light emission control step for controlling the light emission mode of the light-emitting unit, The light emission control step controls the light emission pattern so as to be able to identify the relative direction based on the recognition result from the target recognition step, and changes the light emission pattern in accordance with the change when the relative direction changes due to the movement of the vehicle. A data processing method characterized by the following features.

Images