Control method and control device

Abstract

The present invention provides a control method and control device that can control in-vehicle equipment even when gestures from multiple occupants are detected. [Solution] The control method according to the present invention is a control method for determining a control gesture in a vehicle having a control unit that controls in-vehicle equipment by a gesture made by an occupant, wherein when the control unit of the vehicle detects the gesture made by the fingers of multiple occupants inside the vehicle from a captured image taken inside the vehicle, it identifies at least a part of the occupant's body to which the detected fingers belong, identifies a vehicle part in contact with the occupant from the identified part of the body, determines the seating position of the occupant based on the identified vehicle part, and selects one gesture from among the detected multiple gestures as a control gesture for controlling the equipment inside the vehicle based on the seating position.

Description

Technical Field

[0001] The present invention relates to a control method and a control device.

Background Art

[0002] Patent Document 1 discloses a device control apparatus that controls in-vehicle devices based on the gestures of passengers. In this device control apparatus, the posture and gestures of passengers are recognized from images captured by an in-vehicle camera, and based on these, the in-vehicle devices are controlled. Specifically, when all requirements for the 3D coordinates of the passenger's skeleton, gestures, and the time of gestures are satisfied, the in-vehicle devices are operated.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the above device control apparatus, only registered passengers can control in-vehicle devices by gestures, but control in the case where gestures by multiple passengers are detected is not disclosed. Therefore, control in such a situation has been desired. The present invention has been made to solve this problem, and an object thereof is to provide a control method and a control device capable of controlling in-vehicle devices even when gestures by multiple passengers are detected.

Means for Solving the Problems

[0005] The control method according to the present invention is a control method for determining a control gesture in a vehicle having a control unit that controls in-vehicle equipment by a gesture made by an occupant, wherein when the control unit of the vehicle detects the gesture made by the fingers of multiple occupants inside the vehicle from a captured image taken inside the vehicle, it identifies at least a part of the occupant's body to which the detected fingers belong, identifies a vehicle part in contact with the occupant from the identified part of the body, determines the seating position of the occupant based on the identified vehicle part, and selects one gesture from among the detected multiple gestures as a control gesture for controlling the equipment inside the vehicle based on the seating position.

[0006] The control device according to the present invention is a control device for determining a control gesture in a vehicle having a control unit that controls in-vehicle equipment by a gesture made by an occupant, and comprises a control unit, the control unit, when it detects the gesture made by the fingers of multiple occupants in the vehicle from a captured image taken inside the vehicle, identifies at least a part of the occupant's body to which the detected fingers belong, identifies a vehicle part in contact with the occupant from the identified part of the body, determines the occupant's seating position based on the identified vehicle part, and selects one gesture from the detected multiple gestures as a control gesture for controlling the equipment inside the vehicle, based on the seating position. [Effects of the Invention]

[0007] According to the present invention, in-vehicle equipment can be controlled even when gestures from multiple occupants are detected. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram showing the hardware configuration of a control device according to one embodiment of the present invention. [Figure 2] This is a block diagram showing the software configuration of the control device in Figure 1. [Figure 3] This is an example of an image taken inside a vehicle. [Figure 4]This is a flowchart showing the operation of the control device. [Figure 5] This flowchart shows other operations of the control device. [Figure 6] This flowchart shows other operations of the control device. [Modes for carrying out the invention]

[0009] The following describes a vehicle control device according to one embodiment of the present invention, with reference to the drawings. This control device is configured to detect gestures made by the occupant's fingers and to control in-vehicle equipment based on these gestures.

[0010] <1. Hardware configuration of the control unit> This control device 1 is a device that controls the operation of various in-vehicle devices 2 connected via a communication network or signal lines, and comprises a control unit 11 and a storage unit 12. The in-vehicle devices 2 are connected to the control device 1 as described above, and an imaging unit 3 is also connected to it. Examples of in-vehicle devices 2 include, but are not limited to, power windows, air conditioners, wipers, stereos, navigation systems, headlights, and interior lighting.

[0011] The imaging unit 3 is an imaging device such as a camera that captures images of the occupants. The imaging unit 10 is positioned to capture the occupants' upper bodies, their hands located in the space near their upper bodies, and a portion of the seat in which they are seated (for example, near the roof at the front of the vehicle). The imaging unit 3 may be provided for each seat to capture one occupant, or it may be provided at a specific location inside the vehicle to capture multiple occupants simultaneously. The camera used as the imaging unit 3 is not particularly limited, but for example, a visible light camera or an infrared camera can be used. More specifically, for example, it may be a device that combines a camera and a depth sensor, such as Kinect®. The imaging unit 3 can continuously capture images and acquire, for example, 10 to 30 images per second. The captured images are then transmitted to the control device 1.

[0012] The control unit 11 of the control device 1 includes a CPU, RAM, ROM, etc., and is configured to perform various information processing based on programs and various data. The storage unit 12 is composed of an auxiliary storage device such as an HDD or SSD, and stores programs 121, gesture data 122, and various data 123, etc.

[0013] Program 121, as described later, is a program for identifying gestures made by the occupant and controlling the in-vehicle equipment 2 based on the identified gestures. Gesture data 122 is data that registers the content and meaning of the gestures made by the occupant. Various data 123 are data for driving the control device 1, etc. However, the location where the data is stored is not particularly limited and can be changed as appropriate. For example, at least a portion of the data stored in the memory unit 12 can be stored in the RAM or ROM of the control unit 11.

[0014] <2. Software configuration of the control unit> Next, the software configuration of the control device 1 will be described with reference to Figure 2. Figure 2 is a block diagram showing the software configuration of the control device. As shown in Figure 4, when the control unit 11 loads the program 131 stored in the memory unit 13 into RAM, the CPU interprets and executes the program 131, and the control unit functions as a computer equipped with a recognition unit 111, a decision unit 112, and an equipment control unit 113.

[0015] The recognition unit 111 detects at least one gesture made by the occupant's fingers from the image captured by the imaging unit 3. That is, when multiple occupants are each making a gesture, all the gestures are detected. Also, gesture data 122 indicating the meaning of each gesture is registered in the storage unit 12 in advance. For example, the following meanings of gestures can be registered in the gesture data 122. As shown in Table 1, the gesture data 122 registers the shape and orientation of the gesture, time, in-vehicle device, and control content. For example, if the shape of the detected gesture is "fist" and that shape is maintained for 1 second or more, it indicates that control is performed to turn on the air conditioner. However, Table 1 is an example, and various gestures and their meanings can be registered.

Table 1

[0016] Next, the recognition unit 111 detects the shoulder of the occupant making the detected gesture from the detected gesture. In this case, the recognition unit 111 recognizes the occupant's skeleton by a known skeleton detection algorithm (for example, the algorithm implemented in the above-mentioned Kinect (registered trademark)), and based on the recognized skeleton, the shoulder of the occupant can be specified from the gesture.

[0017] Furthermore, the recognition unit 111 specifies the seat belt closest to the detected shoulder of each occupant from the detected positions of the shoulders of the occupants, and specifies the seat on which the occupant is seated from the specified seat belt. That is, each seat belt is associated with a seating position, and it is possible to specify which seating position the seat belt is from the detected position, shape, etc. of the seat belt. For example, assume that a driver's seat S1 and a passenger seat S2 are provided as front seats in a vehicle, and a rear seat S3 is provided behind the front seats. Suppose an image shown in FIG. 3 is acquired by the imaging unit 3. In the example of FIG. 3, occupants are seated in the driver's seat S1 and the passenger seat S2, and three occupants are seated in the rear seat S3. And in this example, the occupant in the driver's seat S1 and the occupant on the right side of the rear seat S3 are making gestures with their fingers.

[0018] In the present embodiment, the seating positions are specified as the driver's seat S1, the passenger seat S2, and the rear seat S3. However, they can also be specified as the areas where these seats are provided. For example, the areas separated by the dashed lines in FIG. 3 can be determined as the driver's seat S1, the passenger seat S2, and the rear seat S3. In this case, the setting of the areas can be appropriately set according to the vehicle structure.

[0019] The recognition unit 111 detects two gestures G1 and G2 from the captured image, and identifies the shoulders B1 and B2 of the passengers from each detected gesture G1 and G2. Subsequently, the seat belts V1 and V2 closest to the identified shoulder positions are identified, and the seating positions of the passengers can be identified by the seat belts V1 and V2. That is, it can be identified that the seating position of the passenger performing the gesture G1 is the driver's seat S1, and the seating position of the passenger performing the gesture G2 is the driver's seat S1.

[0020] In the example of FIG. 3, the gesture G1 is a gesture with the index finger pointing upward, and it is identified that this gesture is made by the passenger sitting in the driver's seat S1. Also, the gesture G2 is a gesture indicating a V sign, and it is identified that this gesture is made by the passenger sitting in the rear seat S3.

[0021] As described above, when a plurality of gestures are detected, the seating positions of the passengers who performed the gestures are identified from the shoulders and seat belts.

[0022] Next, the determination unit 112 will be described. When the determination unit 112 detects multiple gestures, it determines which gesture to prioritize. For this determination, the gesture data 122 stored in the memory unit 13 stores a priority order. For example, if the priority order is set to driver's seat, passenger seat, and rear seat, when multiple gestures are detected, the gesture of the occupant in the driver's seat S1 will have the highest priority. For example, in the example in Figure 3, gestures G1 and G2 of the occupants in the driver's seat S1 and rear seat S3 are detected, but it is determined that the gesture G1 of the occupant in the driver's seat S1 will be prioritized. Then, the gesture G1 of the occupant in the driver's seat S1 will be selected as the control gesture. Note that, as will be described later, when multiple gestures are detected, if all occupants who performed the gestures are not wearing seat belts, no control gesture is selected.

[0023] Next, the equipment control unit 113 controls the in-vehicle equipment 2 based on the control gesture. In the example shown in Figure 3, the gesture G1 of the occupant in the driver's seat S1 was selected as the control gesture, and based on this control gesture, the wipers are turned ON as shown in Table 1.

[0024] Furthermore, if a new gesture is detected after a predetermined time has elapsed since a control gesture was selected, the control gesture is selected based on the above process, and the in-vehicle device 2 is controlled anew.

[0025] Furthermore, if, within a predetermined time after detecting multiple gestures and identifying that the occupants performing these gestures are seated in the passenger seat or rear seat, it is determined that the seated position of a newly detected occupant performing a gesture is the driver's seat S1, then the gesture of the occupant seated in the driver's seat S1 is adopted as the control gesture. In other words, the control of the in-vehicle device 2 by the gesture of the occupant seated in the passenger seat S2 or rear seat S3 is stopped, and the in-vehicle device 2 is controlled by the gesture of the occupant seated in the driver's seat S1.

[0026] <3. Operation of the control device> Next, the operation of the control device configured as described above will be explained with reference to the flowchart shown in Figure 4.

[0027] As shown in Figure 4, the imaging unit 3 starts taking images, and images of the interior of the vehicle are acquired at predetermined time intervals (step S11). When the recognition unit 111 detects a gesture in the captured image (YES in step S12), the shoulder to which the fingers that made the gesture are connected is detected (step S13). Once the shoulder is detected, the seating position of the occupant is determined from the seat belt closest to that shoulder (step S14).

[0028] When the seating position is determined, if there is only one occupant making a gesture (YES in step S15), that gesture is used as the control gesture and the in-vehicle device 2 is controlled (step S16). On the other hand, if there are multiple occupants making gestures (NO in step S15), it is detected that one of them is wearing a seat belt (YES in step S17), and if that occupant is seated in the driver's seat S1 (YES in step S18), the gesture of the occupant in the driver's seat S1 is adopted as the control gesture and the in-vehicle device 2 is controlled (step S19).

[0029] On the other hand, if an occupant wearing a seat belt is not seated in the driver's seat S1 (step S18 NO), and that occupant is seated in the passenger seat S2 (step S120 YES), the gesture of the occupant in the passenger seat S2 is adopted as the control gesture to control the in-vehicle device 2 (step S121). Furthermore, if an occupant wearing a seat belt is not seated in either the driver's seat S1 or the passenger seat S2 (step S120 NO), and that occupant is seated in the rear seat S3 (step S122 YES), the gesture of the occupant in the rear seat S3 is adopted as the control gesture to control the in-vehicle device (step S123).

[0030] Furthermore, if the occupant performing the gesture is not wearing a seatbelt (NO in step S17), the control gesture is not selected, and the acquisition of images continues (step S11).

[0031] Next, referring to the flowchart in Figure 5, another example of prioritizing gestures made by the driver's seat occupant will be described. When a gesture made by an occupant in the passenger seat S2 or rear seat S3 is selected as the control gesture and the in-vehicle device 2 is being controlled (step S21), if a new gesture is detected within a predetermined time since that control was performed (YES in step S22), if any of the occupants making the gesture are wearing a seat belt (YES in step S23), and if any of those occupants are seated in the driver's seat S1 (YES in step S24), the control made by the control gesture made by the occupant in the passenger seat S2 or rear seat S3 that was already adopted is stopped (step S25), the newly detected gesture made by the occupant in the driver's seat S1 is adopted as the control gesture, and the in-vehicle device is controlled (step S26). On the other hand, if the answer to any of steps S22 to S24 is NO, control by the passenger in the front seat S2 or the rear seat S3, which has already been adopted, is continued (step S27).

[0032] <4. Features> According to this embodiment, the following effects can be obtained. (1) When multiple occupants' gestures are detected, the system is configured to select one of the gestures as the control gesture based on a priority order determined by the pre-set seating position, and to control the in-vehicle device 2 based on the selected control gesture. Therefore, when multiple gestures are detected, the control gesture is selected based on the seating position, eliminating the need to register occupants as in conventional technology, and simplifying operation.

[0033] (2) While it is difficult to determine the occupant's seating position based solely on the gesture position, in this embodiment, the seat belt is identified, and the seating position is determined based on this. This is because the seat belts for the driver's seat S1, passenger seat S2, and rear seat S3 are positioned so as not to interfere with each other, and the seating position can be easily determined based on the seat belt.

[0034] (3) If a gesture from the driver's seat occupant is detected after a gesture from the passenger seat or rear seat occupant has been selected, the system is configured to stop controlling the in-vehicle equipment up to that point and adopt the driver's seat occupant's gesture to control the in-vehicle equipment. This allows the driver's operation to be prioritized, thereby improving safety.

[0035] <5. Variation> Although one embodiment of the present invention has been described above, the present invention is not limited to the above embodiment, and various modifications are possible without departing from the spirit of the invention. For example, the following modifications are possible. Furthermore, the gist of the following modifications can be combined with each other as appropriate, and can also be combined with the above embodiment.

[0036] (1) In the above embodiment, the rear seat S3 is set as a single seating position, but the rear seat can also be divided into multiple seating positions. For example, if three people can sit in the rear seat, three seating positions can be set. Also, if the vehicle has three or more seats, seating positions can be set accordingly.

[0037] (2) In the above embodiment, the seating position is determined by detecting the shoulder connected to the fingers making the gesture and the seat belt closest to that person, but it is not limited to this. That is, it is not limited to the shoulder, but it is also possible to detect other parts of the body connected to the fingers making the gesture. Furthermore, it is not limited to the seat belt, but a specific part of the seat (such as the headrest) can also be used as a vehicle component to determine the seating position. In this case, the gesture of an occupant not wearing a seat belt can also be adopted as the control gesture.

[0038] (3) The priority order for seating positions is not limited to those described above and may be changed as appropriate.

[0039] (4) When the area around the vehicle is congested, the driver becomes highly engrossed in driving (the dynamic traffic element), and their level of attention and cognitive load are high. Operating in-vehicle equipment in this state may impair the ability to perform appropriate driving operations. In particular, the working memory of the dorsolateral prefrontal cortex is used for the recognition and judgment of visual information, but in the situations described above, there is a risk that working memory may overflow, which may impair the ability to perform appropriate driving operations. Furthermore, depending on individual characteristics and the degree of familiarity with driving, the level of attention and cognitive load may be high even when the area around the vehicle is quiet.

[0040] Therefore, it is possible to detect the driver's level of attention and cognitive load while driving, and select control gestures based on this. The method for detecting the driver's level of attention and cognitive load is not particularly limited, but for example, at least one of the following devices can be installed in the vehicle: a brain sensor to detect the driver's brain waves, a magnetoencephalograph to detect brain magnetic fields, an electroencephalograph to detect brain potentials, or a brain function imaging device that measures the activity state of the driver's brain surface using functional near-infrared spectroscopy (fNIRS), thereby measuring working memory (amount of brain activity). It is also possible to determine the level of attention and cognitive load based on the magnitude of brain activity. Furthermore, it is also possible to determine the amount of brain activity using equipment attached to the occupants.

[0041] When determining the level of brain activity in this way, control gestures can be selected, for example, as shown in the flowchart in Figure 5. As shown in Figure 5, first, the attention level or cognitive load of the occupant in the driver's seat S1 is detected (step S31). If the detected value is greater than the threshold (YES in step S32), the priority of the gesture by the occupant in the driver's seat S1 is lowered, and a control gesture is selected (step S33). For example, in the initial settings, the priority is set in the order of driver's seat S1, passenger seat S2, and rear seat S3, but this can be changed to, for example, passenger seat S2, rear seat S3, and driver's seat S1. In this case, if multiple gestures are detected, a control gesture is selected according to the newly set priority. Note that the change in priority is not particularly limited; it could also be in the order of passenger seat S2, driver's seat S1, and rear seat S3, or other priority can be set if there are other seating positions.

[0042] On the other hand, if the detected value is below the threshold (NO in step S32), a control gesture is selected based on the initial priority (step S34). [Explanation of Symbols]

[0043] 1: Control device 2:In-vehicle equipment 3: Photography Department 11: Control Unit B1,B2:Shoulder G1, G2: Gesture G2: Gesture S1: Driver's seat S2: Passenger seat S3: Rear seats V1, V2: Seat belt

Claims

1. A control method for determining a control gesture in a vehicle having a control unit that controls in-vehicle equipment by a gesture made by an occupant, The control unit of the aforementioned vehicle is: When the gestures made by the fingers of multiple occupants are detected from images taken inside the vehicle, at least a part of the occupant's body to which the detected fingers belong is identified. From at least one part of the body identified above, the vehicle parts that come into contact with the occupant are identified, Based on the identified vehicle parts, the seating position of the occupant is determined. Based on the seating position, one gesture is selected from among the detected multiple gestures to be used as a control gesture for controlling the in-vehicle equipment. Control method.

2. At least a part of the occupant's body is at the position of the occupant's shoulder, The control method according to claim 1.

3. The aforementioned vehicle component is a seat belt. The control method according to claim 2.

4. The control unit, Prioritizing multiple seating positions, When multiple gestures are detected by the imaging unit, the gesture performed by the occupant in the seating position with the highest priority is selected as the control gesture. The control method according to claim 1.

5. The aforementioned priority for the front seats is set higher than the aforementioned priority for the rear seats. The control method according to claim 4.

6. The priority of the driver's seat in the front row is set higher than the priority of the passenger seat in the front row. The control method according to claim 5.

7. The control gesture performed by the occupant seated in the driver's seat may also be used to change or cancel the control gesture performed by an occupant seated in another previously selected seat. The control method according to claim 1.

8. The control unit, Prioritizing multiple seating positions, If at least one of the attention level and cognitive load of the occupant seated in the driver's seat regarding driving operations is higher than a threshold, the priority of the driver's seat is reduced. The control method according to claim 1.

9. The amount of brain activity detected by at least one of an electroencephalogram (EEG) sensor, a magnetoencephalograph (MEG), an electroencephalograph (ENT), and an fNIRS device is defined as the attention level or cognitive load. The control method according to claim 8.

10. In a vehicle having a control unit that controls in-vehicle equipment by a gesture made by an occupant, a control device for determining the control gesture, Equipped with a control unit, The control unit, When the gestures made by the fingers of multiple occupants are detected from images taken inside the vehicle, at least a part of the occupant's body to which the detected fingers belong is identified. From at least one part of the body identified above, the vehicle parts that come into contact with the occupant are identified, Based on the identified vehicle parts, the seating position of the occupant is determined. Based on the seating position, one gesture is selected from among the detected multiple gestures to be used as a control gesture for controlling the in-vehicle equipment. Control device.

Images