State recognition device, state recognition method, computer readable medium, task execution system, and task execution method

Abstract

A state recognition device (2000) detects that a mirror (20) of a vehicle (10) is operated. The state recognition device (2000) determines a head position of a driver (30) of the vehicle (10) at a point in time when the mirror (20) is operated. The state recognition device (2000) sets the determined head position as a reference position of the head in the vehicle (10).

Description

Technical Field

[0001] The present disclosure relates to a state recognition device, a state recognition method, a computer-readable medium, a task execution system, and a task execution method. Background Art

[0002] A technology for detecting the state of a driver in a vehicle such as a car has been developed. For example, Patent Document 1 discloses a technology for detecting that the driver is looking away by using the direction of the driver's face when adjusting an outside mirror as a reference value.

[0003] Reference List

[0004] Patent Literature

[0005] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2009-009244. Summary of the invention

[0006] Technical issues

[0007] In Patent Document 1, the direction of the driver's face is used as a reference of the driver's state. An object of the present disclosure is to disclose a more effective technique for determining a reference of the driver's state.

[0008] Problem Solution

[0009] According to the present disclosure, a state recognition device includes: a detection device for detecting that a vehicle's reflector is operated; a determination device for determining the head position of the driver of the vehicle at the time point when the operation is performed; and a setting device for setting the determined head position as a reference position of the head in the vehicle.

[0010] According to the present disclosure, a state recognition method is performed by a computer. The state recognition method includes: a detection step of detecting that a reflector of a vehicle is operated; a determination step of determining the head position of a driver of the vehicle at a time point when the operation is performed; and a setting step of setting the determined head position as a reference position of the head in the vehicle.

[0011] According to the present disclosure, the program causes a computer to execute the state recognition method of the present disclosure.

[0012] Advantageous Effects of the Invention

[0013] According to the present disclosure, a more efficient technique for determining a reference for a driver's state is provided. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a diagram showing an overview of the operation of a state recognition apparatus according to an exemplary embodiment.

[0015] Figure 2 is a block diagram showing a functional configuration of a state recognition device.

[0016] Figure 3 is a block diagram showing the hardware configuration of a computer that implements the state recognition device.

[0017] Figure 4 is a flowchart showing the flow of processing performed by the state recognition device.

[0018] Figure 5 This is a diagram showing a case where the head position is expressed by a combination of coordinates of a plurality of points.

[0019] Figure 6 is a block diagram showing a functional configuration of a system including a state recognition device and an application device.

[0020] Figure 7 is a flowchart showing the flow of processing executed by the application device. DETAILED DESCRIPTION

[0021] Hereinafter, example embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the accompanying drawings, the same or corresponding elements are represented by the same reference numerals, and for the sake of clarity, repeated descriptions are omitted when necessary. In addition, unless otherwise described, predetermined values ​​such as predetermined values ​​and thresholds are pre-stored in a storage device or the like that can be accessed from a device using the values. In addition, unless otherwise described, the storage unit includes one or any larger number of storage devices.

[0022] <Overview>

[0023] Figure 1 2 is a diagram showing an overview of a state recognition device 2000 according to an exemplary embodiment. Here, Figure 1 The diagram is for facilitating understanding of the overview of the state recognition device 2000, and the operation of the state recognition device 2000 is not limited to Figure 1 The operation shown.

[0024] The state recognition device 2000 sets the position of the driver 30 at the time point when the reflector 20 provided in the vehicle 10 is operated as a reference position of the head in the vehicle 10 (hereinafter referred to as the reference head position). The vehicle 10 is, for example, a car. The reflector 20 is a reflector used by the driver 30 during driving, and is, for example, a rearview mirror or a side mirror.

[0025] Note that only one type of reflector may be regarded as the reflector 20, or a plurality of types of reflectors may be regarded as the reflector 20. In the latter case, for example, if any one of the rearview mirror and the two side mirrors is operated, the state recognition device 2000 determines that the reflector 20 is operated by the driver 30.

[0026] For example, the state recognition device 2000 operates as follows. First, the state recognition device 2000 detects the operation of the reflector 20 by the driver 30. In addition, the state recognition device 2000 determines the position of the head of the driver 30 (head position 40) at the time point when the driver 30 operates the reflector 20. Then, the state recognition device 2000 sets the determined head position 40 as the reference head position in the vehicle 10.

[0027] Note that although the head position 40 is determined by Figure 1 The head position 40 is represented by one point in , but the head position 40 can be represented by a combination of multiple points. Details on how to represent the head position 40 will be described later.

[0028] <Examples of beneficial effects>

[0029] According to the state recognition device 2000, in response to detecting the driver 30's operation of the reflector 20, the driver 30's head position at the time of the operation is set as the reference head position. As described above, according to the state recognition device 2000, a new technique for determining the reference of the vehicle driver's state is provided.

[0030] Here, it is considered that the position of the driver's 30 head at the time when the reflector 20 is operated is close to the position of the head at the time when the driver 30 looks forward at the vehicle 10. Therefore, according to the state recognition device 2000, the position of the driver's 30 head in a state close to the forward-looking state can be easily set as the reference position of the head.

[0031] Hereinafter, the state recognition device 2000 in this exemplary embodiment will be described in more detail.

[0032] <Example of function configuration>

[0033] Figure 2 2 is a block diagram showing a functional configuration of a state recognition device 2000 in an example embodiment. The state recognition device 2000 includes a detection unit 2020, a determination unit 2040, and a setting unit 2060. The detection unit 2020 detects an operation of the reflector 20 by the driver 30. The determination unit 2040 determines the head position 40 at a point in time when the driver 30 operates the reflector 20. The setting unit 2060 sets the determined head position 40 as a reference head position in the vehicle 10.

[0034] <Example of hardware configuration>

[0035] Each functional configuration unit of the state recognition device 2000 may be implemented by hardware that implements each functional configuration unit (e.g., a hard-wired electronic circuit, etc.), or may be implemented by a combination of hardware and software (e.g., a combination of an electronic circuit and a program that controls the electronic circuit, etc.). Hereinafter, a case where each functional configuration unit of the state recognition device 2000 is implemented by a combination of hardware and software will be further described.

[0036] Figure 3 1 is a block diagram showing a hardware configuration of a computer 1000 that implements the state recognition device 2000. The computer 1000 is any computer. For example, the computer 1000 is an electronic control unit (ECU), a navigation device, a tablet terminal, etc. provided inside the vehicle 10. In another example, the computer 1000 is a personal computer (PC) or a server machine provided outside the vehicle 10. The computer 1000 may be a dedicated computer designed to implement the state recognition device 2000, or may be a general-purpose computer.

[0037] For example, when a predetermined application is installed in the computer 1000, each function of the state recognition device 2000 is implemented in the computer 1000. The above-mentioned application is configured with a program for implementing the functional configuration unit of the state recognition device 2000. Note that the method of acquiring the program is arbitrary. For example, the program can be acquired from a storage medium (DVD disk, USB memory, etc.) storing the program. For example, the program can also be acquired by downloading the program from a server device that manages a storage device storing the program.

[0038] The computer 1000 includes a bus 1020, a processor 1040, a memory 1060, a storage device 1080, an input / output interface 1100, and a network interface 1120. The bus 1020 is a data transmission path for the processor 1040, the memory 1060, the storage device 1080, the input / output interface 1100, and the network interface 1120 to transmit and receive data with each other. However, the method of connecting the processor 1040 and the like to each other is not limited to the bus connection.

[0039] The processor 1040 is any processor such as a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), or a digital signal processor (DSP). The memory 1060 is a main storage device implemented by using a random access memory (RAM) or the like. The storage device 1080 is an auxiliary storage device implemented by using a hard disk, a solid state drive (SSD), a memory card, a read-only memory (ROM), or the like.

[0040] The input / output interface 1100 is an interface that connects the computer 1000 and an input / output device. For example, an input device such as a keyboard and an output device such as a display device are connected to the input / output interface 1100.

[0041] The network interface 1120 is an interface that connects the computer 1000 to a network. The network may be a local area network (LAN) or a wide area network (WAN).

[0042] The storage device 1080 stores a program (a program for realizing the above-mentioned application) for realizing each functional configuration unit of the state recognition device 2000. The processor 1040 realizes each functional configuration unit of the state recognition device 2000 by reading and executing the program in the memory 1060.

[0043] The state recognition device 2000 may be implemented by one computer 1000, or may be implemented by a plurality of computers 1000. In the latter case, the configurations of the computers 1000 need not be the same, and may be different from each other.

[0044] <Processing Flow>

[0045] Figure 4 2 is a flowchart showing a process flow performed by the state recognition device 2000 in the exemplary embodiment. The detection unit 2020 detects the operation of the reflector 20 by the driver 30 (S102). The determination unit 2040 determines the head position 40 at the time point when the driver 30 operates the reflector 20 (S104). The setting unit 2060 sets the determined head position 40 as the reference head position (S106).

[0046] <Operation Detection of Reflecting Mirror 20: S102>

[0047] The detection unit 2020 detects the operation of the reflector 20 by the driver 30 ( S102 ). Hereinafter, some specific methods will be described by way of example.

[0048] <Example 1 of Detection Method>

[0049] For example, the detection unit 2020 detects a specific operation performed on the operation interface of the reflector 20 provided in the vehicle 10, thereby detecting the operation of the reflector 20. In this case, the detection unit 2020 determines the time point when the specific operation is performed as the time point when the driver 30 operates the reflector 20.

[0050] The operation interface of the reflector 20 may be a mechanical interface or a software interface. In the former case, the operation interface of the reflector 20 is a mechanical switch (button, lever, etc.), etc. In this case, the detection unit 2020 detects that the mirror 20 is operated by detecting a specific operation performed on a specific mechanical switch.

[0051] In the case where the operation interface of the reflective mirror 20 is a software interface, the operation interface is, for example, a software interface (buttons, sliders, etc.) displayed on a touch panel or the like provided in the vehicle 10. In this case, the detection unit 2020 detects that the mirror 20 is operated by detecting a specific operation performed on a specific software interface.

[0052] Note that a well-known technique can be used as a technique for detecting a specific operation performed on a specific mechanical interface or software interface.

[0053] <Example 2 of Detection Method>

[0054] For example, the detection unit 2020 analyzes time series data including captured images of the reflector 20 and the driver 30 (hereinafter referred to as an image sequence), thereby detecting the operation of the reflector 20. In this case, the vehicle 10 is provided with a camera (hereinafter referred to as a first camera) whose image capturing range includes the reflector 20 and its vicinity. The first camera generates an image sequence by repeatedly performing image capturing. The image sequence is referred to as a first image sequence below.

[0055] For example, the detection unit 2020 determines whether the action of moving the reflector 20 is included for each captured image included in the first image sequence. If it is determined that the action of moving the reflector 20 is included in a certain captured image, the detection unit 2020 detects that the reflector 20 is operated by the driver 30. In this case, the detection unit 2020 determines the generation time point of the captured image as the time point when the driver 30 operates the reflector 20.

[0056] In order to achieve the above determination, for example, there is a predetermined definition for determining the condition that "the action of moving the reflector 20 by hand is included in the captured image" (hereinafter referred to as the first predetermined condition). In this case, the detection unit 2020 determines whether each captured image satisfies the first predetermined condition. If a specific captured image satisfies the first predetermined condition, the detection unit 2020 determines that the captured image includes the action of moving the reflector 20 by hand. As the first predetermined condition, for example, the condition of "the driver 30 touches the reflector 20" or "the driver 30 holds the reflector 20" can be adopted.

[0057] For example, in a case where the first predetermined condition is satisfied in one captured image of the first image sequence, the detection unit 2020 determines that the driver 30 operates the reflector 20. In another example, in a case where each of a plurality of captured images included within a predetermined time in the first image sequence satisfies the first predetermined condition, the detection unit 2020 determines that the driver 30 operates the reflector 20. In the latter case, the detection unit 2020 may determine that the driver 30 operates the reflector 20 if each of all captured images included within the predetermined time satisfies the first predetermined condition, or may determine that the driver 30 operates the reflector 20 if the first predetermined condition is satisfied in each of a predetermined ratio or more or a predetermined number or more of captured images included within the predetermined time.

[0058] The specific method for determining whether the above-mentioned first predetermined condition is satisfied is arbitrary. For example, the detection unit 2020 performs object recognition processing on the captured image to detect an image area representing the reflector 20 (hereinafter referred to as the reflector area) and an image area representing the hand of the driver 30 (hereinafter referred to as the hand area). If the reflector area and the hand area are in contact with each other or if the reflector area and the hand area overlap with each other, the detection unit 2020 determines that the first predetermined condition is satisfied.

[0059] The first predetermined condition may be a condition related to the positional relationship between the joint points of the hand of the driver 30 and the reflector 20. In this case, the detection unit 2020 detects the joint points of the hand of the driver 30 from the captured image, and determines whether the positional relationship between the joint points of the hand of the driver 30 and the reflector 20 satisfies the first predetermined condition, thereby detecting the operation of the reflector 20 by the driver 30. As the joint points of the hand of the driver 30, for example, the joint points of the corresponding fingers of the hand and the joint points of the wrist are detected.

[0060] More specifically, as the first predetermined condition, the condition of "the distance between the joint points of the driver's 30 hand and the reflector area is equal to or less than a threshold value" can be adopted. In this case, the detection unit 2020 detects the joint points of the driver's 30 hand and the reflector area for each captured image included in the first image sequence. The detection unit 2020 further determines whether the distance between the joint points of the driver's 30 hand and the reflector area is less than or equal to a threshold value. If it is determined that the distance between the joint points of the driver's 30 hand and the reflector area is equal to or less than a threshold value of a certain captured image, the detection unit 2020 determines that the driver 30 operates the reflector 20 at the time point when the captured image is generated.

[0061] Note that if multiple joints of the driver's hand 30 are detected, the detection unit 2020 calculates the distance to the reflector area for each of the multiple joints. For example, when at least one of the multiple calculated distances is less than or equal to a threshold, the detection unit 2020 determines that the driver 30 operates the reflector 20.

[0062] <Example 3 of Detection Method>

[0063] For example, the detection unit 2020 analyzes the operation interface of the reflector 20 and an image sequence (hereinafter referred to as the second image sequence) including the driver 30, thereby detecting that the reflector 20 is operated. In this case, a camera (hereinafter referred to as the second camera) is provided in the vehicle 10, and the image capture range of the camera includes the operation interface of the reflector 20 and its vicinity. The detection unit 2020 acquires and analyzes the image sequence generated by the second camera.

[0064] For example, the detection unit 2020 determines whether each captured image included in the second image sequence includes an action of operating the operating interface of the reflector 20. If a certain captured image includes an action of operating the operating interface of the reflector 20, the detection unit 2020 determines that the reflector 20 is operated at the generation time point of the captured image.

[0065] Here, as a method for determining whether "an action of operating the operating interface of the reflector 20 is included", a method similar to the above-mentioned method for determining whether "an action of moving the reflector 20 by hand is included" can be adopted. Specifically, for each captured image included in the second image sequence, a method for determining whether a condition such as "the driver 30 touches the operating interface of the reflector 20" or "the distance between the joint point of the hand of the driver 30 and the operating interface of the reflector 20 is equal to or less than a threshold value" (hereinafter referred to as the second predetermined condition) is satisfied can be adopted. If a captured image satisfying the second predetermined condition is detected, the detection unit 2020 determines that the driver 30 operates the reflector 20 at the generation time point of the captured image.

[0066] Here, if the second predetermined condition is satisfied in one captured image in the second image sequence, the detection unit 2020 may determine that the driver 30 operates the reflector 20, or if the second predetermined condition is satisfied in each of a plurality of captured images included within a predetermined time in the second image sequence, the detection unit 2020 may determine that the driver 30 operates the reflector 20. In addition, in the latter case, if the second predetermined condition is satisfied in each of all captured images included within the predetermined time, the detection unit 2020 may determine that the driver 30 operates the reflector 20, or if the second predetermined condition is satisfied in each of a predetermined ratio or more or a predetermined number or more of captured images included within the predetermined time, the detection unit 2020 may determine that the driver 30 operates the reflector 20.

[0067] <Determination of Head Position 40: S104>

[0068] The determination unit 2040 determines the head position 40 at the time point when the driver 30 operates the reflector 20 (S104). Here, various methods for determining the time point when the driver 30 operates the reflector 20 are as described above.

[0069] For example, the head position 40 is represented by the coordinates of one or more points. For example, in the case where the position of the head is represented by the coordinates of one point, the head position 40 is represented by the coordinates of a point representing a specific position of the head. The specific position of the head is, for example, the center of the head. In another example, the specific position of the head may be the center of a point representing each of a plurality of parts of the head (eyes, nose, etc.).

[0070] For example, in the case where the head position 40 is represented by coordinates of a plurality of points, the head position 40 is represented by a combination of coordinates of points representing each of a plurality of parts of the head. Figure 5 is a diagram showing a case where the head position 40 is represented by a combination of coordinates of a plurality of points. In this example, the head position 40 is represented by a combination of coordinates P1 of a right eye 51, coordinates P2 of a left eye 52, coordinates P3 of a nose 53, coordinates P4 of a right ear 54, and coordinates P5 of a left ear 55 included in the head 50. Note that the head position 40 may be represented by a combination of coordinates of any two or more of a plurality of specific parts.

[0071] For example, a captured image including the head of the driver 30 and the vicinity thereof is used to determine the head position 40. In this case, a camera (hereinafter referred to as a third camera) whose image capturing range includes the head of the driver 30 is provided in the vehicle 10.

[0072] For example, the determination unit 2040 determines a captured image generated at a time point closest to the time point at which the reflector 20 is operated from the captured images generated by the third camera. Then, the determination unit 2040 determines the head position 40 by analyzing the determined captured image. In another example, the determination unit 2040 may cause the third camera to perform image capture in response to detecting the operation of the mirror 20 by the driver 30, and analyze the captured image generated as a result of the image capture.

[0073] The method for determining the head position 40 by analyzing the captured image will be described below. For example, the determination unit 2040 detects an image region representing a human head (hereinafter referred to as a head region) from the captured image. Then, the determination unit 2040 determines the coordinates of a specific position of the head region (for example, the center position of the head region) as the head position 40.

[0074] In another example, the determination unit 2040 detects a specific part of a person, such as an eye or a nose, from the captured image. Then, the determination unit 2040 determines a combination of coordinates of each specific part as the head position 40. In addition, the determination unit 2040 may calculate the coordinates of a specific point, such as a center position, from the coordinates of a plurality of specific parts, and determine the calculated coordinates as the head position 40.

[0075] Here, the position of the head of the driver 30 can be represented by two-dimensional coordinates on the captured image, or can be represented by three-dimensional coordinates on the virtual three-dimensional space. In the latter case, the determination unit 2040 converts the two-dimensional coordinates on the captured image obtained from the third camera into three-dimensional coordinates in a predetermined virtual three-dimensional space. Here, a well-known technology can be used as a technology for converting the two-dimensional coordinates on the image into three-dimensional coordinates in a specific virtual three-dimensional space.

[0076] Here, a plurality of captured images obtained by capturing the head of the driver 30 from different directions may be used to calculate coordinates in a virtual three-dimensional space. In this case, the vehicle 10 is provided with a plurality of third cameras.

[0077] <Setting of reference head position: S106>

[0078] The setting unit 2060 sets the determined head position 40 as a reference head position (S106). For example, the setting unit 2060 sets the reference head position by generating information indicating the reference head position (hereinafter, reference information) and outputting the generated reference information in a prescribed manner. The reference information is output in a manner usable by a device (hereinafter, referred to as an application device) that uses the reference information. The application device may be provided inside the vehicle 10 or may be provided outside the vehicle 10. Note that the application device may be provided integrally with the state recognition device 2000 (i.e., the state recognition device 2000 may also have a function of operating as an application device), or may be provided separately from the state recognition device 2000.

[0079] The method for outputting the reference information is arbitrary. For example, the setting unit 2060 stores the reference information in a storage unit accessible from the application device. In another example, the setting unit 2060 sends the reference information to the application device.

[0080] The setting unit 2060 can determine the posture of the driver's 30 head by using the head position 40, and include the determined posture of the head in the reference information as the reference posture of the head (hereinafter referred to as the reference head posture). The posture of the head is represented by, for example, the rotation angle of each of the three-dimensional axes (pitch, roll, and yaw). In the case of calculating the posture of the head in this way, the head position 40 is represented by a combination of two or more coordinates. Here, a well-known technology can be used as a specific method for calculating the posture of an object based on the coordinates of multiple parts of the object.

[0081] The setting unit 2060 can determine the identification information of the driver 30 and include the determined identification information in the reference information. By including the identification information of the driver 30 in the reference information in this way, when the vehicle 10 is shared by multiple people, the reference head position can be set for each person.

[0082] The driver 30 is identified by using, for example, a feature value (hereinafter referred to as a facial feature value) obtained from an image of the face of the driver 30. For example, the setting unit 2060 uses the facial feature value of the driver 30 as identification information of the driver 30. In this case, at the time of generating the reference information, the setting unit 2060 acquires a captured image including the face of the driver 30. Then, the setting unit 2060 determines the facial feature value of the driver 30 by analyzing the acquired captured image, and includes the determined facial feature value in the reference information.

[0083] In another example, the setting unit 2060 may use an identification string pre-assigned to the driver 30 as the identification information of the driver 30. In this case, for each person who may drive the vehicle 10, the storage unit accessed from the setting unit 2060 may pre-store person information in which the person's identification string is associated with the facial feature value. At the time of generating the reference information, the setting unit 2060 determines the person information including the facial feature value matching the facial feature value of the driver 30. Then, the setting unit 2060 includes the identification string contained in the determined person information as the identification information of the driver 30 in the reference information.

[0084] Here, in order to be able to obtain a captured image including the face of the driver 30, it is assumed that the vehicle 10 is provided with a camera whose image capturing range includes the face of the driver 30. However, as a captured image including the face of the driver 30, a captured image generated by a third camera (a captured image including the head of the driver 30) may be used.

[0085] <Update of benchmark information>

[0086] In the case where the vehicle 10 is repeatedly used, the state recognition device 2000 may repeatedly generate the reference information. For example, each time the vehicle 10 is started, the state recognition device 2000 generates the reference information. In this case, the state recognition device 2000 updates the reference information output in the past by outputting the newly generated reference information. The application device uses the newly output reference information instead of the reference information output in the past.

[0087] Note that, in the case where reference information is generated for each person, the state recognition device 2000 updates the reference information generated in the past for the current driver 30 with the reference information newly generated for the driver 30 .

[0088] <Use of default header position>

[0089] In the case where the operation of the driver 30 on the reflector 20 is not detected, the state recognition device 2000 can generate reference information in which the default head position is indicated as the reference head position. For example, the state recognition device 2000 detects the operation of the driver 30 on the reflector 20 within a predetermined period. If the reflector operation of the driver 30 on the reflector 20 is detected within the predetermined period, the state recognition device 2000 sets the head position 40 determined by the above-mentioned various methods as the reference head position. On the other hand, if the reflector operation of the driver 30 on the reflector 20 is not detected within the predetermined period, the state recognition device 2000 sets the default head position as the reference head position.

[0090] The predetermined time period is, for example, a time period of a predetermined length of time starting from the start time point of the vehicle 10. As the start time point of the vehicle 10, the time point when the engine of the vehicle 10 is started, the time point when the power supply of the vehicle 10 is started, etc. can be adopted. In another example, the predetermined time period can be a period from the start time point of the vehicle 10 to the time point when the vehicle 10 starts to move.

[0091] For example, the reference head position indicated by the latest reference information is used as the default head position. As a result, the same position as the reference head position at the previous ride is set as the reference head position. In another example, the head position manually set in advance by the administrator can be used as the default head position.

[0092] In the case where the reference head position is set for each person, the state recognition device 2000 uses, as the default head position, the reference head position indicated in the reference information most recently generated for the driver 30. For example, if the reference information has not been generated for the driver 30 in the past, the state recognition device 2000 uses the default head position manually set by the administrator.

[0093] Note that when the reference information includes a reference head pose, similar to the reference head position, a default value is also used for the reference head pose.

[0094] <Examples of application equipment>

[0095] An application device is a device that uses the reference information to perform a predetermined task. Figure 6 is a diagram showing a task execution system 4000 including a state recognition device 2000 and an application device 3000 .

[0096] The application device 3000 includes an acquisition unit 3020, a task execution unit 3040, and an output unit 3060. The acquisition unit 3020 acquires the reference information 60 output by the state recognition device 2000. The task execution unit 3040 executes the task by using the reference information 60. The output unit 3060 outputs output information indicating the execution result of the task.

[0097] The task performed by the task execution unit 3040 is arbitrary. For example, the task performed by the task execution unit 3040 is monitoring the state of the driver 30. In this case, for example, the task execution unit 3040 periodically compares the current head position of the driver 30 with the reference head position indicated by the reference information 60. More specifically, the task execution unit 3040 determines whether the degree of deviation between the current head position of the driver 30 and the reference head position satisfies a predetermined condition. When the degree of deviation between the current head position of the driver 30 and the reference head position satisfies a predetermined condition, the task execution unit 3040 determines that the state of the driver 30 is abnormal.

[0098] When the reference head posture is further included in the reference information 60, the task execution unit 3040 also compares the current head posture of the driver 30 with the reference head posture. That is, the task execution unit 3040 determines whether the degree of deviation between the current head position of the driver 30 and the reference head position and the degree of deviation between the current head posture of the driver 30 and the reference head posture satisfy predetermined conditions.

[0099] The degree of deviation between the current head position and the reference head position is represented by, for example, the distance between the current head position and the reference head position or each of the differences in the x-coordinate, y-coordinate, and z-coordinate between the current head position and the reference head position. In addition, the degree of deviation between the current head posture and the reference head posture is represented by a yaw difference, a pitch difference, a roll difference, etc.

[0100] When it is determined that the state of the driver 30 is abnormal, the output unit 3060 outputs output information notifying the abnormality of the state of the driver 30. For example, the output information is information indicating notification to the driver 30. In this case, the output information is a predetermined message displayed on a display device provided in the vehicle 10 or a predetermined voice output from a speaker provided in the vehicle 10. In another example, the output information may be information indicating notification to the outside of the vehicle 10, such as an emergency call.

[0101] Similar to the hardware configuration of the state recognition device 2000, the hardware configuration of the application device 3000 is, for example, Figure 3 However, the storage device of the application device 3000 stores a program for realizing each functional configuration unit of the application device 3000.

[0102] Figure 7 is a flowchart showing the flow of processing performed by the application device 3000. The acquisition unit 3020 acquires the reference information 60 (S202). The task execution unit 3040 determines whether the degree of deviation between the current head position of the driver 30 and the reference head position satisfies a predetermined condition (S204). When the degree of deviation does not satisfy the predetermined condition (S204: No), the task execution unit 3040 executes S204 again. Note that a predetermined standby time may be provided until S204 is executed next time.

[0103] When the degree of deviation between the current head position of the driver 30 and the reference head position satisfies a predetermined condition (S204: Yes), the output unit 3060 outputs output information (S206). As described above, in the case where the reference information includes a reference head posture, in addition to the comparison between the current head position of the driver 30 and the reference head position, the task execution unit 3040 also compares the current head posture of the driver 30 with the reference head posture.

[0104] The output information output by the output unit 3060 is not limited to a notification indicating that the state of the driver 30 is abnormal. For example, if the degree of deviation between the current head position of the driver 30 and the reference head position satisfies a predetermined condition (i.e., if the state of the driver 30 is abnormal), the output unit 3060 may output, as output information, a control signal for safely stopping the vehicle 10. Thus, for example, if the state of the driver 30 is abnormal, control such as stopping the vehicle 10 on the shoulder of the road may be performed.

[0105] Although the present disclosure has been specifically shown and described with reference to the exemplary embodiments of the present disclosure, the present disclosure is not limited to these exemplary embodiments. It will be understood by those skilled in the art that various changes may be made in form and detail without departing from the spirit and scope of the present disclosure as defined by the claims. And each embodiment may be appropriately combined with at least one embodiment.

[0106] In the above examples, the program includes instructions (or software codes) that, when loaded into a computer, cause the computer to perform one or more functions described in the embodiments. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. As an example and not limitation, a non-transitory computer-readable medium or a tangible storage medium may include a random access memory (RAM), a read-only memory (ROM), a flash memory, a solid-state drive (SSD) or other types of memory technology, a CD-ROM, a digital versatile disk (DVD), a Blu-ray disc or other types of optical disk storage, and a cassette, a magnetic tape, a disk storage or other types of magnetic storage devices. The program may be transmitted on a temporary computer-readable medium or a communication medium. As an example and not limitation, a temporary computer-readable medium or a communication medium may include electrical, optical, acoustic or other forms of propagated signals.

[0107] Some or all of the above-described example embodiments may be described in the following supplementary notes, but are not limited to the following supplementary notes.

[0108] (Supplementary Note 1)

[0109] A state recognition device, comprising:

[0110] A detection device for detecting that a reflector of a vehicle is operated;

[0111] means for determining the head position of the driver of the vehicle at the time of performing the operation; and

[0112] Setting means for setting the determined head position as a reference position of the head in the vehicle.

[0113] (Supplementary Note 2)

[0114] According to the state recognition device of Supplementary Note 1,

[0115] The detection device detects an action of operating the reflector from a first captured image including the reflector to detect that the reflector is operated.

[0116] (Supplementary Note 3)

[0117] According to the state recognition device of Supplementary Note 2,

[0118] The action of operating the reflector is an action of manually moving the reflector or an action of operating an operation interface of the reflector.

[0119] (Supplementary Note 4)

[0120] The state recognition device according to any one of Supplementary Notes 1 to 3,

[0121] The determining device acquires a second captured image including the driver's head, and determines the head position of the driver based on coordinates of one or more points related to the head included in the second captured image.

[0122] (Supplementary Note 5)

[0123] According to the state recognition device of Supplementary Note 1,

[0124] Wherein, the setting device outputs reference information including the determined head position.

[0125] (Supplementary Note 6)

[0126] According to the state recognition device of Supplementary Note 5,

[0127] wherein the determining means determines the head position of the driver based on a combination of coordinates of each of a plurality of parts of the head, and

[0128] The setting device determines the posture of the head based on the combination of the coordinates, and outputs the reference information including the determined posture of the head.

[0129] (Supplementary Note 7)

[0130] According to the state identification device of Supplementary Note 5 or 6,

[0131] The setting device determines identification information of the driver and outputs the reference information including the determined identification information.

[0132] (Supplementary Note 8)

[0133] According to the state recognition device of Supplementary Note 5,

[0134] Wherein, the setting device performs:

[0135] outputting the reference information including the head position of the driver if the head position of the driver is determined within a predetermined period of time; and

[0136] If the head position of the driver is not determined within the predetermined period of time, the reference information including a predetermined head position is output.

[0137] (Supplementary Note 9)

[0138] According to the state recognition device of Supplementary Note 8,

[0139] The predetermined head position is a head position included in the reference information output in the past.

[0140] (Supplementary Note 10)

[0141] A state recognition method executed by a computer, comprising:

[0142] A detection step of detecting that a reflector of a vehicle is operated;

[0143] a determining step of determining a head position of a driver of the vehicle at a point in time when the operation is performed; and

[0144] A setting step of setting the determined head position as a reference position of the head in the vehicle.

[0145] (Supplementary Note 11)

[0146] According to the state recognition method of Supplementary Note 10,

[0147] The detecting step detects an action of operating the reflecting mirror from a first captured image including the reflecting mirror to detect that the reflecting mirror is operated.

[0148] (Supplementary Note 12)

[0149] According to the state identification method of Supplementary Note 11,

[0150] The action of operating the reflector is an action of manually moving the reflector or an action of operating an operation interface of the reflector.

[0151] (Supplementary Note 13)

[0152] The state recognition method according to any one of Supplementary Notes 10 to 12,

[0153] The determining step acquires a second captured image including the driver's head, and determines the head position of the driver based on coordinates of one or more points related to the head included in the second captured image.

[0154] (Supplementary Note 14)

[0155] According to the state recognition method of Supplementary Note 10,

[0156] Wherein, the setting step outputs reference information including the determined head position.

[0157] (Supplementary Note 15)

[0158] According to the state recognition method of Supplementary Note 14,

[0159] wherein the determining step determines the head position of the driver based on a combination of coordinates of each of a plurality of parts of the head, and

[0160] wherein the setting step determines a posture of the head based on the combination of the coordinates, and outputs the reference information including the determined posture of the head.

[0161] (Supplementary Note 16)

[0162] According to the state identification method of Supplementary Note 14 or 15,

[0163] The setting step determines identification information of the driver, and outputs the reference information including the determined identification information.

[0164] (Supplementary Note 17)

[0165] According to the state recognition method of Supplementary Note 14,

[0166] Wherein, the setting steps include:

[0167] outputting the reference information including the head position of the driver if the head position of the driver is determined within a predetermined period of time; and

[0168] If the head position of the driver is not determined within the predetermined period of time, the reference information including a predetermined head position is output.

[0169] (Supplementary Note 18)

[0170] According to the state recognition method of Supplementary Note 17,

[0171] The predetermined head position is a head position included in the reference information output in the past.

[0172] (Supplementary Note 19)

[0173] A non-transitory computer-readable medium storing a program that causes a computer to execute:

[0174] A detection step of detecting that a reflector of a vehicle is operated;

[0175] a determining step of determining a head position of a driver of the vehicle at a point in time when the operation is performed; and

[0176] A setting step of setting the determined head position as a reference position of the head in the vehicle.

[0177] (Supplementary Note 20)

[0178] The computer readable medium according to Supplementary Note 19,

[0179] The detecting step detects an action of operating the reflecting mirror from a first captured image including the reflecting mirror to detect that the reflecting mirror is operated.

[0180] (Supplementary Note 21)

[0181] The computer readable medium according to Supplementary Note 20,

[0182] The action of operating the reflector is an action of manually moving the reflector or an action of operating an operation interface of the reflector.

[0183] (Supplementary Note 22)

[0184] The computer-readable medium according to any one of Supplementary Notes 19 to 21,

[0185] The determining step acquires a second captured image including the driver's head, and determines the head position of the driver based on coordinates of one or more points related to the head included in the second captured image.

[0186] (Supplementary Note 23)

[0187] The computer readable medium according to Supplementary Note 19,

[0188] Wherein, the setting step outputs reference information including the determined head position.

[0189] (Supplementary Note 24)

[0190] The computer readable medium according to Supplementary Note 23,

[0191] wherein the determining step determines the head position of the driver based on a combination of coordinates of each of a plurality of parts of the head, and

[0192] wherein the setting step determines a posture of the head based on the combination of the coordinates, and outputs the reference information including the determined posture of the head being output.

[0193] (Supplementary Note 25)

[0194] The computer readable medium according to Supplementary Note 23 or 24,

[0195] The setting step determines identification information of the driver, and outputs the reference information including the determined identification information.

[0196] (Supplementary Note 26)

[0197] The computer readable medium according to Supplementary Note 23,

[0198] The setting steps include:

[0199] outputting the reference information including the head position of the driver if the head position of the driver is determined within a predetermined period of time; and

[0200] If the head position of the driver is not determined within the predetermined period of time, the reference information including a predetermined head position is output.

[0201] (Supplementary Note 27)

[0202] The computer readable medium according to Supplementary Note 26,

[0203] The predetermined head position is a head position included in the reference information output in the past.

[0204] (Supplementary Note 28)

[0205] A task execution system, comprising:

[0206] State recognition equipment and application equipment,

[0207] Wherein, the state recognition device comprises:

[0208] A detection device, the detection device is used to detect that a reflector of the vehicle is operated;

[0209] a determining device for determining a head position of a driver of the vehicle at a point in time when the operation is performed; and

[0210] a setting device for outputting a reference position including the determined head position, and

[0211] Wherein, the application equipment includes:

[0212] An acquisition device, the acquisition device is used to acquire the reference information;

[0213] A task execution device for executing a task by using the reference information; and

[0214] An output device is used to output output information about the execution result of the task.

[0215] (Supplementary Note 29)

[0216] According to the task execution system in Supplementary Note 28,

[0217] wherein the task execution device determines whether the degree of deviation between the current head position of the driver and the head position included in the reference information satisfies a predetermined condition, and

[0218] When the degree of deviation satisfies the predetermined condition, the output device generates the output information notifying the driver of an abnormal state.

[0219] (Supplementary Note 30)

[0220] A task execution method, comprising:

[0221] State recognition device execution:

[0222] A detection step of detecting whether a reflector of a vehicle has been operated;

[0223] a determining step of determining a head position of a driver of the vehicle at a point in time when the operation is performed; and

[0224] outputting a setting step of a reference position including the determined head position; and

[0225] Application execution:

[0226] A step of obtaining the benchmark information;

[0227] A task execution step of executing the task by using the reference information; and

[0228] An outputting step of outputting output information on the execution result of the task.

[0229] (Supplementary Note 31)

[0230] According to the task execution method of Supplementary Note 30,

[0231] wherein the task execution step determines whether the degree of deviation between the current head position of the driver and the head position included in the reference information satisfies a predetermined condition, and

[0232] If the degree of deviation satisfies the predetermined condition, the output step generates the output information notifying the driver of an abnormal state.

[0233] Reference numerals list

[0234] 10 Vehicles

[0235] 20 Reflector

[0236] 30 Driver

[0237] 40 Head Position

[0238] 50 Head

[0239] 51 Right Eye

[0240] 52 Left Eye

[0241] 53 Nose

[0242] 54 Right Ear

[0243] 55 left ear

[0244] 60 Benchmark Information

[0245] 1000 Computer

[0246] 1020 Bus

[0247] 1040 processor

[0248] 1060 Memory

[0249] 1080 Storage Devices

[0250] 1100 Input / Output Interface

[0251] 1120 Network Interface

[0252] 2000 Status Identification Device

[0253] 2020 Detection Unit

[0254] 2040 Determine Unit

[0255] 2060 Setting Unit

[0256] 3000 Application Equipment

[0257] 3020 Get Unit

[0258] 3040 Task Execution Unit

[0259] 3060 Output Unit

[0260] 4000 Mission Execution System

Claims

1. A state recognition device, comprising: A detection device, the detection device is used to detect that a reflector of the vehicle is operated; a determining device for determining a head position of a driver of the vehicle at a point in time when the operation is performed; as well as A setting device is used to set the determined head position as a reference position of the head in the vehicle.

2. The state recognition device according to claim 1, in, The detection device detects an action of operating the reflecting mirror from a first captured image including the reflecting mirror to detect that the reflecting mirror is operated.

3. The state recognition device according to claim 2, in, The action of operating the reflector is an action of manually moving the reflector or an action of operating an operation interface of the reflector.

4. The state recognition device according to any one of claims 1 to 3, in, The determination device acquires a second captured image including the driver's head, and determines the position of the driver's head based on coordinates of one or more points related to the head included in the second captured image.

5. The state recognition device according to claim 1, in, The setting device outputs reference information including the determined head position.

6. The state recognition device according to claim 5, in, The determining means determines the head position of the driver based on a combination of coordinates of each of a plurality of parts of the head, and The setting device determines the posture of the head based on the combination of the coordinates, and outputs the reference information including the determined posture of the head.

7. The state recognition device according to claim 5 or 6, in, The setting device determines identification information of the driver, and outputs the reference information including the determined identification information.

8. The state recognition device according to claim 5, in, The setting device performs: outputting the reference information including the head position of the driver if the head position of the driver is determined within a predetermined period of time; as well as If the head position of the driver is not determined within the predetermined period of time, the reference information including a predetermined head position is output.

9. The state recognition device according to claim 8, in, The predetermined head position is a head position included in the reference information output in the past.

10. A state recognition method executed by a computer, comprising: A detection step, the detection step is used to detect that a reflector of the vehicle is operated; a determining step for determining a head position of a driver of the vehicle at a point in time when the operation is performed; as well as A setting step for setting the determined head position as a reference position of the head in the vehicle.

11. The state recognition method according to claim 10, in, The detecting step detects an action of operating the reflecting mirror from a first captured image including the reflecting mirror to detect that the reflecting mirror is operated.

12. The state recognition method according to claim 11, in, The action of operating the reflector is an action of manually moving the reflector or an action of operating an operation interface of the reflector.

13. The state recognition method according to any one of claims 10 to 12, in, The determining step acquires a second captured image including a head of the driver, and determines the position of the head of the driver based on coordinates of one or more points related to the head included in the second captured image.

14. The state recognition method according to claim 10, in, The setting step outputs reference information including the determined head position.

15. The state recognition method according to claim 14, in, The determining step determines the head position of the driver based on a combination of coordinates of each of a plurality of parts of the head, and wherein the setting step determines a posture of the head based on the combination of the coordinates, and outputs the reference information including the determined posture of the head.

16. The state recognition method according to claim 14 or 15, in, The setting step determines identification information of the driver, and outputs the reference information including the determined identification information.

17. The state recognition method according to claim 14, in, The setting steps include: outputting the reference information including the head position of the driver if the head position of the driver is determined within a predetermined period of time; and If the head position of the driver is not determined within the predetermined period of time, the reference information including a predetermined head position is output.

18. The state recognition method according to claim 17, in, The predetermined head position is a head position included in the reference information output in the past.

19. A non-transitory computer-readable medium storing a program, the program causing a computer to execute: A detection step, the detection step is used to detect that a reflector of the vehicle is operated; a determining step for determining a head position of a driver of the vehicle at a point in time when the operation is performed; as well as A setting step for setting the determined head position as a reference position of the head in the vehicle.

20. The computer readable medium of claim 19, in, The detecting step detects an action of operating the reflecting mirror from a first captured image including the reflecting mirror to detect that the reflecting mirror is operated.

21. The computer readable medium of claim 20, in, The action of operating the reflector is an action of manually moving the reflector or an action of operating an operation interface of the reflector.

22. The computer readable medium according to any one of claims 19 to 21, in, The determining step acquires a second captured image including a head of the driver, and determines the position of the head of the driver based on coordinates of one or more points related to the head included in the second captured image.

23. The computer readable medium of claim 19, in, The setting step outputs reference information including the determined head position.

24. The computer readable medium of claim 23, in, The determining step determines the head position of the driver based on a combination of coordinates of each of a plurality of parts of the head, and wherein the setting step determines a posture of the head based on the combination of the coordinates, and outputs the reference information including the determined posture of the head being output.

25. The computer readable medium according to claim 23 or 24, in, The setting step determines identification information of the driver, and outputs the reference information including the determined identification information.

26. The computer readable medium of claim 23, in, The setting steps include: outputting the reference information including the head position of the driver if the head position of the driver is determined within a predetermined period of time; and If the head position of the driver is not determined within the predetermined period of time, the reference information including a predetermined head position is output.

27. The computer readable medium of claim 26, in, The predetermined head position is a head position included in the reference information output in the past.

28. A task execution system, comprising: State recognition equipment and application equipment, Wherein, the state recognition device comprises: A detection device, the detection device is used to detect that a reflector of the vehicle is operated; a determining device for determining a head position of a driver of the vehicle at a point in time when the operation is performed; and a setting device for outputting a reference position including the determined head position, and Wherein, the application equipment includes: An acquisition device, the acquisition device is used to acquire the reference information; A task execution device for executing a task by using the reference information; and An output device is used to output output information about the execution result of the task.

29. The task execution system according to claim 28, in, The task execution device determines whether the degree of deviation between the current head position of the driver and the head position included in the reference information satisfies a predetermined condition, and When the degree of deviation satisfies the predetermined condition, the output device generates the output information notifying that the driver's state is abnormal.

30. A task execution method, comprising: State recognition device execution: A detection step, the detection step is used to detect that a reflector of the vehicle is operated; a determining step for determining a head position of a driver of the vehicle at a point in time when the operation is performed; as well as a setting step for outputting a reference position including the determined head position; as well as Application execution: An acquisition step, wherein the acquisition step is used to acquire the reference information; A task execution step for executing a task by using the reference information; and An output step is used to output output information about the execution result of the task.

31. The task execution method according to claim 30, in, The task execution step determines whether the degree of deviation between the current head position of the driver and the head position included in the reference information satisfies a predetermined condition, and If the degree of deviation satisfies the predetermined condition, the output step generates the output information notifying the driver that the state is abnormal.

Images