State recognition device, state recognition method, program, and task execution system

Abstract

A state recognition device (2000) detects that a mirror (20) of a vehicle (10) has been adjusted. The state recognition device (2000) specifies the head position of a driver (30) of the vehicle (10) at the time point when the mirror (20) was adjusted. The state recognition device (2000) sets the specified head position as a reference head position in the vehicle (10).

Description

State recognition device, state recognition method, computer-readable medium, task execution system, and task execution method

[0001] The present disclosure relates to a state recognition device, a state recognition method, a computer-readable medium, a task execution system, and a task execution method.

[0002] Technologies for detecting the state of a driver in a vehicle such as an automobile have been developed. For example, Patent Document 1 discloses a technology for detecting whether the driver is looking away by using the direction of the driver's face when adjusting the outer mirror as a reference value.

[0003] JP 2009-009244 A

[0004] In Patent Document 1, the direction of the driver's face is used as a criterion for the driver's state. An object of the present disclosure is to disclose a more efficient technique for determining a criterion for the driver's state.

[0005] The state recognition device disclosed herein includes a detection means for detecting that a vehicle mirror has been operated, an identification means for identifying the head position of the driver of the vehicle at the time the operation was performed, and a setting means for setting the identified head position as a reference head position in the vehicle.

[0006] The state recognition method according to the present disclosure is executed by a computer and includes a detecting step of detecting that a mirror of a vehicle has been operated, a specifying step of specifying the head position of the driver of the vehicle at the time the mirror was operated, and a setting step of setting the specified head position as a reference head position for the vehicle.

[0007] The program of the present disclosure causes a computer to execute the state recognition method of the present disclosure.

[0008] The present disclosure provides a more efficient technique for determining criteria related to a driver's condition.

[0009] FIG. 1 is a diagram illustrating an example of an outline of the operation of a state recognition device according to an embodiment. FIG. 2 is a block diagram illustrating an example of the functional configuration of a state recognition device. FIG. 3 is a block diagram illustrating an example of the hardware configuration of a computer that realizes the state recognition device. FIG. 4 is a flowchart illustrating an example of the flow of processing executed by the state recognition device. FIG. 5 is a diagram illustrating an example of a case in which a head position is represented by a combination of coordinates of multiple points. FIG. 6 is a block diagram illustrating an example of the functional configuration of a system including a state recognition device and an application device. FIG. 7 is a flowchart illustrating an example of the flow of processing executed by the application device.

[0010] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In each drawing, the same or corresponding elements are designated by the same reference numerals, and duplicate explanations will be omitted as necessary for clarity. Furthermore, unless otherwise specified, predetermined values ​​such as predetermined values ​​and threshold values ​​are stored in advance in a storage device accessible from a device that uses the values. Furthermore, unless otherwise specified, the storage unit is composed of one or any number of storage devices.

[0011] <Overview> Fig. 1 is a diagram illustrating an overview of a state recognition device 2000 according to an embodiment. Fig. 1 is a diagram for facilitating understanding of the overview of the state recognition device 2000, and the operation of the state recognition device 2000 is not limited to that shown in Fig. 1.

[0012] The state recognition device 2000 sets the position of the driver 30 at the time when the mirror 20 provided on the vehicle 10 is operated as a reference head position in the vehicle 10 (hereinafter referred to as the reference head position). The vehicle 10 is, for example, an automobile. The mirror 20 is a mirror used by the driver 30 when driving, such as a rearview mirror or a side mirror.

[0013] Note that only one type of mirror may be treated as the mirror 20, or multiple types of mirrors may be treated as the mirror 20. In the latter case, for example, the state recognition device 2000 determines that the mirror 20 has been operated by the driver 30 when either the rearview mirror or one of the two side mirrors is operated.

[0014] For example, the state recognition device 2000 operates as follows: First, the state recognition device 2000 detects the operation of the mirror 20 by the driver 30. Then, the state recognition device 2000 identifies the head position of the driver 30 (head position 40) at the time when the driver 30 operates the mirror 20. Then, the state recognition device 2000 sets the identified head position 40 as the reference head position for the vehicle 10.

[0015] 1, the head position 40 is represented by a single point, but the head position 40 may be represented by a combination of multiple points. Details regarding how the head position 40 is represented will be described later.

[0016] <Example of Effects> According to the state recognition device 2000, in response to detection of an operation of the mirror 20 by the driver 30, the head position of the driver 30 at the time of the operation is set as a reference head position. In this way, the state recognition device 2000 provides a new technique for determining a reference state of the vehicle driver.

[0017] Here, when the mirror 20 is being operated, the head position of the driver 30 is considered to be close to the head position when the driver 30 is looking ahead of the vehicle 10. Therefore, the state recognition device 2000 can easily set the head position of the driver 30 in a state close to the state where the driver 30 is looking ahead as the reference head position.

[0018] The state recognition device 2000 of this embodiment will be described in more detail below.

[0019] 2 is a block diagram illustrating the functional configuration of a state recognition device 2000 according to an embodiment. The state recognition device 2000 includes a detection unit 2020, an identification unit 2040, and a setting unit 2060. The detection unit 2020 detects operation of the mirror 20 by the driver 30. The identification unit 2040 identifies the head position 40 at the time the mirror 20 is operated by the driver 30. The setting unit 2060 sets the identified head position 40 as a reference head position for the vehicle 10.

[0020] <Example of Hardware Configuration> Each functional component of the state recognition device 2000 may be realized by hardware that realizes the functional component (e.g., a hardwired electronic circuit, etc.), or may be realized by a combination of hardware and software (e.g., a combination of an electronic circuit and a program that controls it, etc.). Below, a case where each functional component of the state recognition device 2000 is realized by a combination of hardware and software will be further described.

[0021] 3 is a block diagram illustrating an example of the hardware configuration of a computer 1000 that realizes the state recognition device 2000. The computer 1000 is any computer. For example, the computer 1000 is an ECU (Electronic Control Unit), a navigation device, or a tablet terminal that is provided inside the vehicle 10. Alternatively, the computer 1000 may be a PC (Personal Computer) or a server machine that is provided outside the vehicle 10. The computer 1000 may be a dedicated computer designed to realize the state recognition device 2000, or may be a general-purpose computer.

[0022] For example, by installing a predetermined application on the computer 1000, each function of the state recognition device 2000 is realized on the computer 1000. The application is configured as a program for realizing each functional component of the state recognition device 2000. Note that any method for acquiring the program is possible. For example, the program can be acquired from a storage medium (such as a DVD disk or USB memory) on which the program is stored. Alternatively, the program can be acquired by downloading it from a server device that manages the storage device on which the program is stored.

[0023] The computer 1000 has a bus 1020, a processor 1040, a memory 1060, a storage device 1080, an input / output interface 1100, and a network interface 1120. The bus 1020 is a data transmission path for the processor 1040, the memory 1060, the storage device 1080, the input / output interface 1100, and the network interface 1120 to transmit and receive data to and from each other. However, the method of connecting the processor 1040 and the like to each other is not limited to bus connection.

[0024] The processor 1040 is one of various processors, such as a central processing unit (CPU), a graphics processing unit (GPU), a field-programmable gate array (FPGA), or a digital signal processor (DSP). The memory 1060 is a main storage device realized using a random access memory (RAM) or the like. The storage device 1080 is an auxiliary storage device realized using a hard disk, a solid state drive (SSD), a memory card, a read only memory (ROM), or the like.

[0025] The input / output interface 1100 is an interface for connecting the computer 1000 to an input / output device. For example, the input / output interface 1100 is connected to an input device such as a keyboard and an output device such as a display device.

[0026] The network interface 1120 is an interface for connecting the computer 1000 to a network. This network may be a LAN (Local Area Network) or a WAN (Wide Area Network).

[0027] The storage device 1080 stores programs (programs that realize the above-mentioned applications) that realize the functional components of the state recognition device 2000. The processor 1040 reads these programs into the memory 1060 and executes them to realize the functional components of the state recognition device 2000.

[0028] The state recognition device 2000 may be realized by one computer 1000 or by multiple computers 1000. In the latter case, the configurations of the computers 1000 do not need to be the same, and can be different from each other.

[0029] <Processing Flow> Figure 4 is a flowchart illustrating the processing flow executed by the state recognition device 2000 according to this embodiment. The detection unit 2020 detects the operation of the mirror 20 by the driver 30 (S102). The identification unit 2040 identifies the head position 40 at the time the mirror 20 is operated by the driver 30 (S104). The setting unit 2060 sets the identified head position 40 as the reference head position (S106).

[0030] <Detection of Operation of Mirror 20: S102> The detection unit 2020 detects the operation of the mirror 20 by the driver 30 (S102). Some specific detection methods will be exemplified below.

[0031] <<Detection Method Example 1>> For example, the detection unit 2020 detects an operation of the mirror 20 by detecting that a specific operation has been performed on an operation interface of the mirror 20 provided in the vehicle 10. In this case, the detection unit 2020 identifies the time when the specific operation was performed as the time when the mirror 20 was operated by the driver 30.

[0032] The operation interface of the mirror 20 may be a mechanical interface or a software interface. In the former case, the operation interface of the mirror 20 is a mechanical switch (such as a button or lever). In this case, the detection unit 2020 detects that a specific operation has been performed on a specific mechanical switch, thereby detecting that the mirror 20 has been operated.

[0033] When the operation interface of the mirror 20 is a software interface, the operation interface is, for example, a software interface (such as a button or a slider) displayed on a touch panel or the like provided on the vehicle 10. In this case, the detection unit 2020 detects that a specific operation has been performed on a specific software interface, thereby detecting that the mirror 20 has been operated.

[0034] Here, existing technology can be used to detect that a specific operation has been performed on a specific mechanical interface or software interface.

[0035] <<Detection Method Example 2>> For example, the detection unit 2020 detects that the mirror 20 has been operated by analyzing time-series data of captured images (hereinafter referred to as an image sequence) that include the mirror 20 and the driver 30. In this case, the vehicle 10 is provided with a camera (hereinafter referred to as a first camera) whose imaging range includes the mirror 20 and its surroundings. The first camera repeatedly captures images to generate an image sequence. Hereinafter, this image sequence will be referred to as a first image sequence.

[0036] For example, the detection unit 2020 determines whether or not each captured image included in the first image sequence includes an action of moving the mirror 20. When it is determined that a certain captured image includes an action of moving the mirror 20, the detection unit 2020 detects that the mirror 20 has been operated by the driver 30. In this case, the detection unit 2020 identifies the time when the captured image was generated as the time when the mirror 20 was operated by the driver 30.

[0037] To realize the above determination, for example, a condition (hereinafter referred to as a first predetermined condition) for determining that "the action of moving the mirror 20 by hand is included in the captured image" is defined in advance. In this case, the detection unit 2020 determines whether or not the first predetermined condition is satisfied for each captured image. Then, when the first predetermined condition is satisfied by a certain captured image, the detection unit 2020 determines that the captured image includes the action of moving the mirror 20 by hand. The first predetermined condition may be, for example, "the mirror 20 is being touched by the driver 30" or "the mirror 20 is being held by the driver 30."

[0038] For example, the detection unit 2020 determines that the driver 30 has operated the mirror 20 when a first predetermined condition is satisfied in one captured image of the first image sequence. Alternatively, for example, the detection unit 2020 determines that the driver 30 has operated the mirror 20 when the first predetermined condition is satisfied in each of a plurality of captured images included in a certain period of time in the first image sequence. In the latter case, the detection unit 2020 may determine that the driver 30 has operated the mirror 20 when the first predetermined condition is satisfied in each of all captured images included in the certain period of time, or may determine that the driver 30 has operated the mirror 20 when a predetermined percentage or a predetermined number of captured images among a plurality of captured images included in the certain period of time satisfy the first predetermined condition.

[0039] The specific method for determining whether the above-mentioned first predetermined condition is satisfied is arbitrary. For example, the detection unit 2020 performs object recognition processing on the captured image to detect an image area representing the mirror 20 (hereinafter, "mirror area") and an image area representing the hands of the driver 30 (hereinafter, "hand area"). The detection unit 2020 then determines that the first predetermined condition is satisfied when the mirror area and the hand area are in contact with each other or when the mirror area and the hand area overlap each other.

[0040] The first predetermined condition may be a condition regarding the positional relationship between the joint points of the hand of the driver 30 and the mirror 20. In this case, the detection unit 2020 detects the joint points of the hand of the driver 30 from the captured image and determines whether the positional relationship between the joint points of the hand of the driver 30 and the mirror 20 satisfies the first predetermined condition, thereby detecting the operation of the mirror 20 by the driver 30. As the joint points of the hand of the driver 30, for example, the joint points of each finger of the hand and the joint point of the wrist are detected.

[0041] More specifically, the first predetermined condition may be that "the distance between the joint points of the driver's 30's hands and the mirror area is equal to or less than a threshold value." In this case, the detection unit 2020 detects the joint points of the driver's 30's hands and the mirror area for each captured image included in the first image sequence. The detection unit 2020 further determines whether the distance between the joint points of the driver's 30's hands and the mirror area is equal to or less than a threshold value. Then, when it is determined that the distance between the joint points of the driver's 30's hands and the mirror area is equal to or less than the threshold value for a certain captured image, the detection unit 2020 determines that the mirror 20 was operated by the driver 30 at the time the captured image was generated.

[0042] When multiple joint points of the hand of the driver 30 are detected, the detection unit 2020 calculates the distance between each of the multiple joint points and the mirror area. Then, for example, when one or more of the calculated distances is equal to or less than a threshold value, the detection unit 2020 determines that the mirror 20 has been operated by the driver 30.

[0043] <<Detection Method Example 3>> For example, the detection unit 2020 detects that the mirror 20 has been operated by analyzing an image sequence (hereinafter referred to as the second image sequence) that includes the operation interface of the mirror 20 and the driver 30. In this case, a camera (hereinafter referred to as the second camera) whose imaging range includes the operation interface of the mirror 20 and its surroundings is provided on the vehicle 10. The detection unit 2020 acquires and analyzes the image sequence generated by the second camera.

[0044] For example, the detection unit 2020 determines whether or not each captured image included in the second image sequence includes an action of operating the operation interface of the mirror 20. If a captured image includes an action of operating the operation interface of the mirror 20, the detection unit 2020 determines that the mirror 20 was operated at the time the captured image was generated.

[0045] Here, the method for determining "whether or not the operation of operating the operation interface of the mirror 20 is included" can be the same as the method for determining "whether or not the operation of moving the mirror 20 with a hand" described above. Specifically, for each captured image included in the second image sequence, a method can be used in which it is determined whether or not a condition (hereinafter, a second predetermined condition) such as "the operation interface of the mirror 20 is touched by the driver 30" or "the distance between the joint point of the driver 30's hand and the operation interface of the mirror 20 is equal to or less than a threshold value" is satisfied. When a captured image satisfying the second predetermined condition is detected, the detection unit 2020 determines that the mirror 20 was operated by the driver 30 at the time of generation of the captured image.

[0046] Here, the detection unit 2020 may determine that the driver 30 has operated the mirror 20 when the second predetermined condition is satisfied in one captured image of the second image sequence, or may determine that the driver 30 has operated the mirror 20 when the second predetermined condition is satisfied in each of a plurality of captured images included in a certain period of time in the second image sequence. In the latter case, the detection unit 2020 may determine that the driver 30 has operated the mirror 20 when the second predetermined condition is satisfied in each of all captured images included in the certain period of time, or may determine that the driver 30 has operated the mirror 20 when the second predetermined condition is satisfied in at least a predetermined percentage or a predetermined number of captured images among the plurality of captured images included in the certain period of time.

[0047] <Identifying Head Position 40: S104> The identification unit 2040 identifies the head position 40 at the time when the mirror 20 is operated by the driver 30 (S104). Here, the various methods for identifying the time when the mirror 20 is operated by the driver 30 are as described above.

[0048] For example, the head position 40 is represented by the coordinates of one or more points. When the head position is represented by the coordinates of a single point, the head position 40 is represented by the coordinates of a point representing a specific position of the head. The specific position of the head is, for example, the center of the head. Alternatively, the specific position of the head may be the center of points representing multiple parts of the head (such as both eyes and the nose).

[0049] When the head position 40 is represented by the coordinates of multiple points, for example, the head position 40 is represented by a combination of coordinates of points representing multiple parts of the head. Fig. 5 is a diagram illustrating a case where the head position 40 is represented by a combination of coordinates of multiple points. In this example, the head position 40 is represented by a combination of coordinates P1 of the right eye 51, coordinate P2 of the left eye 52, coordinate P3 of the nose 53, coordinate P4 of the right ear 54, and coordinate P5 of the left ear 55 included in the head 50. Note that the head position 40 may also be represented by a combination of coordinates of any two or more parts out of a specific number of parts.

[0050] To identify the head position 40, for example, a captured image including the head of the driver 30 and its surroundings is used. In this case, a camera (hereinafter, referred to as a third camera) whose image capturing range includes the head of the driver 30 is provided on the vehicle 10.

[0051] For example, the identification unit 2040 identifies, from among the captured images generated by the third camera, the captured image generated at the time closest to the time when the mirror 20 was operated. Then, the identification unit 2040 analyzes the identified captured image to identify the head position 40. Alternatively, for example, the identification unit 2040 may cause the third camera to capture an image in response to detection of the operation of the mirror 20 by the driver 30, and analyze the captured image generated as a result of the image capture.

[0052] A method for identifying the head position 40 by analyzing the captured image will be described below. For example, the identification unit 2040 detects an image area representing a person's head (hereinafter referred to as the head area) from the captured image. The identification unit 2040 then identifies the coordinates of a specific position in the head area (for example, the center position of the head area) as the head position 40.

[0053] Alternatively, for example, the identification unit 2040 may detect specific parts of a person, such as the eyes and nose, from the captured image. Then, the identification unit 2040 identifies a combination of coordinates of the specific parts as the head position 40. Furthermore, the identification unit 2040 may calculate the coordinates of a specific point, such as a center position, from the coordinates of multiple specific parts, and identify the calculated coordinates as the head position 40.

[0054] Here, the position of the head of the driver 30 may be represented by two-dimensional coordinates on the captured image, or by three-dimensional coordinates in a virtual three-dimensional space. In the latter case, the identification unit 2040 converts the two-dimensional coordinates on the captured image acquired from the third camera into three-dimensional coordinates in a predetermined virtual three-dimensional space. Here, existing technology can be used to convert the two-dimensional coordinates on the image into three-dimensional coordinates in the specific virtual three-dimensional space.

[0055] Here, the calculation of coordinates in the virtual three-dimensional space may use a plurality of captured images obtained by capturing images of the head of the driver 30 from different directions. In this case, a plurality of third cameras are provided in the vehicle 10.

[0056] <Setting the Reference Head Position: S106> The setting unit 2060 sets the identified head position 40 as the reference head position (S106). For example, the setting unit 2060 sets the reference head position by generating information indicating the reference head position (hereinafter referred to as reference information) and outputting the generated reference information in a predetermined manner. The reference information is output in a form that can be used by a device that uses the reference information (hereinafter referred to as application device). The application device may be provided inside the vehicle 10 or outside the vehicle 10. Note that the application device may be provided integrally with the state recognition device 2000 (i.e., the state recognition device 2000 may further have a function to operate as an application device), or may be provided separately from the state recognition device 2000.

[0057] The reference information may be output in any manner. For example, the setting unit 2060 may store the reference information in a storage unit accessible from the application device. Alternatively, the setting unit 2060 may transmit the reference information to the application device.

[0058] The setting unit 2060 may use the head position 40 to identify the head posture of the driver 30, and include the identified head posture in the reference information as a reference head posture (hereinafter referred to as reference head posture). The head posture is expressed, for example, by rotation angles (pitch, roll, and yaw) about three-dimensional axes. When the head posture is calculated in this manner, the head position 40 is expressed by a combination of coordinates of two or more points. Here, existing technology can be used as a specific method for calculating the posture of an object from the coordinates of multiple parts of the object.

[0059] The setting unit 2060 may specify the identification information of the driver 30 and include the specified identification information in the reference information. By including the identification information of the driver 30 in the reference information in this way, in a case where the vehicle 10 is shared by multiple people, it is possible to set a reference head position for each person.

[0060] The driver 30 is identified, for example, by using feature amounts (hereinafter referred to as facial feature amounts) obtained from an image of the face of the driver 30. For example, the setting unit 2060 uses the facial feature amounts of the driver 30 as identification information of the driver 30. In this case, the setting unit 2060 acquires a captured image including the face of the driver 30 when generating reference information. The setting unit 2060 then analyzes the acquired captured image to identify the facial feature amounts of the driver 30 and includes the identified facial feature amounts in the reference information.

[0061] Alternatively, for example, the setting unit 2060 may use an identification character string assigned to the driver 30 in advance as the identification information of the driver 30. In this case, personal information associating the identification character string of each person who may drive the vehicle 10 with the facial feature of that person is stored in advance in a storage unit accessible by the setting unit 2060. When generating the reference information, the setting unit 2060 identifies personal information that includes facial feature values ​​that match the facial feature values ​​of the driver 30. Then, the setting unit 2060 includes the identification character string included in the identified personal information in the reference information as the identification information of the driver 30.

[0062] Here, in order to be able to acquire a captured image including the face of the driver 30, the vehicle 10 is provided with a camera that includes the face of the driver 30 in its imaging range. However, the captured image including the face of the driver 30 may be a captured image generated by a third camera (a captured image including the head of the driver 30).

[0063] <Updating of Reference Information> When the vehicle 10 is used repeatedly, the state recognition device 2000 may repeatedly generate reference information. For example, the state recognition device 2000 generates reference information every time the vehicle 10 is started. In this case, the state recognition device 2000 updates previously outputted reference information by outputting newly generated reference information. The application device uses the newly outputted reference information in place of the previously outputted reference information.

[0064] When the reference information is generated for each person, the state recognition device 2000 updates the reference information previously generated for the current driver 30 with the newly generated reference information for that driver 30.

[0065] <Use of Default Head Position> If no operation of the mirror 20 by the driver 30 is detected, the state recognition device 2000 may generate reference information in which the default head position is indicated as the reference head position. For example, the state recognition device 2000 detects operation of the mirror 20 by the driver 30 within a predetermined period of time. If a mirror operation of the mirror 20 by the driver 30 is detected within the predetermined period of time, the state recognition device 2000 sets the head position 40 identified by the various methods described above as the reference head position. On the other hand, if no mirror operation of the mirror 20 by the driver 30 is detected within the predetermined period of time, the state recognition device 2000 sets the default head position as the reference head position.

[0066] The predetermined period is, for example, a period of a predetermined length of time starting from the start-up point of the vehicle 10. The start-up point of the vehicle 10 may be the point at which the engine of the vehicle 10 starts, the point at which the power of the vehicle 10 is turned on, or the like. Alternatively, for example, the predetermined period may be the period from the start-up point of the vehicle 10 until the vehicle 10 starts moving.

[0067] For example, the default head position may be the reference head position indicated in the most recent reference information. As a result, the same head position as the reference head position at the previous ride is set as the reference head position. Alternatively, for example, the default head position may be a head position manually set in advance by the administrator.

[0068] When a reference head position is set for each person, the state recognition device 2000 uses, as the default head position, the reference head position indicated in the reference information most recently generated for the driver 30. When no reference information has been generated for the driver 30 in the past, the state recognition device 2000 uses, for example, a default head position manually set by an administrator.

[0069] When the reference information includes a reference head posture, a default value is used for the reference head posture as well, similar to the reference head position.

[0070] <Example of Application Device> An application device is a device that executes a predetermined task using reference information. Fig. 6 is a diagram illustrating a task execution system 4000 configured with a state recognition device 2000 and an application device 3000.

[0071] The application device 3000 has an acquisition unit 3020, a task execution unit 3040, and an output unit 3060. The acquisition unit 3020 acquires reference information 60 output by the state recognition device 2000. The task execution unit 3040 executes a task using the reference information 60. The output unit 3060 outputs output information representing the execution result of the task.

[0072] The task executed by the task executing unit 3040 is arbitrary. For example, the task executed by the task executing unit 3040 is monitoring the state of the driver 30. In this case, for example, the task executing unit 3040 periodically compares the current head position of the driver 30 with the reference head position indicated in the reference information 60. More specifically, the task executing unit 3040 determines whether the degree of deviation between the current head position of the driver 30 and the reference head position satisfies a predetermined condition. If the degree of deviation between the current head position of the driver 30 and the reference head position satisfies the predetermined condition, the task executing unit 3040 determines that the state of the driver 30 is abnormal.

[0073] If the reference information 60 further includes a reference head posture, the task executing unit 3040 also compares the current head posture of the driver 30 with the reference head posture. That is, the task executing unit 3040 determines whether or not predetermined conditions are satisfied for the degree of deviation between the current head position of the driver 30 and the reference head position, and the degree of deviation between the current head posture of the driver 30 and the reference head posture.

[0074] The degree of deviation between the current head position and the reference head position is expressed, for example, by the distance between the current head position and the reference head position, or the difference in the x coordinate, y coordinate, and z coordinate between the current head position and the reference head position, etc. Furthermore, the degree of deviation between the current head posture and the reference head posture is expressed by the difference in yaw, pitch, and roll, etc.

[0075] If it is determined that the condition of the driver 30 is abnormal, the output unit 3060 outputs output information for notifying the driver 30 that the condition is abnormal. For example, the output information is information representing a notification to the driver 30. In this case, the output information is a predetermined message displayed on a display device provided in the vehicle 10, or a predetermined sound output from a speaker provided in the vehicle 10. In addition, for example, the output information may be information representing a notification to the outside of the vehicle 10 (such as an emergency call).

[0076] The hardware configuration of the application device 3000 is shown in Fig. 3, for example, similar to the hardware configuration of the state recognition device 2000. However, the storage device of the application device 3000 stores programs that realize each functional component of the application device 3000.

[0077] 7 is a flowchart illustrating the flow of processing executed by the application device 3000. The acquisition unit 3020 acquires the reference information 60 (S202). The task execution unit 3040 determines whether the degree of deviation between the current head position of the driver 30 and the reference head position satisfies a predetermined condition (S204). If the degree of deviation does not satisfy the predetermined condition (S204: NO), the task execution unit 3040 executes S204 again. Note that a predetermined waiting time may be set before executing S204 again.

[0078] If the degree of deviation between the current head position of the driver 30 and the reference head position satisfies a predetermined condition (S204: YES), the output unit 3060 outputs output information (S206). Note that, as described above, if the reference information includes a reference head posture, the task executing unit 3040 compares the current head position of the driver 30 with the reference head position, as well as the current head posture of the driver 30 with the reference head posture.

[0079] The output information output by output unit 3060 is not limited to a notification indicating that the state of driver 30 is abnormal. For example, output unit 3060 may output, as output information, a control signal for safely stopping vehicle 10 when the degree of deviation between the current head position of driver 30 and the reference head position satisfies a predetermined condition (i.e., when the state of driver 30 is abnormal). This makes it possible to control vehicle 10, such as stopping it on the shoulder of the road, when the state of driver 30 is abnormal, for example.

[0080] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

[0081] In the above examples, the program includes instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more functions described in the embodiments. The program may be stored on a program or tangible storage medium. By way of example and not limitation, computer-readable media or tangible storage media include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray (registered trademark) disc or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may also be transmitted on a temporary computer-readable medium or communication medium. By way of example and not limitation, temporary computer-readable media or communication media include electrical, optical, acoustic, or other forms of propagated signals.

[0082] Some or all of the above embodiments can be described as, but are not limited to, the following supplementary notes. (Supplementary Note 1) A state recognition device comprising: a detection means for detecting that a vehicle mirror has been operated; a determination means for determining the head position of the driver of the vehicle at the time the operation was performed; and a setting means for setting the determined head position as a reference position for the head in the vehicle. (Supplementary Note 2) The state recognition device according to Supplementary Note 1, wherein the detection means detects that the mirror has been operated by detecting an action of operating the mirror from a first captured image including the mirror. (Supplementary Note 3) The state recognition device according to Supplementary Note 2, wherein the action of operating the mirror is an action of moving the mirror by hand or an action of operating an operation interface for the mirror. (Supplementary Note 4) The state recognition device according to any one of Supplements 1 to 3, wherein the determination means acquires a second captured image including the driver's head, and determines the head position of the driver based on the coordinates of one or more points related to the head included in the second captured image. (Supplementary Note 5) The condition recognition device according to Supplementary Note 1, wherein the setting means outputs reference information including the identified head position. (Supplementary Note 6) The condition recognition device according to Supplementary Note 5, wherein the identification means identifies the head position of the driver by a combination of coordinates of multiple parts of the head, and the setting means identifies the head posture based on the combination of coordinates and outputs the reference information including the identified head posture. (Supplementary Note 7) The condition recognition device according to Supplementary Note 5 or 6, wherein the setting means identifies identification information of the driver and outputs the reference information including the identified identification information. (Supplementary Note 8) The condition recognition device according to Supplementary Note 5, wherein the setting means outputs the reference information including the head position when the driver's head position is identified within a predetermined period, and outputs the reference information including a predetermined head position when the driver's head position is not identified within the predetermined period. (Supplementary Note 9) The condition recognition device according to Supplementary Note 8, wherein the predetermined head position is a head position included in the reference information previously output.(Supplementary Note 10) A computer-executed state recognition method comprising: a detection step of detecting that a vehicle mirror has been operated; a specification step of specifying the head position of the driver of the vehicle at the time the operation was performed; and a setting step of setting the specified head position as a reference position for the head of the vehicle. (Supplementary Note 11) The state recognition method of Supplementary Note 10, wherein the detection step detects that the mirror has been operated by detecting an action of operating the mirror from a first captured image including the mirror. (Supplementary Note 12) The state recognition method of Supplementary Note 11, wherein the action of operating the mirror is an action of moving the mirror by hand or an action of operating an operation interface of the mirror. (Supplementary Note 13) The state recognition method of any one of Supplements 10 to 12, wherein the specification step acquires a second captured image including the driver's head, and specifies the head position of the driver based on the coordinates of one or more points related to the head included in the second captured image. (Supplementary Note 14) The state recognition method of Supplementary Note 10, wherein the setting step outputs reference information including the specified head position. (Supplementary Note 15) The state recognition method according to Supplementary Note 14, wherein in the specifying step, the head position of the driver is specified by a combination of coordinates of each of a plurality of parts of the head, and in the setting step, the head posture is specified based on the combination of coordinates and the reference information including the specified head posture is output. (Supplementary Note 16) The state recognition method according to Supplementary Note 14 or 15, wherein in the setting step, identification information of the driver is specified and the reference information including the specified identification information is output. (Supplementary Note 17) The state recognition method according to Supplementary Note 14, wherein in the setting step, if the head position of the driver is specified within a predetermined period, the reference information including the head position is output, and if the head position of the driver is not specified within the predetermined period, the reference information including a predetermined head position is output. (Supplementary Note 18) The state recognition method according to Supplementary Note 17, wherein the predetermined head position is a head position included in the reference information output in the past.(Supplementary Note 19) A non-transitory computer-readable medium storing a program that causes a computer to execute the following steps: a detection step of detecting that a vehicle mirror has been operated; a specification step of specifying the head position of the driver of the vehicle at the time the operation was performed; and a setting step of setting the specified head position as a reference position for the head in the vehicle. (Supplementary Note 20) The computer-readable medium of Supplementary Note 19, wherein the detection step detects that the mirror has been operated by detecting an action of operating the mirror from a first captured image that includes the mirror. (Supplementary Note 21) The computer-readable medium of Supplementary Note 20, wherein the action of operating the mirror is an action of moving the mirror by hand or an action of operating an operation interface of the mirror. (Supplementary Note 22) The computer-readable medium of any one of Supplements 19 to 21, wherein the specification step acquires a second captured image that includes the driver's head, and specifies the head position of the driver based on the coordinates of one or more points related to the head that are included in the second captured image. (Supplementary Note 23) The computer-readable medium of Supplementary Note 19, wherein the setting step outputs reference information including the identified head position. (Supplementary Note 24) The computer-readable medium of Supplementary Note 23, wherein the specifying step specifies the head position of the driver by combining coordinates of multiple parts of the head, and the setting step specifies the head posture based on the combination of coordinates and outputs the reference information including the identified head posture. (Supplementary Note 25) The computer-readable medium of Supplementary Note 23 or 24, wherein the setting step specifies identification information of the driver and outputs the reference information including the identified identification information. (Supplementary Note 26) The computer-readable medium of Supplementary Note 23, wherein the setting step: if the driver's head position is specified within a predetermined period, the reference information including the head position is output, and if the driver's head position is not specified within the predetermined period, the reference information including a predetermined head position is output.(Supplementary Note 27) The computer-readable medium of Supplementary Note 26, wherein the predetermined head position is a head position included in the reference information output in the past. (Supplementary Note 28) A task execution system comprising: a state recognition device and an application device, wherein the state recognition device comprises: detection means for detecting that a vehicle mirror has been operated; identification means for identifying the head position of the driver of the vehicle at the time the operation was performed; and setting means for outputting reference information including the identified head position, and the application device comprises: acquisition means for acquiring the reference information; task execution means for executing a task using the reference information; and output means for outputting output information related to the execution result of the task. (Supplementary Note 29) The task execution system of Supplementary Note 28, wherein the task execution means determines whether a degree of deviation between the driver's current head position and the head position included in the reference information satisfies a predetermined condition, and the output means generates the output information notifying that the driver's condition is abnormal if the degree of deviation satisfies the predetermined condition. (Supplementary Note 30) A task execution method in which a state recognition device executes: a detection step of detecting that a vehicle mirror has been operated, an identification step of identifying the head position of the driver of the vehicle at the time the operation was performed, and a setting step of outputting reference information including the identified head position, and an application device executes: an acquisition step of acquiring the reference information, a task execution step of executing a task using the reference information, and an output step of outputting output information related to the execution result of the task. (Supplementary Note 31) The task execution method according to Supplementary Note 30, in which: in the task execution step, it is determined whether a degree of deviation between the current head position of the driver and the head position included in the reference information satisfies a predetermined condition; and in the output step, if the degree of deviation satisfies the predetermined condition, the output information is generated to notify that the driver's condition is abnormal.

[0083] REFERENCE SIGNS LIST 10 Vehicle 20 Mirror 30 Driver 40 Head position 50 Head 51 Right eye 52 Left eye 53 Nose 54 Right ear 55 Left ear 60 Reference information 1000 Computer 1020 Bus 1040 Processor 1060 Memory 1080 Storage device 1100 Input / output interface 1120 Network interface 2000 State recognition device 2020 Detection unit 2040 Identification unit 2060 Setting unit 3000 Application device 3020 Acquisition unit 3040 Task execution unit 3060 Output unit 4000 Task execution system

Claims

1. A detection means for detecting that a mirror of the vehicle is operated; an identification means for identifying a head position of a driver of the vehicle at the time when the operation is performed; and a setting means for setting the identified head position as a reference position for the head in the vehicle.

2. The state recognition device according to claim 1 , wherein the detection means detects that the mirror has been operated by detecting an action of operating the mirror from a first captured image including the mirror.

3. The state recognition device according to claim 2 , wherein the action of operating the mirror is an action of moving the mirror by hand or an action of operating an operation interface of the mirror.

4. 4. The state recognition device according to claim 1, wherein the identification means acquires a second captured image including the driver's head, and identifies the head position of the driver based on coordinates of one or more points related to the head included in the second captured image.

5. The state recognition device according to claim 1 , wherein the setting means outputs reference information including the identified head position.

6. The identification means identifies a head position of the driver by a combination of coordinates of each of a plurality of parts of the head, The state recognition device according to claim 5 , wherein the setting means identifies the head posture based on the combination of coordinates, and outputs the reference information including the identified head posture.

7. 7. The state recognition device according to claim 5, wherein the setting unit specifies identification information of the driver, and outputs the reference information including the specified identification information.

8. a detection step of detecting that a mirror of the vehicle has been operated; a step of identifying a head position of a driver of the vehicle at a time when the operation is performed; and setting the identified head position as a reference position for the head in the vehicle.

9. a detection step of detecting that a mirror of the vehicle has been operated; a step of identifying a head position of a driver of the vehicle at a time when the operation is performed; and a setting step of setting the identified head position as a reference position for the head in the vehicle.

10. A state recognition device and an application device are provided. The state recognition device includes: A detection means for detecting that a mirror of the vehicle is operated; an identification means for identifying a head position of a driver of the vehicle at the time when the operation is performed; and a setting means for outputting reference information including the identified head position, The application device includes: An acquisition means for acquiring the reference information; a task execution means for executing a task using the reference information; and an output unit for outputting output information relating to the execution result of the task.