State recognition device, state recognition method, program, and task execution system

Abstract

A state recognition device (2000) detects that a mirror (20) of a vehicle (10) has been adjusted. The state recognition device (2000) specifies the head position of a driver (30) of the vehicle (10) at the time point when the mirror (20) was adjusted. The state recognition device (2000) sets the specified head position as a reference head position in the vehicle (10).

Description

Technical Field

[0001] The present disclosure relates to a state recognition device, a state recognition method, a computer-readable medium, a task execution system, and a task execution method.

Background Art

[0002] In vehicles such as automobiles, technologies for detecting the state of a driver have been developed. For example, Patent Document 1 discloses a technology for detecting that a driver is looking away by using the direction of the driver's face when adjusting an outer mirror as a reference value.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In Patent Document 1, the direction of the driver's face is used as a reference regarding the state of the driver. An object of the present disclosure is to disclose a more efficient technique for determining a reference regarding the state of the driver.

Means for Solving the Problems

[0005] The state recognition device of the present disclosure includes a detection unit that detects that a mirror of a vehicle has been operated, a specification unit that specifies the head position of the driver of the vehicle at the time when the operation is performed, and a setting unit that sets the specified head position as a reference position of the head in the vehicle.

[0006] The state recognition method disclosed herein is performed by a computer. The state recognition method includes a detection step of detecting that a vehicle's mirror has been operated; a identification step of identifying the head position of the vehicle's driver at the time the operation was performed; and a setting step of setting the identified head position as the reference position of the head in the vehicle.

[0007] The program disclosed herein causes a computer to execute the state recognition method disclosed herein. [Effects of the Invention]

[0008] This disclosure provides a more efficient technology for determining criteria regarding the driver's condition. [Brief explanation of the drawing]

[0009] [Figure 1] This figure illustrates an overview of the operation of the state recognition device according to the embodiment. [Figure 2] This is a block diagram illustrating the functional configuration of a state recognition device. [Figure 3] This is a block diagram illustrating the hardware configuration of a computer that implements a state recognition device. [Figure 4] This flowchart illustrates the flow of processing performed by a state recognition device. [Figure 5] This diagram illustrates a case where the head position is represented by a combination of coordinates of multiple points. [Figure 6] This is a block diagram illustrating the functional configuration of a system having a state recognition device and an application device. [Figure 7] This flowchart illustrates the flow of processing performed by an application device. [Modes for carrying out the invention]

[0010] Embodiments of the present disclosure will be described in detail below with reference to the drawings. In each drawing, the same or corresponding elements are denoted by the same reference numerals, and redundant explanations are omitted as necessary for clarity. Unless otherwise specified, predetermined values ​​such as specified values ​​and thresholds are stored in advance in a storage device accessible from the device that uses those values. Furthermore, unless otherwise specified, the storage unit is composed of one or any number of storage devices.

[0011] <Overview> Figure 1 is a diagram illustrating the overview of the state recognition device 2000 according to the embodiment. Here, Figure 1 is a diagram intended to facilitate understanding of the overview of the state recognition device 2000, and the operation of the state recognition device 2000 is not limited to what is shown in Figure 1.

[0012] The state recognition device 2000 sets the position of the driver 30 at the time the mirror 20 installed on the vehicle 10 is operated as the reference head position (hereinafter referred to as the reference head position) on the vehicle 10. The vehicle 10 is, for example, an automobile. The mirror 20 is a mirror used by the driver 30 when driving, and is, for example, a rearview mirror or a side mirror.

[0013] Furthermore, only one type of mirror may be treated as mirror 20, or multiple types of mirrors may be treated as mirror 20. In the latter case, for example, the state recognition device 2000 determines that mirror 20 has been operated by the driver 30 when either the rearview mirror or one of the two side mirrors is operated.

[0014] For example, the state recognition device 2000 operates as follows: First, the state recognition device 2000 detects the driver 30's operation of the mirror 20. Furthermore, the state recognition device 2000 identifies the driver 30's head position (head position 40) at the time the driver 30 operates the mirror 20. Then, the state recognition device 2000 sets the identified head position 40 as the reference head position in the vehicle 10.

[0015] Although the head position 40 is represented by a single point in FIG. 1, the head position 40 may be represented by a combination of multiple points. Details regarding how the head position 40 is represented will be described later.

[0016] <Examples of effects> According to the state recognition device 2000, in response to detecting an operation of the mirror 20 by the driver 30, the head position of the driver 30 at the time of the operation is set as a reference head position. Thus, according to the state recognition device 2000, a new technique for determining the reference of the state of the vehicle driver is provided.

[0017] Here, when the mirror 20 is being operated, the position of the driver 30's head is considered to be close to the position of the head when the driver 30 is looking ahead of the vehicle 10. Therefore, according to the state recognition device 2000, the position of the driver 30's head in a state close to looking ahead can be easily set as the reference head position.

[0018] Hereinafter, the state recognition device 2000 of the present embodiment will be described in more detail.

[0019] <Examples of functional configurations> FIG. 2 is a block diagram illustrating the functional configuration of the state recognition device 2000 of the embodiment. The state recognition device 2000 includes a detection unit 2020, an identification unit 2040, and a setting unit 2060. The detection unit 2020 detects an operation of the mirror 20 by the driver 30. The identification unit 2040 identifies the head position 40 at the time when the mirror 20 is operated by the driver 30. The setting unit 2060 sets the identified head position 40 as the reference head position in the vehicle 10.

[0020] <Examples of hardware configurations> Each functional component of the state recognition device 2000 may be implemented by hardware (e.g., hardwired electronic circuits) or by a combination of hardware and software (e.g., a combination of electronic circuits and programs that control them). The following will further explain the case where each functional component of the state recognition device 2000 is implemented by a combination of hardware and software.

[0021] Figure 3 is a block diagram illustrating the hardware configuration of the computer 1000 that implements the state recognition device 2000. The computer 1000 is any computer. For example, the computer 1000 may be an ECU (Electronic Control Unit), navigation device, or tablet terminal located inside the vehicle 10. Alternatively, the computer 1000 may be a PC (Personal Computer) or server machine located outside the vehicle 10. The computer 1000 may be a dedicated computer designed to implement the state recognition device 2000, or it may be a general-purpose computer.

[0022] For example, by installing a predetermined application on computer 1000, the various functions of the state recognition device 2000 are realized on computer 1000. The above application consists of programs for realizing each functional component of the state recognition device 2000. The method of obtaining the above program is arbitrary. For example, the program can be obtained from a storage medium (such as a DVD disc or USB memory) on which it is stored. Alternatively, the program can be obtained by downloading it from a server device that manages the storage device on which it is stored.

[0023] Computer 1000 includes a bus 1020, a processor 1040, memory 1060, a storage device 1080, an input / output interface 1100, and a network interface 1120. The bus 1020 is a data transmission path for the processor 1040, memory 1060, storage device 1080, input / output interface 1100, and network interface 1120 to send and receive data from each other. However, the method of connecting the processor 1040 and other components is not limited to bus connection.

[0024] Processor 1040 is a variety of processors such as CPU (Central Processing Unit), GPU (Graphics Processing Unit), FPGA (Field-Programmable Gate Array), and DSP (Digital Signal Processor). Memory 1060 is main memory implemented using RAM (Random Access Memory), etc. Storage device 1080 is auxiliary storage implemented using hard disk, SSD (Solid State Drive), memory card, or ROM (Read Only Memory), etc.

[0025] The input / output interface 1100 is an interface for connecting the computer 1000 with input / output devices. For example, input devices such as keyboards and output devices such as display devices are connected to the input / output interface 1100.

[0026] The network interface 1120 is an interface for connecting computer 1000 to a network. This network may be a LAN (Local Area Network) or a WAN (Wide Area Network).

[0027] The storage device 1080 stores programs that implement each functional component of the state recognition device 2000 (programs that implement the aforementioned applications). The processor 1040 reads these programs into memory 1060 and executes them to implement each functional component of the state recognition device 2000.

[0028] The state recognition device 2000 may be implemented by one computer 1000 or by multiple computers 1000. In the latter case, the configuration of each computer 1000 does not need to be the same and can be different.

[0029] <Processing flow> Figure 4 is a flowchart illustrating the processing flow performed by the state recognition device 2000 of the embodiment. The detection unit 2020 detects the operation of the mirror 20 by the driver 30 (S102). The identification unit 2040 identifies the head position 40 at the time the mirror 20 is operated by the driver 30 (S104). The setting unit 2060 sets the identified head position 40 as the reference head position (S106).

[0030] <Detection of operation of mirror 20: S102> The detection unit 2020 detects the operation of the mirror 20 by the driver 30 (S102). Several specific detection methods are given below as examples.

[0031] <<Example of detection method 1>> For example, the detection unit 2020 detects the operation of the mirror 20, which is provided on the vehicle 10, by detecting that a specific operation has been performed on the operation interface of the mirror 20. In this case, the detection unit 2020 identifies the time when the specific operation was performed as the time when the mirror 20 was operated by the driver 30.

[0032] The operating interface of the mirror 20 may be a mechanical interface or a software interface. In the former case, the operating interface of the mirror 20 is a mechanical switch (such as a button or lever). In this case, the detection unit 2020 detects that the mirror 20 has been operated by detecting that a specific operation has been performed on a specific mechanical switch.

[0033] If the operating interface of the mirror 20 is a software interface, then the operating interface is, for example, a software interface (such as buttons or sliders) displayed on a touch panel provided on the vehicle 10. In this case, the detection unit 2020 detects that the mirror 20 has been operated by detecting that a specific operation has been performed on a specific software interface.

[0034] In this case, existing technologies can be used to detect when a specific operation has been performed on a particular mechanical or software interface.

[0035] <<Example of detection method 2>> For example, the detection unit 2020 detects that the mirror 20 has been operated by analyzing time-series data (hereinafter referred to as the image sequence) of captured images that include the mirror 20 and the driver 30. In this case, the vehicle 10 is equipped with a camera (hereinafter referred to as the first camera) whose imaging range includes the mirror 20 and its surroundings. The first camera generates an image sequence by repeatedly taking images. Hereinafter, this image sequence will be referred to as the first image sequence.

[0036] For example, the detection unit 2020 determines whether each captured image included in the first image sequence includes an action to move the mirror 20. If it is determined that an action to move the mirror 20 is included in a certain captured image, the detection unit 20 detects that the mirror 20 was operated by the driver 30. In this case, the detection unit 2020 identifies the time when the captured image was generated as the time when the mirror 20 was operated by the driver 30.

[0037] To achieve the above determination, for example, a condition (hereinafter referred to as the first predetermined condition) for determining whether "the action of moving the mirror 20 by hand is included in the captured image" is defined in advance. In this case, the detection unit 2020 determines whether the first predetermined condition is met for each captured image. Then, if the first predetermined condition is met for a certain captured image, the detection unit 20 determines that the captured image includes the action of moving the mirror 20 by hand. For example, the first predetermined condition can be a condition such as "the mirror 20 is being touched by the driver 30" or "the mirror 20 is being grasped by the driver 30".

[0038] For example, the detection unit 2020 determines that the driver 30 has operated the mirror 20 if the first predetermined condition is met in one of the captured images in the first image sequence. Alternatively, the detection unit 2020 may determine that the driver 30 has operated the mirror 20 if the first predetermined condition is met in each of the multiple captured images included within a certain time period in the first image sequence. In the latter case, the detection unit 2020 may determine that the driver 30 has operated the mirror 20 if the first predetermined condition is met in each of the captured images included within the certain time period, or it may determine that the driver 30 has operated the mirror 20 if the first predetermined condition is met in each of the multiple captured images included within the certain time period in a predetermined proportion or a predetermined number of images.

[0039] The specific method for determining whether the aforementioned first predetermined condition is met is arbitrary. For example, the detection unit 2020 performs object recognition processing on the captured image to detect an image region representing the mirror 20 (hereinafter referred to as the mirror region) and an image region representing the driver's hand (hereinafter referred to as the hand region). The detection unit 2020 then determines that the first predetermined condition is met if the mirror region and the hand region are in contact, or if the mirror region and the hand region overlap.

[0040] The first predetermined condition may be a condition relating to the positional relationship between the joint points of the driver's hand 30 and the mirror 20. In this case, the detection unit 2020 detects the joint points of the driver's hand 30 from the captured image and determines whether the positional relationship between the joint points of the driver's hand 30 and the mirror 20 satisfies the first predetermined condition, thereby detecting operation of the mirror 20 by the driver 30. Examples of the joint points of the driver's hand that may be detected are the joint points of each finger or the wrist joint.

[0041] More specifically, a first predetermined condition may be adopted that "the distance between the joint points of the driver's hand 30 and the mirror area is less than or equal to a threshold." In this case, the detection unit 2020 detects the joint points of the driver's hand and the mirror area for each captured image included in the first image sequence. Furthermore, the detection unit 2020 determines whether or not the distance between the joint points of the driver's hand 30 and the mirror area is less than or equal to a threshold. If it is determined that the distance between the joint points of the driver's hand 30 and the mirror area is less than or equal to a threshold for a given captured image, the detection unit 20 determines that the mirror 20 was operated by the driver 30 at the time the captured image was generated.

[0042] Furthermore, if multiple joint points of the driver's hand are detected, the detection unit 2020 calculates the distance between each of these joint points and the mirror area. For example, if one or more of the calculated distances are below a threshold, the detection unit 20 determines that the mirror 20 has been operated by the driver 30.

[0043] <<Example of detection method 3>> For example, the detection unit 2020 detects that the mirror 20 has been operated by analyzing an image sequence (hereinafter referred to as the second image sequence) that includes the operation interface of the mirror 20 and the driver 30. In this case, a camera (hereinafter referred to as the second camera) that includes the operation interface of the mirror 20 and its surroundings in its imaging range is provided on the vehicle 10. The detection unit 2020 acquires and analyzes the image sequence generated by the second camera.

[0044] For example, the detection unit 2020 determines whether each captured image included in the second image sequence includes an operation of the mirror 20's operating interface. If a certain captured image includes an operation of the mirror 20's operating interface, the detection unit 2020 determines that the mirror 20 was operated at the time the captured image was generated.

[0045] Here, the method for determining whether or not the operation of the mirror 20's operating interface is included can be the same as the method for determining whether or not the operation of moving the mirror 20 by hand is included, as described above. Specifically, for each captured image included in the second image sequence, a method can be adopted to determine whether or not the conditions (hereinafter referred to as the second predetermined conditions) are met, such as whether or not the operation interface of the mirror 20 is touched by the driver 30 or whether or not the distance between the joint points of the driver 30's hand and the operation interface of the mirror 20 is below a threshold. If an captured image that satisfies the second predetermined conditions is detected, the detection unit 2020 determines that the mirror 20 was operated by the driver 30 at the time the captured image was generated.

[0046] Here, the detection unit 2020 may determine that the driver 30 operated the mirror 20 if the second predetermined condition is met in one of the captured images in the second image sequence, or it may determine that the driver 30 operated the mirror 20 if the second predetermined condition is met in each of the multiple captured images included in a certain period of time in the second image sequence. In the latter case, the detection unit 2020 may also determine that the driver 30 operated the mirror 20 if the second predetermined condition is met in each of the captured images included in a certain period of time, or it may determine that the driver 30 operated the mirror 20 if the second predetermined condition is met in each of the multiple captured images included in a certain period of time that is a predetermined percentage or a predetermined number or more.

[0047] <Identification of head position 40: S104> The identification unit 2040 identifies the head position 40 at the time the mirror 20 is operated by the driver 30 (S104). Hereinafter, various methods for identifying the time when the mirror 20 is operated by the driver 30 are as described above.

[0048] For example, the head position 40 is represented by the coordinates of one or more points. When the head position is represented by the coordinates of a single point, for example, the head position 40 is represented by the coordinates of a point that represents a specific location on the head. A specific location on the head is, for example, the center of the head. Alternatively, for example, a specific location on the head may be the center of points representing multiple parts of the head (such as both eyes or the nose).

[0049] When the head position 40 is represented by the coordinates of multiple points, for example, the head position 40 is represented by a combination of the coordinates of points representing each of several parts of the head. Figure 5 illustrates a case in which the head position 40 is represented by a combination of the coordinates of multiple points. In this example, the head position 40 is represented by a combination of the coordinates P1 of the right eye 51, P2 of the left eye 52, P3 of the nose 53, P4 of the right ear 54, and P5 of the left ear 55, all of which are included in the head 50. Note that the head position 40 may also be represented by a combination of the coordinates of any two or more of several specific parts.

[0050] To determine the head position 40, for example, an image captured that includes the driver's head 30 and its surroundings is used. In this case, a camera (hereinafter referred to as the third camera) that includes the driver's head 30 in its imaging range is installed in the vehicle 10.

[0051] For example, the identification unit 2040 identifies the image captured by the third camera from among the captured images, specifically the image captured at the time closest to the time when the mirror 20 was operated. Then, the identification unit 2040 identifies the head position 40 by analyzing the identified image. Alternatively, for example, the identification unit 2040 may cause the third camera to take an image in response to the detection of the driver 30 operating the mirror 20, and then analyze the image generated as a result of that image capture.

[0052] The following describes a method for identifying the head position 40 by analyzing the captured image. For example, the identification unit 2040 detects an image region representing a person's head (hereinafter referred to as the head region) from the captured image. The identification unit 2040 then identifies the coordinates of a specific position within the head region (for example, the center position of the head region) as the head position 40.

[0053] In addition, for example, the identification unit 2040 detects specific parts of a person, such as the eyes and nose, from the captured image. The identification unit 2040 then identifies the combination of coordinates of each specific part as the head position 40. Alternatively, the identification unit 2040 may calculate the coordinates of a specific point, such as the center position, from the coordinates of multiple specific parts, and identify the calculated coordinates as the head position 40.

[0054] Here, the position of the driver's head 30 may be represented by two-dimensional coordinates on the captured image, or by three-dimensional coordinates in a virtual three-dimensional space. In the latter case, the specific unit 2040 converts the two-dimensional coordinates on the captured image acquired from the third camera into three-dimensional coordinates in a predetermined virtual three-dimensional space. Here, existing technologies can be used for the technique of converting two-dimensional coordinates on an image into three-dimensional coordinates in a specific virtual three-dimensional space.

[0055] Here, to calculate the coordinates in the virtual three-dimensional space, multiple captured images obtained by capturing the driver's head from different directions may be used. In this case, the vehicle 10 is equipped with multiple third cameras.

[0056] <Setting the reference head position: S106> The setting unit 2060 sets the identified head position 40 as the reference head position (S106). For example, the setting unit 2060 sets the reference head position by generating information indicating the reference head position (hereinafter, reference information) and outputting the generated reference information in a predetermined manner. The reference information is output in a manner that can be used by a device that utilizes the reference information (hereinafter, application device). The application device may be provided inside the vehicle 10 or outside the vehicle 10. The application device may be provided as an integral part of the state recognition device 2000 (i.e., the state recognition device 2000 may be further equipped with the function to operate as an application device), or it may be provided separately from the state recognition device 2000.

[0057] The method of outputting the reference information is arbitrary. For example, the setting unit 2060 stores the reference information in a memory unit accessible from the application device. Alternatively, for example, the setting unit 2060 can transmit the reference information to the application device.

[0058] The setting unit 2060 may use the head position 40 to identify the head posture of the driver 30 and include the identified head posture as the reference head posture (hereinafter referred to as the reference head posture) in the reference information. The head posture is represented, for example, by the rotation angles (pitch, roll, and yaw) for each of the three-dimensional axes. When the head posture is calculated in this way, the head position 40 is represented by a combination of coordinates of two or more points. Here, existing technologies can be used as a specific method for calculating the posture of an object from the coordinates of multiple parts of the object.

[0059] The setting unit 2060 may identify the identification information of the driver 30 and include the identified identification information in the reference information. By including the identification information of the driver 30 in the reference information in this way, a reference head position can be set for each person when the vehicle 10 is shared by multiple people.

[0060] The identification of driver 30 is performed, for example, by using features obtained from an image of driver 30's face (hereinafter referred to as facial features). For example, the setting unit 2060 uses the facial features of driver 30 as identification information for driver 30. In this case, when generating reference information, the setting unit 2060 acquires an image containing the face of driver 30. The setting unit 2060 then analyzes the acquired image to identify the facial features of driver 30 and includes the identified facial features in the reference information.

[0061] Alternatively, the setting unit 2060 may use an identification string pre-assigned to the driver 30 as the driver 30's identification information. In this case, for each person who may potentially drive the vehicle 10, person information that associates that person's identification string with their facial features is pre-stored in a memory accessible from the setting unit 2060. When generating the reference information, the setting unit 2060 identifies person information that contains facial features that match those of the driver 30. The setting unit 2060 then includes the identification string contained in the identified person information as the driver 30's identification information in the reference information.

[0062] Here, in order to acquire an image containing the face of the driver 30, the vehicle 10 is provided with a camera whose imaging range includes the face of the driver 30. However, the image containing the face of the driver 30 may be an image generated by a third camera (an image containing the head of the driver 30).

[0063] <Update of standard information> If vehicle 10 is used repeatedly, the status recognition device 2000 may repeatedly generate reference information. For example, the status recognition device 2000 generates reference information each time vehicle 10 is started. In this case, the status recognition device 2000 updates the previously outputted reference information by outputting the newly generated reference information. The application device uses the newly outputted reference information in place of the previously outputted reference information.

[0064] Furthermore, if reference information is generated for each person, the state recognition device 2000 updates the previously generated reference information for the current driver 30 with the newly generated reference information for that driver 30.

[0065] <Using the default head position> If no operation of the mirror 20 by the driver 30 is detected, the state recognition device 2000 may generate reference information indicating the default head position as the reference head position. For example, the state recognition device 2000 detects operation of the mirror 20 by the driver 30 within a predetermined period. If mirror operation of the mirror 20 by the driver 30 is detected within the predetermined period, the state recognition device 2000 sets the head position 40 identified by the various methods described above as the reference head position. On the other hand, if no mirror operation of the mirror 20 by the driver 30 is detected within the predetermined period, the state recognition device 2000 sets the default head position as the reference head position.

[0066] The predetermined period is, for example, a period of a predetermined length starting from the time the vehicle 10 is started. The time the vehicle 10 is started can be the time when the engine of the vehicle 10 is started or the time when the power to the vehicle 10 is turned on. Alternatively, the predetermined period may be the period from the time the vehicle 10 is started until the vehicle 10 starts moving.

[0067] The default head position may, for example, be the reference head position shown in the most recent reference information. This sets the reference head position to the same position as the reference head position during the previous ride. Alternatively, the default head position may be a head position that has been manually set in advance by the administrator.

[0068] If a reference head position is set for each person, the state recognition device 2000 uses the reference head position indicated in the most recently generated reference information for the driver 30 as the default head position. If no reference information has been generated for the driver 30 in the past, for example, the state recognition device 2000 uses the default head position manually set by the administrator.

[0069] If the reference information includes a reference head posture, the default value will be used for the reference head posture as well, similar to the reference head position.

[0070] <Example of an application device> An application device is a device that performs a predetermined task using reference information. Figure 6 is an example of a task execution system 4000 consisting of a state recognition device 2000 and an application device 3000.

[0071] The application device 3000 includes an acquisition unit 3020, a task execution unit 3040, and an output unit 3060. The acquisition unit 3020 acquires reference information 60 output by the state recognition device 2000. The task execution unit 3040 executes a task using the reference information 60. The output unit 3060 outputs output information representing the task execution result.

[0072] The tasks executed by the task execution unit 3040 are arbitrary. For example, a task executed by the task execution unit 3040 is monitoring the state of the driver 30. In this case, for example, the task execution unit 3040 periodically compares the current head position of the driver 30 with the reference head position shown in the reference information 60. More specifically, the task execution unit 3040 determines whether the degree of deviation between the current head position of the driver 30 and the reference head position satisfies predetermined conditions. If the degree of deviation between the current head position of the driver 30 and the reference head position satisfies predetermined conditions, the task execution unit 3040 determines that the state of the driver 30 is abnormal.

[0073] If the reference information 60 also includes a reference head posture, the task execution unit 3040 also compares the driver's 30's current head posture with the reference head posture. That is, the task execution unit 3040 determines whether predetermined conditions are met regarding the degree of deviation between the driver's 30's current head position and the reference head position, and the degree of deviation between the driver's 30's current head posture and the reference head posture.

[0074] The degree of deviation between the current head position and the reference head position can be expressed, for example, by the distance between the current head position and the reference head position, or by the differences in the x, y, and z coordinates between the current head position and the reference head position. Similarly, the degree of deviation between the current head orientation and the reference head orientation can be expressed by the differences in yaw, pitch, and roll.

[0075] If the driver 30 is determined to be in an abnormal state, the output unit 3060 outputs output information to notify the driver 30 of the abnormal state. For example, the output information is information representing a notification to the driver 30. In this case, the output information is a predetermined message displayed on a display device installed in the vehicle 10, or a predetermined sound output from a speaker installed in the vehicle 10. Alternatively, the output information may also be information representing a notification to the outside of the vehicle 10 (such as an emergency call).

[0076] The hardware configuration of the application device 3000 is similar to that of the state recognition device 2000, as shown in Figure 3, for example. However, the storage device of the application device 3000 stores programs that implement each functional component of the application device 3000.

[0077] Figure 7 is a flowchart illustrating the processing flow performed by the application device 3000. The acquisition unit 3020 acquires reference information 60 (S202). The task execution unit 3040 determines whether the degree of deviation between the driver's 30 current head position and the reference head position satisfies predetermined conditions (S204). If the degree of deviation does not satisfy the predetermined conditions (S204: NO), the task execution unit 3040 executes S204 again. A predetermined waiting time may be provided before executing S204 again.

[0078] If the degree of deviation between the driver's 30 current head position and the reference head position satisfies a predetermined condition (S204: YES), the output unit 3060 outputs output information (S206). As mentioned above, if the reference information includes a reference head posture, the task execution unit 3040 compares the driver's 30 current head position with the reference head position, as well as the driver's 30 current head posture with the reference head posture.

[0079] The output information output by the output unit 3060 is not limited to notifications indicating that the driver 30's condition is abnormal. For example, the output unit 3060 may output a control signal as output information to safely stop the vehicle 10 when the degree of deviation between the driver 30's current head position and a reference head position satisfies a predetermined condition (i.e., the driver 30's condition is abnormal). This makes it possible to control the vehicle 10, for example, to stop it on the shoulder of the road when the driver 30's condition is abnormal.

[0080] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above embodiments. Various modifications to the structure and details of the present invention can be made, which can be understood by those skilled in the art within the scope of the present invention.

[0081] In the above examples, the program includes a set of instructions (or software code) that, when loaded into a computer, causes the computer to perform one or more of the functions described in the embodiments. The program may be stored in a program or a physical storage medium. Examples, but not limited to, include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technologies, CD-ROM, digital versatile disc (DVD), Blu-ray® disc or other optical disc storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices. The program may be transmitted over a temporary computer-readable medium or a communication medium. Examples, but not limited to, include temporary computer-readable medium or a communication medium that includes electrical, optical, acoustic or other forms of propagating signals.

[0082] Some or all of the above embodiments may also be described as follows, but are not limited to the following: (Note 1) A detection means for detecting when a vehicle's mirror has been operated, A means for identifying the position of the driver's head at the time the aforementioned operation was performed, A state recognition device having setting means for setting the identified head position to a reference position of the head in the vehicle. (Note 2) The state recognition device according to Appendix 1, wherein the detection means detects that the mirror has been operated by detecting an operation to operate the mirror from a first captured image that includes the mirror. (Note 3) The state recognition device as described in Appendix 2, wherein the operation of the mirror is the operation of moving the mirror by hand or the operation of the mirror's operating interface. (Note 4) The state recognition device according to any one of the appendices 1 to 3, wherein the identifying means acquires a second image including the driver's head and identifies the driver's head position based on the coordinates of one or more points relating to the head included in the second image. (Note 5) The setting means is a state recognition device as described in Appendix 1, which outputs reference information including the identified head position. (Note 6) The aforementioned identification means identifies the driver's head position by a combination of coordinates of each of the multiple parts of the head, The state recognition device according to Appendix 5, wherein the setting means identifies the posture of the head based on the combination of coordinates and outputs the reference information which includes the identified posture of the head. (Note 7) The setting means is a state recognition device according to Appendix 5 or 6, which identifies the driver's identification information and outputs the reference information that includes the identified identification information. (Note 8) The setting means is, If the driver's head position is identified within a predetermined period, the reference information including that head position is output. The state recognition device described in Appendix 5, which outputs the reference information including a predetermined head position if the driver's head position is not determined within the predetermined period. (Note 9) The state recognition device described in Appendix 8, wherein the predetermined head position is the head position included in the reference information output in the past. (Note 10) A detection step that detects when the vehicle's mirrors have been operated, A step of identifying the position of the driver's head at the time the above operation was performed, A computer-based state recognition method comprising: a setting step of setting the identified head position to a reference position of the head in the vehicle. (Note 11) The state recognition method according to Appendix 10, wherein in the detection step, the mirror is detected to have been operated by detecting an operation to operate the mirror from a first captured image that includes the mirror. (Note 12) The state recognition method described in Appendix 11, wherein the operation of the mirror is an operation of moving the mirror by hand or an operation of the operation interface of the mirror. (Note 13) The state recognition method according to any one of appendices 10 to 12, wherein in the specified step, a second image is acquired that includes the driver's head, and the position of the driver's head is determined based on the coordinates of one or more points relating to the head included in the second image. (Note 14) The state recognition method described in Appendix 10, wherein in the setting step, reference information including the identified head position is output. (Note 15) In the aforementioned specific step, the position of the driver's head is determined by a combination of coordinates of multiple parts of the head, The state recognition method according to Appendix 14, wherein in the setting step, the posture of the head is identified based on the combination of coordinates, and the reference information including the identified posture of the head is output. (Note 16) The state recognition method according to Appendix 14 or 15, wherein in the setting step, the driver's identification information is identified and the reference information including the identified identification information is output. (Note 17) In the above setting step, If the driver's head position is identified within a predetermined period, the reference information including that head position is output. The state recognition method described in Appendix 14, which outputs the reference information including a predetermined head position if the driver's head position is not determined within the predetermined period. (Note 18) The state recognition method described in Appendix 17, wherein the predetermined head position is a head position included in the reference information output in the past. (Note 19) A detection step that detects when the vehicle's mirrors have been operated, A step of identifying the position of the driver's head at the time the above operation was performed, A non-temporary computer-readable medium containing a program that causes a computer to perform a setting step of setting the identified head position to the reference position of the head in the vehicle. (Note 20) The computer-readable medium according to Appendix 19, wherein in the detection step, the operation of the mirror is detected from the first captured image which includes the mirror, thereby detecting that the mirror has been operated. (Note 21) The operation of the mirror is the operation of moving the mirror by hand or the operation of the operation interface of the mirror, as described in Appendix 20, for a computer-readable medium. (Note 22) A computer-readable medium according to any one of appendices 19 to 21, wherein in the specified step, a second image including the driver's head is acquired, and the position of the driver's head is determined based on the coordinates of one or more points relating to the head included in the second image. (Note 23) A computer-readable medium as described in Appendix 19, which outputs reference information including the identified head position in the setting step. (Note 24) In the aforementioned specific step, the position of the driver's head is determined by a combination of coordinates of multiple parts of the head, A computer-readable medium according to Appendix 23, which in the setting step identifies the posture of the head based on the combination of coordinates and outputs the reference information including the identified posture of the head. (Note 25) A computer-readable medium according to Appendix 23 or 24, which in the setting step identifies the driver's identification information and outputs the reference information containing the identified identification information. (Note 26) In the above setting step, If the driver's head position is identified within a predetermined period, the reference information including that head position is output. A computer-readable medium as described in Appendix 23, which outputs the reference information including a predetermined head position if the driver's head position is not determined within the predetermined period. (Note 27) The predetermined head position is the head position included in the reference information output in the past, as described in Appendix 26, for the computer-readable medium. (Note 28) It has a state recognition device and an application device, The state recognition device is A detection means for detecting when a vehicle's mirror has been operated, A means for identifying the position of the driver's head at the time the aforementioned operation was performed, The system includes a setting means for outputting reference information that includes the identified head position, The aforementioned application device is An acquisition means for acquiring the aforementioned reference information, A task execution means that executes a task using the aforementioned reference information, A task execution system having output means for outputting output information relating to the execution results of the task. (Note 29) The task execution means determines whether the degree of deviation between the driver's current head position and the head position included in the reference information satisfies a predetermined condition. The output means generates output information indicating that the driver's condition is abnormal when the degree of deviation satisfies predetermined conditions, as described in Appendix 28, for the task execution system. (Note 30) The state recognition device, A detection step that detects when the vehicle's mirrors have been operated, A step of identifying the position of the driver's head at the time the above operation was performed, The setting step of outputting reference information that includes the identified head position is performed, The application device Acquisition step to acquire the aforementioned reference information, A task execution step in which a task is executed using the aforementioned reference information, A task execution method that performs an output step of outputting output information regarding the execution result of the aforementioned task. (Note 31) In the task execution step, it is determined whether the degree of discrepancy between the driver's current head position and the head position included in the reference information satisfies a predetermined condition. The task execution method according to Appendix 30, wherein in the output step, if the degree of deviation satisfies predetermined conditions, output information is generated to notify that the driver's condition is abnormal. [Explanation of symbols]

[0083] 10 vehicles 20 Mirror 30 Driver 40 Head position 50 heads 51 Right eye 52 Left eye 53 Nose 54 Right ear 55 Left ear 60. Standard Information 1000 computers 1020 Bus 1040 processor 1060 memory 1080 Storage Devices 1100 Input / Output Interface 1120 Network Interface 2000 State Recognition Device 2020 Detection Unit 2040 Specific section 2060 Settings Section 3000 Application Devices 3020 Acquisition Department 3040 Task Execution Unit 3060 Output Section 4000 Task Execution System

Claims

1. A detection means for detecting when a vehicle's mirror has been operated, A means for identifying the position of the driver's head at the time the aforementioned operation was performed, A state recognition device having setting means for setting the identified head position to a reference position of the head in the vehicle.

2. The state recognition device according to claim 1, wherein the detection means detects that the mirror has been operated by detecting an operation to operate the mirror from a first captured image that includes the mirror.

3. The state recognition device according to claim 2, wherein the operation of the mirror is an operation of moving the mirror by hand or an operation of the operation interface of the mirror.

4. The state recognition device according to any one of claims 1 to 3, wherein the identifying means acquires a second image including the driver's head, and identifies the driver's head position based on the coordinates of one or more points relating to the head included in the second image.

5. The state recognition device according to claim 1, wherein the setting means outputs reference information that includes the identified head position.

6. The aforementioned identification means identifies the driver's head position by a combination of coordinates of each of the multiple parts of the head, The state recognition device according to claim 5, wherein the setting means identifies the posture of the head based on the combination of coordinates and outputs the reference information which includes the identified posture of the head.

7. The state recognition device according to claim 5 or 6, wherein the setting means identifies the driver's identification information and outputs the reference information which includes the identified identification information.

8. A detection step that detects when the vehicle's mirrors have been operated, A step of identifying the position of the driver's head at the time the above operation was performed, A computer-based state recognition method comprising: a setting step of setting the identified head position to a reference position of the head in the vehicle.

9. A detection step that detects when the vehicle's mirrors have been operated, A step of identifying the position of the driver's head at the time the above operation was performed, A program that causes a computer to perform a setting step of setting the identified head position as the reference position of the head in the vehicle.

10. It has a state recognition device and an application device, The state recognition device is A detection means for detecting when a vehicle's mirror has been operated, A means for identifying the position of the driver's head at the time the aforementioned operation was performed, The system includes a setting means for outputting reference information that includes the identified head position, The aforementioned application device is An acquisition means for acquiring the aforementioned reference information, A task execution means that executes a task using the aforementioned reference information, A task execution system having output means for outputting output information relating to the execution results of the task.

Images