Inmate monitoring device, inmate monitoring procedure and inmate monitoring program

Abstract

Inmate monitoring device (310), with: a procurement unit (401) that procures a captured image obtained by imaging an area in which there is a probability that an occupant's (X) face is present in a vehicle (1); a determination unit (402) that determines whether the captured image obtained by the procurement unit (401) corresponds to at least a first image that does not include a feature section of the occupant's (X) face but includes at least part of the occupant's (X) body, or a second image that does not include the occupant's (X) body; a processing unit (404) that monitors a change in the state of the occupant's face (X) based on the captured image, and switches whether an occupant parameter set to monitor the change in the state of the occupant's face (X) is to be reset or not, according to a determination result in the determination unit (402), wherein the determination unit (402) determines whether the recorded image corresponds to the first image, the second image or a third image that includes the face of the occupant (X), In a case where the determination unit (402) determines that the captured image corresponds to the first image, the processing unit (404) maintains the occupant parameter without acquiring facial information regarding the occupant's face (X) based on the captured image, In a case where the determination unit (402) determines that the captured image corresponds to the second image, the processing unit (404) resets the occupant parameter without capturing facial information based on the captured image, and In a case where the determination unit (402) determines that the captured image corresponds to the third image, the processing unit (404) captures the facial information based on the captured image and updates the occupant parameter.

Description

TECHNICAL AREA

[0001] This disclosure relates to an inmate monitoring device, an inmate monitoring procedure, and an inmate monitoring program. BACKGROUND

[0002] The prior art includes a technique for monitoring changes in the state of an occupant's face based on a captured image obtained by photographing or imaging the occupant's face in a vehicle. This prior art occupant monitoring process involves capturing facial information about the occupant's face (and tracking the captured facial information) based on an occupant parameter that is adjusted as needed according to the captured facial information reflected in the image.

[0003] In the prior art, for example, in a case where an occupant is replaced or exchanged, it is useful to reset the occupant parameters and readjust them from an initial value to handle the replacement occupant. Therefore, a point in time for resetting the occupant parameters could be when a situation arises in which, for example, a feature area (the eyes, nose, or mouth) of the occupant's face is not captured in a recorded image.

[0004] However, the situation in which a facial feature of an occupant is not captured in a photograph could occur not only when an occupant is not present in the vehicle during a replacement, but also when an occupant is present in the vehicle in a position where a facial feature is not reflected in the photograph. In the latter case, since an occupant is not being replaced, a costly process for resetting the occupant parameters would be necessary, or the accuracy of the occupant monitoring process could be reduced.

[0005] Therefore, there is a need for an occupant monitoring device, an occupant monitoring procedure, and an occupant monitoring program that is capable of adequately performing a switch as to whether or not an occupant parameter is to be reset.

[0006] Publication JP 2005-327072A discloses an eye-tracking device comprising a position-relationship decision section for comparing an eye-nose distance obtained during tracking by mistakenly viewing a facial part other than the eyes of an object person as the eye, with an eye-nose distance determined from a complete image in a subsequent process. At the time of the above comparison, the position-relationship decision section uses a frequency distribution of the eye-nose distance generated by a frequency distribution generation section.The positional relationship decision section decides that the eye determined from the overall image is incorrect if the two above-mentioned eye-nose distances are located at similar positions in the distribution, and that the eye determined from the overall image is correctly determined if the two distances are located at different positions in the distribution.

[0007] German patent application DE 10 2007 023 140 A1 discloses a method in which a movement or sequence of movements of a driver or person is captured by a camera system and evaluated by an image processing unit for the purpose of recognizing or identifying the driver or person. This involves capturing and evaluating conscious or unconscious movements and sequences of movements, in particular gestures, of the driver or person.

[0008] Publication JP 2010 - 3 117 A discloses a device and a program that determine with high reliability whether an object is wearing an attached object such as a mask or sunglasses.

[0009] Publication JP 2018 - 60 466 A discloses an image processing device with a transformation parameter storage unit that stores transformation parameters calculated according to the position where a camera is installed to photograph seats, and an image conversion unit that converts the image photographed by the camera based on the transformation parameters and outputs the converted image, wherein the transformation parameters are parameters that convert the image so that the appearance of the seats in the image approximates a predetermined appearance. SUMMARY

[0010] An inmate monitoring device, as an example of this disclosure, comprises a procurement unit that obtains a recorded image, which is then used to create a replica orA capture of an area in which there is a probability that an occupant's face is present in a vehicle; a determination unit that determines whether the captured image, obtained by the acquisition unit, corresponds to at least a first image that does not include a feature segment of the occupant's face but includes at least part of the occupant's body, or a second image that does not include the occupant's body; and a processing unit that monitors a change in the state of the occupant's face based on the captured image and toggles whether an occupant parameter set to monitor the change in the state of the occupant's face should be reset or not, based on a determination result in the determination unit.

[0011] The occupant monitoring device can appropriately perform a switch as to whether an occupant parameter is to be reset or not, insofar as a recorded image corresponds to the first image or the second image, that is, whether there is a high probability that an occupant exchange may have occurred or not.

[0012] In the occupant monitoring device, the determination unit determines whether the captured image corresponds to the first image, the second image, or a third image that includes the occupant's face. If the determination unit determines that the captured image corresponds to the first image, the processing unit maintains the occupant parameter without capturing facial information about the occupant's face based on the captured image. If the determination unit determines that the captured image corresponds to the second image, the processing unit resets the occupant parameter without capturing facial information based on the captured image. If the determination unit determines that the captured image corresponds to the third image, the processing unit captures facial information based on the captured image and updates the occupant parameter.According to this configuration, it is possible to switch whether facial information is to be captured according to whether a captured image corresponds to the first image, the second image or the third image, and to switch whether an occupant parameter is to be held, reset or updated.

[0013] In this case, the procurement unit can successively acquire the captured image several times; in a case where the destination unit determines that the captured image corresponds to the third image, the processing unit can capture the facial information, update the occupant parameter, and then perform a facial information tracking based on the next captured image acquired by the procurement unit; and in a case where the destination unit determines that the captured image corresponds to the first or second image, the processing unit can hold or reset the occupant parameter without capturing the facial information, and then switch whether or not to capture the facial information and whether to reset, hold, or update the occupant parameter, according to a determination result in the destination unit with reference to the captured image.which is procured next by the procurement unit without performing facial tracking. According to this configuration, it is possible to appropriately switch whether facial tracking is performed, depending on whether a captured image corresponds to the third image with which facial tracking can be performed.

[0014] In the occupant monitoring device, with its identification unit that determines whether the captured image corresponds to any of the first, second, and third images, the acquisition unit can successively acquire the captured image multiple times. In cases where the identification unit determines that the captured image corresponds to the second image, and then the identification unit determines that the captured image corresponds to the third image, the processing unit can identify the occupant based on the captured image and execute a service pre-configured for each occupant based on the identification result. According to this configuration, in cases where there is a high probability of an occupant being replaced, an occupant is identified after a replacement, thus making it possible to execute an appropriate service for each occupant.

[0015] In the occupant monitoring system, the occupant parameter can include the position of the center of the occupant's eyeball within a three-dimensional model representing a structure that captures the three-dimensional shape of the occupant's face. According to this configuration, it is possible to appropriately switch whether or not the position of the occupant's eyeball center within a three-dimensional model is reset as an occupant parameter.

[0016] In the occupant monitoring system, the identification unit can determine whether the captured image, obtained by the acquisition unit, corresponds to at least the first or second image, based on a learned model generated by learning a training image containing information similar to the captured image. This training image can then be determined through machine learning. According to this configuration, it is possible to easily identify a captured image based on a trained model.

[0017] An occupant monitoring method, as another example of this disclosure, comprises a procurement step of obtaining a captured image obtained by imaging an area in which there is a probability that an occupant's face is present in a vehicle; a determination step of determining whether the captured image obtained in the procurement step corresponds to at least a first image that does not include a feature portion of the occupant's face but includes at least part of the occupant's body, or a second image that does not include the occupant's body; and a processing step of monitoring a change in the state of the occupant's face based on the first image and toggling whether or not to reset an occupant parameter set for monitoring the change in the state of the occupant's face.according to a determination result in the determination step. In the determination step, it is determined whether the captured image corresponds to the first image, the second image, or a third image that includes the occupant's face. If the determination step determines that the captured image corresponds to the first image, the occupant parameter is maintained in the processing step without capturing any facial information about the occupant's face based on the captured image. If the determination step determines that the captured image corresponds to the second image, the occupant parameter is reset in the processing step without capturing any facial information based on the captured image. If the determination step determines that the captured image corresponds to the third image,In the processing step, facial information is captured based on the recorded image, and the occupant parameter is updated.

[0018] According to the occupant monitoring procedure, it is possible to appropriately perform a switch as to whether an occupant parameter is to be reset or not, according to whether a recorded image corresponds to the first image or the second image, that is, whether there is a high probability that an exchange of an occupant may occur or not.

[0019] An occupant monitoring program, as another example of this disclosure, causes a computer to perform the following steps: a procurement step of obtaining a captured image obtained by mapping an area where there is a probability that an occupant's face is present in a vehicle; a determination step of determining whether the captured image obtained in the procurement step corresponds to at least a first image that does not include a feature portion of the occupant's face but includes at least part of the occupant's body, or a second image that does not include the occupant's body; and a processing step of monitoring a change in the state of the occupant's face based on the captured image, and toggling whether an occupant parameter set to monitor the change in the state of the occupant's face,Whether or not the parameter is to be reset depends on a determination result in the determination step. The determination step determines whether the captured image corresponds to the first image, the second image, or a third image that includes the occupant's face. If the determination step determines that the captured image corresponds to the first image, the occupant parameter is retained in the processing step without capturing any facial information related to the occupant's face, based on the captured image. If the determination step determines that the captured image corresponds to the second image, the occupant parameter is reset in the processing step without capturing any facial information, based on the captured image. If the determination step determines that the captured image corresponds to the third image,In the processing step, facial information is captured based on the recorded image, and the occupant parameter is updated.

[0020] According to the occupant monitoring program, it is possible to perform a switch appropriately, whether an occupant parameter is to be reset or not, according to whether a recorded image corresponds to the first image or the second image, that is, whether there is a high probability that an occupant exchange may occur or not. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The foregoing and additional features and characteristics of this disclosure will become more apparent from the following detailed description, which is considered in relation to the attached drawings, wherein: Fig. 1 is an exemplary and schematic diagram that represents an example of a vehicle configuration according to an embodiment; Fig. 2 is an exemplary and schematic diagram that represents an example of an arrangement of an imaging device according to the embodiment; Fig. 3 is an exemplary and schematic block diagram that represents a configuration of an occupant monitoring system according to the embodiment; Fig. 4 is an exemplary and schematic block diagram illustrating the functions of an occupant monitoring device according to the embodiment; Fig. 5 is an exemplary and schematic diagram that represents an example of a first image according to the embodiment; Fig. 6 is an exemplary and schematic diagram that represents an example of a second image according to the embodiment; Fig. 7 is an exemplary and schematic diagram that represents an example of a third image according to the embodiment; Fig. 8 is an exemplary and schematic flowchart that represents an initial detection process carried out by the occupant monitoring device according to the embodiment; Fig. 9 is an exemplary and schematic flowchart representing an individual identification process carried out by the occupant monitoring device according to the embodiment; and Fig. 10 is an exemplary and schematic flowchart that represents a tracking process carried out by the occupant monitoring device according to the embodiment shown. DETAILED DESCRIPTION

[0022] An embodiment of this disclosure is described below with reference to the drawings. Configurations of the embodiment described below, and operations and results (effects) resulting from the configurations, are merely examples and are not limited to the content described below.

[0023] First, with reference to the Fig. 1 and Fig. 2. A description of a schematic configuration of a vehicle 1 according to an exemplary embodiment has been provided. Fig. Figure 1 is an exemplary and schematic diagram that represents an example of a configuration of vehicle 1 according to the embodiment, and Fig. Figure 2 is an exemplary and schematic diagram that represents an example of an arrangement of an imaging device 201 according to the embodiment.

[0024] As in Fig. As shown in Figure 1, the vehicle 1 according to the exemplary embodiment is a four-wheeled automobile with two left and right front wheels 3F and two left and right rear wheels 3R. For the sake of simplicity, the front wheels 3F and the rear wheels 3R ​​will in some cases be referred to as vehicle wheels 3 below. In the exemplary embodiment, the slip angles of some or all of the four vehicle wheels 3 are changed (rotated) in response to steering by a steering unit 303a.

[0025] As in Fig. As shown in Figure 1, the vehicle 1 according to the exemplary embodiment comprises a vehicle cabin 2a in which an occupant (in Fig. 1 not shown). The steering unit 3a is provided in a state in which it is able to be controlled by a driver X (in Fig. (1 not shown) as being operated by an occupant in a driver's seat 2b in the vehicle cabin 2a. The steering unit 3a is configured with a steering wheel or a handle provided so that it protrudes from a dashboard (instrument panel) 12. It is not necessary to mention that in the exemplary embodiment, seats other than the driver's seat 2b may be present in the vehicle cabin 2a.

[0026] A monitor unit 11, comprising a display unit 8 that outputs various images and a sound output unit 9 that outputs various sounds, is provided in the vehicle cabin 2a. The monitor unit 11 is provided, for example, on a central section of the dashboard in the vehicle's width direction (left-right direction) in the vehicle cabin 2a. In the Fig. In the example shown, the display unit 8 is covered with an operation input unit that detects a coordinate of a position to which an indicator, such as a finger or a stylus, approaches (including contact). Accordingly, the occupant can visually recognize an image displayed on the display unit 8 and can input various operations by performing an input operation (for example, a touch operation) using an indicator on the operation input unit 10.

[0027] As in Fig. As shown in Figure 2, the imaging device 201 is provided on a support section 202, which supports the steering unit 303a. The imaging device 201 is configured, for example, with a CCD camera (CCD, charge-coupled device).

[0028] In this embodiment, the imaging device 201 is configured to image an area in which there is a probability that the face of driver X, sitting in driver's seat 2b, might be present. More precisely, the viewing angle and the posture or orientation of the imaging device 201 are adjusted so that the face of driver X, sitting in driver's seat 2b, is located in the center of a field of view.

[0029] The imaging device 201 periodically performs an imaging or recording of the driver X's face at a predetermined time interval while the vehicle 1 is being driven and outputs sequentially recorded images (corresponding image data) obtained by the imaging to an occupant monitoring device 310 (see Fig. 3 and Fig. 4), which is described later. As detailed below, the occupant monitoring device 310 performs an occupant monitoring process which includes acquiring facial information relating to the face of driver X (and tracking the acquired facial information) based on a captured image obtained by imaging in the imaging device 201, and thus monitors a change in the state of the face of driver X.

[0030] Although it is in Fig. Since Figure 2 is not shown, in the exemplary embodiment a lighting unit could be provided that illuminates an area imaged by the imaging device 201. In this case, if an infrared lamp that uses infrared light is used, it is possible to assist the imaging device 201 in its imaging without dazzling the driver X.

[0031] The following is a description of an occupant monitoring process that is executed based on a captured image obtained by the imaging device 201 and includes information regarding an area in which there is a probability that the face of driver X, seated in driver's seat 2b, may be present, but the occupant monitoring process can be executed with respect to an occupant other than driver X. In other words, the occupant monitoring process could be executed in a similar manner based on a captured image obtained by an imaging unit installed at a position different from the position of the imaging device 201 and captures an area in which there is a probability that the face of an occupant seated in a seat other than driver's seat 2b may be present.

[0032] Next, with reference to Fig. 3. A description of a configuration of an occupant monitoring system 300, which includes the occupant monitoring device 310 that performs the occupant monitoring process, was provided. The system configuration described in Fig. The figure shown in 3 is just an example and can be adjusted (changed) in various ways.

[0033] Fig. Figure 3 is an exemplary and schematic block diagram that illustrates a configuration of the occupant monitoring system 300 according to the embodiment shown. As in Fig. As shown in Figure 3, the occupant monitoring system 300 comprises a braking system 301, an acceleration system 302, a steering system 303, a gearshift system 304, an obstacle sensor 305, a driving condition sensor 306, the imaging device 201, the monitor device 11, the occupant monitoring device 310 and a vehicle-side network 350.

[0034] The braking system 301 controls a deceleration of the vehicle 1. The braking system 301 comprises a brake unit 301a, a brake control unit 301b and a brake unit sensor 301c.

[0035] The brake unit 301a is a device used to slow down the vehicle 1, such as a brake pedal.

[0036] The brake control unit 301b is an electronic control unit (ECU) configured with a computer that includes a hardware processor, such as a CPU. Based on an instruction input, for example via the vehicle network 350, the brake control unit 301b drives an actuator (not shown) and actuates the brake unit 301a to control the degree of deceleration of the vehicle 1.

[0037] The brake unit sensor 301c is a device used to detect the state of the brake unit 301a. For example, if the brake unit 301a is configured with a brake pedal, the brake unit sensor 301c detects the position of the brake pedal or the pressure applied to the brake pedal as a state of the brake unit 301a. The brake unit sensor 301c outputs the detected state of the brake unit 301a to the vehicle network 350.

[0038] The acceleration system 302 controls the acceleration of the vehicle 1. The acceleration system 302 comprises an acceleration unit 302a, an acceleration control unit 302b and an acceleration unit sensor 302c.

[0039] The acceleration unit 302a is a device used to accelerate the vehicle 1, such as an accelerator pedal.

[0040] The acceleration control unit 302b is an ECU configured with a computer that includes a hardware processor, such as a CPU. The acceleration control unit 302b drives an actuator (not shown) based on an instruction input, for example via the vehicle network 350, and actuates the acceleration unit 302a to control the degree of acceleration of the vehicle 1.

[0041] The accelerator unit sensor 302c is a device used to detect the state of the accelerator unit 302a. For example, if the accelerator unit 302a is configured with an accelerator pedal, the accelerator unit sensor 302c detects the position of the accelerator pedal or the pressure applied to the accelerator pedal. The accelerator unit sensor 302c outputs the detected state of the accelerator unit 302a to the vehicle network 350.

[0042] The steering system 303 controls a direction of movement or forward direction of the vehicle 1. The steering system 303 comprises the steering unit 302a, a steering control unit 303b and a steering unit sensor 303c.

[0043] The steering unit 303a is a device used to turn a steerable wheel of the vehicle 1, such as a steering wheel or a handle.

[0044] The steering control unit 303b is an ECU configured with a computer that includes a hardware processor, such as a CPU. Based on an instruction input, for example via the vehicle network 350, the steering control unit 303b drives an actuator (not shown) and actuates the steering unit 303a to control the direction of movement of the vehicle 1.

[0045] The steering unit sensor 303c is a device used to detect the state of the steering unit 303a. For example, if the steering unit 303a is configured with a steering wheel, the steering unit sensor 303c detects the position or angle of rotation of the steering wheel. If the steering unit 303a is configured with a handle, the steering unit sensor 303c could detect the position of the handle or the pressure applied to the handle. The steering unit sensor 303c outputs the detected state of the steering unit 303a to the vehicle network 350.

[0046] The gearshift system 304 controls a gear ratio of the vehicle 1. The gearshift system 304 comprises a gearshift unit 304a, a gearshift control unit 304b, and a gearshift unit sensor 304c. The gearshift unit 304a is a device used to change a gear ratio of the vehicle 1, such as a gearshift lever.

[0047] The gearshift control unit 304b is an ECU configured with a computer that includes a hardware processor, such as a CPU. Based on an instruction input, for example via the vehicle network 350, the gearshift control unit 304b drives an actuator (not shown) and actuates the gearshift unit 304a to control a gear ratio of the vehicle 1.

[0048] The gearshift unit sensor 304c is a device used to detect the state of the gearshift unit 304a. For example, if the gearshift unit 304a is configured with a shift lever, the gearshift unit sensor 304c detects the position of the shift lever or the pressure applied to the shift lever. The gearshift unit sensor 304c outputs the detected state of the gearshift unit 304a to the vehicle network 350.

[0049] The obstacle sensor 305 is a device used to acquire information about an obstacle that might be present in the periphery of the vehicle 1. The obstacle sensor 305 includes a distance measuring sensor, such as sonar, that detects the distance to an obstacle. The obstacle sensor 305 outputs the acquired information to the vehicle network 350.

[0050] The driving condition sensor 306 is a device used to detect the driving condition of the vehicle 1. The driving condition sensor 306 includes, for example, a vehicle wheel sensor that detects the rotational speed of the vehicle 1, an acceleration sensor that detects the acceleration of the vehicle 1 in a longitudinal or lateral direction, or a gyro sensor that detects the rotational speed (angular velocity) of the vehicle 1. The driving condition sensor 306 outputs the detected driving condition to the vehicle network 350.

[0051] The occupant monitoring device 310 is a device that holistically controls the occupant monitoring system 300. The occupant monitoring device 310 performs an occupant monitoring process, including the acquisition of facial information relating to the face of driver X (and the tracking of the acquired facial information) based on a captured image obtained by the imaging device 201, and thus monitors any change in the state of the face of driver X. The facial information consists of data that includes eye information relating to the eyes of driver X, mouth information relating to the mouth of driver X, and expression information relating to an expression of driver X.The eye information is data that indicates an open or closed state of the eyes (eyelids) or a direction of a line of sight, and the mouth information is data that indicates an open or closed state (conversation state) of the mouth.

[0052] The occupant monitoring device 310 is configured with an ECU comprising a central processing unit (CPU) 310a, a read-only memory (ROM) 310b, a random access memory (RAM) 310c, a solid state drive (SSD) 310d, a display control unit 310e and a sound control unit 310f.

[0053] The CPU 310a is a hardware processor that comprehensively controls the 310 occupant monitoring device. The CPU 310a reads various control programs (computer programs) stored in the ROM 310b or similar and executes various functions according to instructions defined in these programs. The various control programs include an occupant monitoring program for implementing an occupant monitoring process.

[0054] The ROM 310b is a non-volatile main memory device that stores parameters or similar information required to run the various control programs.

[0055] The RAM 310c is a volatile main memory device that provides a workspace for the CPU 310a.

[0056] The SSD 310d is a rewritable non-volatile auxiliary storage device. In the inmate monitoring device 310 according to the exemplary embodiment, a hard disk drive (HDD) could be provided as an auxiliary storage device instead of the SSD 310d (or in addition to the SSD 310b).

[0057] Among the various processes performed by the occupant monitoring device 310, the display control unit 310e mainly performs image processing with respect to a captured image obtained from the imaging device 201, or generates image data to be output to the display unit 8 of the monitor device 11.

[0058] Among the various processes carried out by the occupant monitoring device 310, the sound control unit 310f mainly generates sound data to be output to the sound output unit 9 of the monitor device 11.

[0059] The vehicle-side network 350 connects the braking system 301, the acceleration system 302, the steering system 303, the gearshift system 304, the obstacle sensor 305, the driving condition sensor 306, the operation input unit 10 of the monitor device 11 and the occupant monitoring device 310 communicatively with each other.

[0060] Meanwhile, in the prior art, an occupant monitoring process is executed based on an occupant parameter, which is adjusted as needed according to the capture result of an occupant's face, reflected in a captured image. Examples of occupant parameters can include the position (a coordinate in a predetermined coordinate system) of the center of an occupant's eyeball in a three-dimensional model representing a structure including a three-dimensional shape of the occupant's face, and the positions (coordinates in a predetermined coordinate system) of a multitude of points to specify the three-dimensional model.

[0061] In the prior art, for example, in a case where an occupant is exchanged or replaced, it is useful to reset the occupant parameter and readjust it from an initial value in order to manage the occupant after the exchange. Therefore, a point in time for resetting the occupant parameter could be a point in time when a situation arises in which, for example, a feature area (the eyes, nose, or mouth) of the occupant's face is not captured by a recorded image.

[0062] However, the situation in which a facial feature of the occupant is not captured in a recorded image occurs not only when the occupant is absent from vehicle 1 during a replacement, but also when the occupant is present in vehicle 1 but is located in a position where a facial feature is not reflected in the recorded image. In the latter case, since an occupant is not replaced, a complex process to readjust the occupant parameter might be necessary, or the accuracy of an occupant monitoring process could be reduced if the occupant parameter is reset.

[0063] Therefore, in the exemplary embodiment, the occupant monitoring device 310 includes functions that are in Fig. 4 are shown, and thus a switch to whether an occupant parameter is to be reset or not is performed appropriately.

[0064] Fig. Figure 4 is an exemplary and schematic block diagram illustrating the functions of the occupant monitoring device 310 according to the embodiment shown. Fig. The four functions shown are implemented through cooperation between software and hardware. In other words, in the example shown in Fig. As shown in Figure 4, the functions of the occupant monitoring device 310 are realized as a result of the CPU 310a reading and executing a predetermined control program stored in the RAM 310b. In this embodiment, at least some of the functions described in Figure 4 can be implemented as follows: Fig. The elements shown in Figure 4 are implemented using dedicated hardware (circuit).

[0065] As in Fig. As shown in Figure 4, the occupant monitoring device 310 according to the embodiment comprises a procurement unit 401, a determination unit 402 and a processing unit 404.

[0066] Procurement unit 401 successively acquires images obtained by imaging device 201 several times at predetermined control times. As described above, the acquired image includes information regarding an area in which there is a probability that the face of driver X could be present in vehicle 1.

[0067] The determination unit 402 determines whether a captured image, procured by the procurement unit 401, corresponds to a first image that does not include a feature section (the eyes, the nose, or the mouth) of the face but includes at least part of the body, a second image that includes neither a feature section of the face nor the body, or a third image that includes a feature section of the face.

[0068] More precisely, the determination unit 402 comprises a learned model 403, generated as a result of machine learning, which determines whether the captured image corresponds to one of the first, second, or third images, based on the learned model 403. The learned model 403 can be generated, for example, through supervised learning, in which labels corresponding to the first, second, and third images are added to a variety of training images, and combinations of the training images and the labels are used as training data. It is assumed that the training image is an image captured under the same conditions as, for example, a captured image, and contains the same information as the captured image.In the embodiment, machine learning to generate the learned model 403 is not limited to supervised learning and could be unsupervised learning, reinforcement learning, or a combination of one or more of the three learning schemes.

[0069] Here, a brief description of specific examples from the first image, the second image, and the third image is given.

[0070] Fig. Figure 5 is an exemplary and schematic diagram that illustrates an example of the first image according to the embodiment. Figure 500, which is shown in Fig. The image shown in Figure 5 does not include any feature of the driver X's face, but it does include at least part of the body and is thus an example of the first image. More precisely, Figure 500 depicts a situation in which the driver X is sitting in driver's seat 2b, part of the body including the driver X's hand H is reflected, but a feature of the face is obscured by the hand H and is therefore not reflected.

[0071] Since facial information regarding the face of driver X is not available from the first image, such as image 500, which is in Fig. Since the facial features shown in image 5 cannot be captured, image 500 cannot be used for an occupant monitoring process including the capture (and tracking) of facial information, but it can capture that the driver X is present in the vehicle. For convenience, the first image will in some cases be referred to as a covert image or obscuration image.

[0072] Fig. Figure 6 is an exemplary and schematic diagram that illustrates an example of a second image according to the embodiment. Figure 600, which is shown in Fig. The image shown in Figure 6 does not include the driver X's face or any part of his body, and is therefore an example of the second image. More precisely, Figure 600 depicts a situation in which the driver X is not fully reflected and only the driver's seat 2b is reflected.

[0073] Since facial information regarding the face of driver X is not available from the second image, such as image 600, which is in Fig. Since the information shown in image 6 cannot be captured, image 600 cannot be used for an occupant monitoring process, including the capture of facial information (and the tracking of the captured facial information). In other words, since neither facial information of driver X nor the presence of driver X in the vehicle is captured by the second image, it can be said that the second image represents a situation in which driver X is not present in the vehicle due to an exchange or similar event. For convenience, the second image will also be referred to as an absence image in some instances below.

[0074] Fig. Figure 7 is an exemplary and schematic diagram that illustrates an example of a third image according to the embodiment. Figure 700, which is shown in Fig. The image shown in Figure 7 comprises a feature section of driver X's face and is thus an example of the third image. More precisely, Figure 700 depicts a situation in which driver X's entire face is exposed.

[0075] Facial information of driver X can be obtained from the third image, such as image 700, which is in Fig. The image shown in Figure 7 can be captured. Therefore, the third image can be used for an occupant monitoring process, including the capture of facial information (and the tracking of the captured facial information). For convenience, the third image will in some cases be referred to below as an image that enables capture or a capture-enabling image.

[0076] Below, in a case where a captured image obtained by Procurement Unit 401 corresponds to an absence image, it is determined that there is a high probability that driver X could be replaced, and therefore it is appropriate to reset an occupant parameter.

[0077] On the other hand, in a case where a captured image obtained by procurement unit 401 corresponds to an obscuration image, facial information cannot be captured, but driver X is not replaced, and consequently it can be determined that facial information of an identical person is successively captured, and thus it is appropriate to maintain an occupant parameter without resetting it.

[0078] In a case where the captured image obtained by procurement unit 401 corresponds to a capture enabling image, the most up-to-date facial information can be captured from the capture enabling image, and thus it is appropriate to update an occupant parameter based on the most up-to-date facial information.

[0079] As mentioned above, there is a difference in an appropriate procedure for handling an occupant parameter depending on whether a captured image obtained by Procurement Unit 401 corresponds to an absence image, a cover image, or a capture-enabling image.

[0080] Therefore, with reference to Fig. 4. Processing unit 404 switches whether an occupant parameter, set to monitor a change in the state of the driver's face X, should be reset or not, according to a determination result in the determination unit. Functions of processing unit 404 are described in more detail below.

[0081] The processing unit 404 comprises functional modules that correspond to two control modules, such as an initial acquisition mode for the first acquisition of facial information from a recorded image as an initial step in the occupant monitoring process, and a tracking mode for tracking the facial information acquired in the initial acquisition process.

[0082] In other words, the processing unit 404 comprises an initial capture processing section 405, which executes an initial capture process as a process according to the initial capture operating mode, and a tracking processing section 406, which executes a tracking process as a process according to the tracking operating mode.

[0083] For example, in a case where the procurement unit 401 procures a captured image, the control operating mode of the processing unit 404 is set to the initial acquisition operating mode.

[0084] As described above, in a case where a captured image corresponds to a capture-enabling image, facial information of the driver X can be captured. Therefore, in the exemplary embodiment, in a case where a captured image corresponds to a capture-enabling image, the initial capture processing section 405 of the processing unit 404 captures facial information and also updates an occupant parameter based on the captured image and the current occupant parameter.

[0085] More precisely, in the exemplary embodiment, in a case where a captured image corresponds to the capture enabling image, the initial capture processing section 405 of the processing unit 404 appropriately adjusts an occupant parameter, which represents a three-dimensional model obtained as data representing a structure including a three-dimensional shape of the face, to match feature points of the capture enabling image, thus capturing facial information and also updating the occupant parameter.

[0086] In a case where the facial information is normally acquired by the initial acquisition processing section 405, the control operating mode of the processing unit 404 is set to the tracking operating mode.

[0087] The tracking processing section 406 of processing unit 404 performs tracking of the facial information acquired in the initial capture process, based on a captured image that is subsequently obtained and the updated occupant parameter.

[0088] On the other hand, as described above, in a case where a captured image corresponds to an obscuration image or an absence image, facial information of driver X cannot be captured. Therefore, in the exemplary embodiment, in a case where a captured image corresponds to an obscuration image or an absence image, the initial capture processing section 405 of the processing unit 404 does not capture facial information and does not update the occupant parameter.

[0089] More precisely, in the embodiment, the initial acquisition processing section 405 of the processing unit 404 holds an occupant parameter without acquiring facial information in a case where a captured image corresponds to an occlusion image, and resets the occupant parameter without acquiring facial information in a case where the captured image corresponds to an absence image.

[0090] As mentioned above, in a case where a captured image corresponds to an obscuration image or an absence image, facial information that is a tracking target is not captured in the initial capture process. Therefore, in this case, processing unit 404 causes the initial capture processing section 405 to execute the initial capture process again, without the tracking processing section 406 performing any tracking of facial information after the initial capture processing section 405 has completed the initial capture process. In other words, in this case, processing unit 404 toggles whether or not to capture facial information and toggles whether to hold, reset, or update occupant parameters, according to a determination result in determination unit 402 with respect to a captured image that is acquired next.

[0091] As described above, captured images are successively acquired multiple times by Procurement Unit 401. In this configuration, the point in time at which a captured image changes from an absence image to a capture-enabling image corresponds to an initial point in time at which driver X is replaced, and thus a new driver X can be clearly identified based on the captured image. In a case where an individual recognition process is performed at this point in time, it is possible to appropriately provide a service that is pre-configured for each driver X (driver-specific), such as setting an audio device or air conditioning, or adjusting the position of the driver's seat 2b, to the identified new driver X.

[0092] Therefore, in this embodiment, the processing unit 404 includes an individual recognition processing section 407, which performs the individual recognition process. In a case where the determination unit 402 determines that a captured image corresponds to an absence image, and then determines for the first time that the captured image corresponds to a capture-enabling image, the individual recognition processing section 407 identifies the driver X based on the captured image and performs, as the individual recognition process, a process of executing a service that is preset according to an identification result. The identification of the driver X is performed based on the identification information of an individual, which can be obtained from the captured image by, for example, an image recognition process.

[0093] The initial capture process, the individual recognition process, and the tracking process are described in detail below with reference to flowcharts, according to the exemplary implementation.

[0094] First, the initial data collection process is described in detail.

[0095] Fig. Figure 8 is an exemplary and schematic flowchart illustrating an initial acquisition process performed by the occupant monitoring device 310 according to this embodiment. The initial acquisition process is executed, for example, in a case where the control operating mode of the processing unit 404 is set to the initial acquisition operating mode and a captured image is obtained by the acquisition unit 401.

[0096] As in Fig. As shown in Figure 8, in the initial acquisition process, the initial acquisition processing section 405 first captures a position of a face in the captured image in step S801.

[0097] In S802, the initial acquisition processing section 405 detects a direction of a face in the captured image.

[0098] In S803, the identification unit 402 determines the captured image.

[0099] In S804, the initial acquisition processing section 405 determines whether the captured image should be matched to an acquisition enabling image, such as image 700, which is in Fig. 7 is shown, corresponds or does not correspond, based on a determination result in S803.

[0100] In a case where S804 determines that the captured image corresponds to the capture-enabling image, the process proceeds to S805. In S805, the individual recognition processing section 407 executes an individual recognition process, which is Fig. 9 is shown.

[0101] Fig. Figure 9 is an exemplary and schematic flowchart representing an individual identification process performed by the occupant monitoring device 310 according to the embodiment. As described below, the individual identification process is a process of executing a service that is preset for each driver X according to a driver X exchange.

[0102] As in Fig. As shown in S901, the individual recognition processing section 407 in the individual recognition process first determines whether the preceding determination process, which was carried out by the determination unit 402 (see S803, which is shown in S901), has been completed. Fig. 8 is shown), is determined or not, that a recorded image corresponds to an absence image.

[0103] In a case where S901 determines that the captured image was not identified as corresponding to an absence image in the preceding determination process, it could be determined that driver X has not been replaced. Therefore, in this case, the process terminates without executing an identification of driver X, and the process in S805, which is in Fig. The period shown in section 8 will end.

[0104] On the other hand, in a case where S901 determines that the captured image was determined in the preceding determination process to correspond to an absence image, the current time can be determined as an initial time at which driver X is replaced, and the new driver X can be clearly identified based on the captured image. Therefore, in this case, the process proceeds to S902, and in S902, the individual recognition processing section 407 performs an image recognition process or similar with respect to the captured image to specify the new driver X and obtains identification information to identify an individual from the captured image.

[0105] In S903, the individual identification processing section 407 decides or determines a service that is pre-configured for driver X, following an exchange according to the identification information obtained in S902. For example, it is assumed that elements of identification information of a multitude of drivers X and services that are pre-configured for the multitude of corresponding drivers X (driver-specific) are stored in the occupant monitoring device 310 in a state in which they are correlated with each other.

[0106] In S904, the individual recognition processing section 407 performs the service specified in S903. Consequently, a service such as setting an audio device or air conditioning system, or adjusting the position of the driver's seat 2b, with suitable content can be provided to the driver X after an exchange.

[0107] In a case where the process in S904 is completed, the process in S805, which is in Fig. As shown in 8, it is finished.

[0108] Referring again to Fig. 8. The initial acquisition processing section 408 in S806 performs a fitting between the captured image and a three-dimensional model, which is obtained as data representing a structure including a three-dimensional shape of the face, while the currently held occupant parameters are adjusted, and captures the position and direction of the face contained in the captured image in more detail. The fitting in the initial acquisition process is performed, for example, based on an initial model that is preset as statistical data representing a structure including a three-dimensional shape of the face.

[0109] In S807, the initial acquisition processing section 405 captures facial information regarding the face of driver X based on a result of the matching in S806. As described above, the facial information is data that includes eye information (opening or closing state of eyelids or direction of a line of sight) regarding the eyes of driver X, mouth information (conversation state) regarding the mouth of driver X, and expression information regarding an expression of driver X.

[0110] In S808, the Initial Capture Processing Section 405 determines whether the facial information was captured normally. For example, Initial Capture Processing Section 405 calculates a score that indicates the reliability of each of the processes in S805 through S807 as a set of processes that were recently executed to capture the facial information, and performs the S808 determination based on this score.

[0111] In a case where S808 determines that the facial information was captured normally, the process proceeds to S809. In S809, the initial capture processing section 405 updates the occupant parameter to reflect the result of the matching in S806.

[0112] In S810, the initial acquisition processing section 405 sets the next control operating mode of the processing unit 404 to the tracking operating mode. The process is terminated.

[0113] On the other hand, in a case where S808 determines that the facial information was not captured normally, the process proceeds to S811. In S811, the initial capture processing section 405 holds the occupant parameters without updating them.

[0114] In S812, the initial acquisition processing section 405 sets the next control operating mode of the processing unit 404 to the initial acquisition operating mode. The process is terminated.

[0115] In a case where S804 determines that the captured image does not correspond to a capture-enabling image, the process proceeds to S813. In S813, the initial capture processing section 405 determines whether the captured image is an occlusion image, such as image 500, which is in Fig. 5 is shown, based on the determination result in S803.

[0116] In a case where S813 determines that the captured image corresponds to an occlusion image, facial information is not captured and occupant parameters are not updated. Therefore, in this case, the process proceeds to S811. The processes in S811 and the subsequent steps have already been described, and thus a description of them is omitted here.

[0117] On the other hand, in a case where S813 determines that the captured image does not correspond to the occlusion image, the process proceeds to S814. In S814, the initial capture processing section 405 determines whether the captured image corresponds to an absence image, such as image 600, which is in Fig. Figure 6 is shown, based on the determination result in S803.

[0118] In a case where S814 determines that the captured image corresponds to an absence image, the occupant parameter is reset without facial information being acquired. Therefore, in this case, the process proceeds to S815, and in S815, the initial acquisition processing section 405 resets the occupant parameter to an initial value.

[0119] In a case where the process in S815 is completed, the process proceeds to S812, and, in a case where it is determined in S814 that the captured image does not correspond to an absence image, the process also proceeds to S812, but the processes in S812 and the subsequent steps have already been described, and thus a description is omitted here.

[0120] The tracking process will be described in detail next.

[0121] Fig. Figure 10 is an exemplary and schematic flowchart illustrating a tracking process performed by the occupant monitoring device 310 according to the embodiment. The tracking process is shown, for example, in a case where the control operating mode of the processing unit 404 is set to the tracking operating mode and a captured image is obtained by the acquisition unit 401.

[0122] As in Fig. As shown in Figure 10, in the tracking process, tracking processing section 406 in S1001 first performs tracking of a face's position in the captured image. Tracking can be used to restrict a retrieval area, to use a template for matching, or to specify a modified section based on a difference between successive frames, using the various capture results from the initial capture process. Therefore, tracking generally has a higher processing speed than the various captures in the initial capture process.

[0123] In S1002, the tracking processing section 406 performs tracking of a direction of the face in the captured image.

[0124] In S1003, the tracking processing section 406 performs a comparison between the captured image and a three-dimensional model while adjusting the currently held occupant parameters. This comparison in the tracking process is performed, for example, based on the most recent three-dimensional model after it has been adjusted in the preceding initial capture or tracking process.

[0125] In S1004, the tracking processing section 406 performs a tracking of facial information captured in the initial capture process, based on a result of the matching in S1003.

[0126] In S1005, the tracking processing section 406 determines whether the facial information tracking was performed normally. For example, tracking processing section 406 calculates a score that indicates the reliability of each of the processes that were recently performed to track the facial information and performs the determination in S1005 based on the score.

[0127] In a case where S1005 determines that facial information tracking was performed normally, the process proceeds to S1006. In S1006, tracking processing section 406 updates the occupant parameters to reflect the result of the matching in S1003.

[0128] In S1007, the tracking processing section 406 sets the next control operating mode of the processing unit 404 to the tracking operating mode. The process is terminated.

[0129] On the other hand, in a case where S1005 determines that facial information tracking was not performed normally, the process proceeds to S1008. In S1008, tracking processing section 406 retains the occupant parameters without updating them.

[0130] In S1009, the tracking processing section 406 sets the next control operating mode of the processing unit 404 to the initial acquisition operating mode. The process is terminated.

[0131] As described above, the occupant monitoring device 310, according to the exemplary embodiment, comprises the acquisition unit 401, the determination unit 402, and the processing unit 404. The acquisition unit 404 acquires a captured image obtained by imaging an area in which there is a probability that the face of the driver X could be present as an occupant in the vehicle 1. The determination unit 402 determines whether the captured image obtained by the acquisition unit 401 matches at least one of an obscuration image, such as image 500, which is shown in Fig. 5 is shown, that is, the first image, which does not include a feature section of the driver X's face but includes at least a part of the driver X's body, and an absence image, such as image 600, which is shown in Fig. Figure 6 is shown, that is, it corresponds to the second image, which does not include the body of driver X. The processing unit 404 monitors a change in the state of an occupant's face based on a captured image and switches whether an occupant parameter, which is set to monitor a change in the state of driver X's face, should be reset or not, according to a determination result in the determination unit 402.

[0132] The occupant monitoring device 310 according to the embodiment can appropriately perform a switch as to whether the occupant parameter is to be reset or not, according to whether a recorded image corresponds to an obscuration image or an absence image, that is, whether there is a high probability that a replacement of the driver X will occur or not.

[0133] In the occupant monitoring device 310 according to the exemplary embodiment, the determination unit 402 determines whether a recorded image is any of the following: a concealment image, an absence image, and a detection-enabling image, such as the image 700, which is in Fig.Figure 7 is shown, that is, it corresponds to the third image, which includes an occupant's face. In a case where the determination unit 402 determines that the captured image corresponds to an obscuration image, the processing unit 404 retains the occupant parameter without capturing facial information regarding the driver X's face based on the captured image. In a case where the determination unit 402 determines that the captured image corresponds to an absence image, the processing unit 404 resets the occupant parameter without capturing facial information based on the captured image. In a case where the determination unit 402 determines that the captured image corresponds to a capture-enabling image, the processing unit 404 captures facial information based on the captured image and updates the occupant parameters.According to this configuration, it is possible to switch whether facial information is to be captured or not, according to whether a captured image corresponds to an obscuration image, an absence image or a capture enabling image, and to switch whether an occupant parameter is to be held, reset or updated.

[0134] In the occupant monitoring device 310 according to the exemplary embodiment, the procurement unit 401 successively obtains captured images several times. In a case where the determination unit 402 determines that a captured image corresponds to a capture-enabling image, the processing unit 404 acquires facial information, updates the occupant parameters, and then performs a facial information tracking based on the next captured image obtained by the procurement unit 401.In a case where the determination unit 402 determines that the captured image corresponds to an obscuration image or an absence image, the processing unit 404 holds the occupant parameters without capturing facial information or resets the occupant parameters, and then switches whether or not to capture facial information and whether or not to reset or update the occupant parameters, according to a determination result in the determination unit 402 with respect to a captured image that is next acquired by the acquisition unit 401, without performing facial information tracking. According to this configuration, it is possible to appropriately switch whether or not to perform facial information tracking, according to whether a captured image corresponds to a capture-enabling image with which facial information tracking can be performed.

[0135] In the occupant monitoring device 310 according to the exemplary embodiment, in a case where the determination unit 402 determines that a captured image corresponds to an absence image, and then the determination unit 402 determines for the first time that the captured image corresponds to a capture-enabling image, the processing unit 404 identifies the driver X based on the captured image and performs a service that is pre-configured for each driver X based on an identification result. According to this configuration, in a case where there is a high probability that a substitution of driver X may occur, the driver X is identified after a substitution, and thus it is possible to perform an appropriate service for each driver X.

[0136] In the occupant monitoring device 310 according to the exemplary embodiment, the occupant parameter comprises the position of the center of the driver X's eyeball in a three-dimensional model, which represents a structure including a three-dimensional shape of the driver X's face. According to this configuration, it is possible to appropriately switch whether or not the position of the center of the driver X's eyeball in the three-dimensional model is to be reset as the occupant parameter.

[0137] In the occupant monitoring device 310 according to the exemplary embodiment, the determination unit 402 determines whether a captured image, acquired by the acquisition unit 401, corresponds to a first image or a second image, based on the learned model 403, which is generated by learning a training image containing information similar to the captured image, and whether the training image corresponds to the first image or the second image, by machine learning. According to this configuration, it is possible to easily determine a captured image based on the learned model 403.

[0138] In this embodiment, the occupant parameter is not limited to a parameter specifying a structure, including the shape of an occupant's face, such as the position of the center of the driver X's eyeball in a three-dimensional model. In other words, the occupant parameters can include a parameter that has no direct relationship to an occupant's face, such as gender or age, as long as the occupant parameter is a parameter set to monitor a change in the state of the occupant's face.

[0139] An occupant monitoring program running in the occupant monitoring device 310 according to the exemplary embodiment can be provided or distributed via a network, such as the Internet. In other words, the occupant monitoring program running in the occupant monitoring device 310 according to the exemplary embodiment could be provided in a form that allows it to be downloaded over a network and stored on a computer connected to the network, such as the Internet.

[0140] As mentioned above, the embodiment of this disclosure has been described, but this embodiment is only an example and is not intended to limit the scope of this disclosure. The new embodiment can be implemented in various forms, and various omissions, substitutions, and modifications can be made within the scope without departing from the concept of this disclosure. The embodiment and modifications thereof are within the scope or concept of this disclosure and are also within the invention disclosed in the claims and their equivalents.

[0141] The principles, preferred embodiment, and mode of operation of the present invention have been described in the preceding specification. However, the invention to be protected is not to be interpreted as being limited to certain embodiments disclosed herein. Furthermore, the embodiments described herein are to be understood as illustrative and not limiting. Variations and modifications may be made by others, and equivalents may be used, without departing from the spirit of the invention. Accordingly, it is expressly intended that all such variations, modifications, and equivalents that fall within the spirit and scope of the present invention as defined by the claims are included herein.

[0142] An occupant monitoring device (310) comprises: a procurement unit (401) that obtains a captured image obtained by imaging an area in which there is a probability that an occupant's face (X) is present in a vehicle (1); a determination unit (402) that determines whether the captured image obtained by the procurement unit corresponds to at least a first image that does not include a feature segment of the occupant's face but includes at least part of the occupant's body, or a second image that does not include the occupant's body; and a processing unit (404) that monitors a change in the state of the occupant's face based on the captured image and toggles whether an occupant parameter set to monitor the change in the state of the occupant's face is to be reset or not, according to a determination result in the determination unit.

Claims

[1] Inmate monitoring device (310), with: a procurement unit (401) that procures a captured image obtained by imaging an area in which there is a probability that an occupant's (X) face is present in a vehicle (1); a determination unit (402) that determines whether the captured image obtained by the procurement unit (401) corresponds to at least a first image that does not include a feature section of the occupant's (X) face but includes at least part of the occupant's (X) body, or a second image that does not include the occupant's (X) body; a processing unit (404) that monitors a change in the state of the occupant's face (X) based on the captured image, and switches whether an occupant parameter set to monitor the change in the state of the occupant's face (X) is to be reset or not, according to a determination result in the determination unit (402), wherein the determination unit (402) determines whether the recorded image corresponds to the first image, the second image or a third image that includes the face of the occupant (X), In a case where the determination unit (402) determines that the captured image corresponds to the first image, the processing unit (404) maintains the occupant parameter without acquiring facial information regarding the occupant's face (X) based on the captured image, In a case where the determination unit (402) determines that the captured image corresponds to the second image, the processing unit (404) resets the occupant parameter without capturing facial information based on the captured image, and In a case where the determination unit (402) determines that the captured image corresponds to the third image, the processing unit (404) captures the facial information based on the captured image and updates the occupant parameter. [2] Occupant monitoring device (310) according to claim 1, wherein the procurement unit (401) procured the recorded image several times successively, in a case where the determination unit (402) determines that the captured image corresponds to the third image, the processing unit (404) captures the facial information, updates the occupant parameter, and then performs a tracking of the facial information based on the captured image that is next acquired by the procurement unit (401), and In a case where the determination unit (402) determines that the captured image corresponds to the first image or the second image, the processing unit (404) holds or resets the occupant parameter without capturing the face information, and then switches whether to capture the face information or not, and whether to reset, hold or update the occupant parameter, according to a determination result in the determination unit (402) with reference to the captured image that is next obtained by the procurement unit (401), without executing any tracking of the face information. [3] Occupant monitoring device (310) according to claim 1 or 2, wherein the procurement unit (401) successively procured the recorded image several times, and In a case where the Identification Unit (402) determines that the captured image corresponds to the second image, and then the Identification Unit (402) determines for the first time that the captured image corresponds to the third image, the Processing Unit (404) identifies the occupant (X) based on the captured image and performs a service that is pre-configured for each occupant (X) based on an identification result. [4] Occupant monitoring device (310) according to one of claims 1 to 3, wherein the occupant parameter comprises a position of the center of an occupant's eyeball (X) in a three-dimensional model that represents a structure including a three-dimensional shape of the occupant's face (X). [5] Occupant monitoring device (310) according to one of claims 1 to 4, wherein the determination unit (402) determines whether the recorded image obtained by the procurement unit (401) corresponds at least to the first image or the second image, based on a learned model (403) generated by learning a learning image including information similar to the recorded image, and whether the learning image corresponds at least to the first image or the second image, by machine learning. [6] Inmate monitoring procedures, including: a procurement step of obtaining a recorded image obtained by mapping an area in which there is a probability that an occupant's face (X) is present in a vehicle (1); a determination step of determining whether the recorded image, which is procured in the procurement step, corresponds to at least a first image that does not include a feature segment of the occupant's (X) face but includes at least a part of the occupant's (X) body, or to a second image that does not include the body of the occupant (X); and a processing step of monitoring a change in the state of the occupant's face (X) based on the captured image and switching whether an occupant parameter set to monitor the change in the state of the occupant's face (X) is to be reset or not, according to a determination result in the determination step, wherein In the determination step, it is determined whether the recorded image corresponds to the first image, the second image, or a third image that includes the face of the occupant (X). in a case where, in the determination step, it is determined that the recorded image corresponds to the first image, in the processing step the occupant parameter is maintained without capturing facial information regarding the occupant's face (X) based on the recorded image, in a case where, in the determination step, it is determined that the recorded image corresponds to the second image, in the processing step the occupant parameters are reset without capturing facial information based on the recorded image, and In a case where, in the determination step, it is determined that the recorded image corresponds to the third image, in the processing step the facial information is captured based on the recorded image and the occupant parameter is updated. [7] Inmate monitoring program that causes a computer to execute: a procurement step of obtaining a recorded image obtained by mapping an area in which there is a probability that a face of an occupant (X) is present in a vehicle (1); a determination step of determining whether the captured image, which is created in the procurement step, corresponds at least to a first image that does not include a feature section of the occupant's (X) face but includes at least a part of the occupant's (X) body or to a second image that does not include the occupant's (X) body; a processing step of monitoring a change in the state of the occupant's face (X) based on the captured image and switching whether an occupant parameter set to adjust the change in the state of the occupant's face (X) is to be reset or not, according to a determination result in the determination step, wherein In the determination step, it is determined whether the recorded image corresponds to the first image, the second image, or a third image that includes the face of the occupant (X). in a case where, in the determination step, it is determined that the recorded image corresponds to the first image, in the processing step the occupant parameter is maintained without capturing facial information regarding the occupant's face (X) based on the recorded image, in a case where, in the determination step, it is determined that the recorded image corresponds to the second image, in the processing step the occupant parameters are reset without capturing facial information based on the recorded image, and In a case where, in the determination step, it is determined that the recorded image corresponds to the third image, in the processing step the facial information is captured based on the recorded image and the occupant parameter is updated.

Images