Attention direction assessment device and attention direction assessment procedure

Abstract

Attention direction assessment device (200) comprising an image acquisition section (1) for acquiring an image of occupants in a vehicle taken by a camera (9), a facial feature inference section (2) which infers facial features of an inmate based on the image acquired through the image acquisition section (1), a physis inference section (5) which, based on the image acquired by the image acquisition section (1), infers a direction in which an inmate's hand is extended, a face direction detection section (6) which detects a face direction of the occupant based on the facial features inferred by the face feature inference section (2), a seat information acquisition section (21) for acquiring seat information of the occupants in the vehicle, and an attention direction assessment section (7) which, based on the direction in which the occupant's hand is extended as inferred by the physis inference section (5), the face direction of the occupant detected by the face direction detection section (6) and seat information acquired by the seat information acquisition section (21), assesses the direction in which the occupant's attention is directed, where, if the direction inferred by the physique inference section (5) in which the occupant's hand is extended is a neighboring seat direction, the direction of the occupant's face detected by the face direction detection section (6) is the neighboring seat direction, and furthermore, the seat information acquired by the seat information acquisition section (21) shows that there is no occupant in the neighboring seat, the attention direction assessment section (7) judges that the direction in which the occupant's attention is directed is the neighboring seat direction.

Description

[Technical field]

[0001] The present disclosure relates to an attention direction assessment device and an attention direction assessment method for assessing the direction in which the attention of an occupant in a vehicle is directed. [Background technology]

[0002] A conventional technology is disclosed in which, based on an image taken by a camera installed in a vehicle, body features of the occupants in the vehicle are detected, and based on changes in the position of the detected body features, actions of an occupant in the vehicle are predicted (see, for example, patent publication no. JP 2011-123733 A).

[0003] Further state of the art is known from US 2018 / 0 239 975 A1, DE 10 2012 020 713 A1, JP 2018 - 150 694 A, JP 2018 - 92 489 A and US 2014 / 0 334 670 A1. [Overview of the invention][Problem to be solved by the invention]

[0004] However, the problem with the prior art is that while changes in body position can be used to predict an occupant's actions in the vehicle, the direction in which the occupant's attention is directed is not inferred. Consequently, actions are predicted that are unrelated to the direction in which the occupant is directing their attention. For example, the prior art has the problem that even if an occupant has changed their seating position without directing their attention towards an onboard device, the change in body position is used to predict that the occupant's action is to operate an onboard device.

[0005] The present disclosure was made to solve this problem and aims to provide an attention direction assessment device and an attention direction assessment procedure for assessing the direction in which an occupant's attention is directed. [Means of solving the problem]

[0006] The attention direction assessment device according to the present disclosure comprises an image acquisition section for acquiring an image of occupants in a vehicle, captured by a camera; a facial feature inference section, which infers facial features of an occupant based on the image acquired by the image acquisition section; a physical inference section, which infers the direction in which the occupant's hand is extended based on the image acquired by the image acquisition section; a facial direction detection section, which detects the facial direction of the occupant based on the facial features inferred by the facial feature inference section; and an attention direction assessment section, which, based on the direction inferred by the physical inference section in which the occupant's hand is extended,and the direction in which the occupant's attention is directed is assessed based on the occupant's facial direction detected by the facial direction detection section.

[0007] The attention direction assessment method according to the present disclosure comprises a step of acquiring an image of occupants in a vehicle captured by a camera, a step in which facial features of an occupant are inferred based on the image, a step in which the direction in which the occupant's hand is extended is inferred based on the image, a step in which the face direction of the occupant is detected based on the facial features, and a step in which the direction in which the occupant's hand is extended and the face direction of the occupant is assessed based on the direction in which the occupant's attention is directed. [Effects of the invention]

[0008] According to the attention direction assessment device as disclosed herein, the direction in which an occupant's attention is directed can be assessed.

[0009] According to the attention direction assessment procedure as disclosed herein, the direction in which an occupant's attention is directed can be assessed. [Brief explanation of the characters] [ Fig. 1] Fig. Figure 1 is a block diagram showing an example of the construction of an attention direction assessment device according to a first embodiment of the present disclosure. [ Fig. 2] Fig. Figure 2 is a view showing an example of an image of an occupant sitting in the passenger seat according to the first embodiment of the present disclosure. [ Fig. 3] Fig. Figure 3 is a view showing an example of the facial features of the occupant according to the first embodiment of the present disclosure. [ Fig. 4] Fig. Figure 4 is a view showing an example of a plurality of body parts of an occupant in an image according to the first embodiment of the present disclosure. [ Fig. 5] Fig. Figure 5 is a view showing an example of a plurality of body parts of an occupant in an image according to the first embodiment of the present disclosure. [ Fig. 6] Fig. Figure 6 is a view showing an example of a plurality of body parts of an occupant in an image according to the first embodiment of the present disclosure. [ Fig. 7] Fig. Figure 7 is a view showing an example of a plurality of body parts of an occupant in an image according to the first embodiment of the present disclosure. [ Fig. 8] Fig. Figure 8 is a view showing an example in which the direction in which the hand of an occupant is extended in an image according to the first embodiment of the present disclosure is detected. [ Fig. 9] Fig. Figure 9 is a view showing an example of the face direction of an occupant according to the first embodiment of the present disclosure. [ Fig. 10] Fig. Figure 10 is a view showing an example of the face direction of an occupant taking into account the occupant's gaze direction according to the first embodiment of the present disclosure. [ Fig. 11] Fig. Figure 11 is a flowchart showing an example of the operation of an attention direction assessment device according to the first embodiment of the present disclosure. [ Fig. 12] Fig. Figure 12 is a block diagram showing an example of the construction of an attention direction assessment device according to a second embodiment of the present disclosure. [ Fig. 13] Fig. Figure 13 is a flowchart showing an example of the operation of an attention direction assessment device according to the second embodiment of the present disclosure. [ Fig. 14] Fig. Figure 14 is a block diagram showing an example of the hardware setup of an attention direction assessment device according to the first and second embodiments of the present disclosure. [ Fig. 15] Fig. Figure 15 is a block diagram showing an example of the hardware setup of an attention direction assessment device according to the first and second embodiments of the present disclosure. [Forms for carrying out the invention]

[0010] To further explain the present disclosure, the following sections describe various ways of implementing it with reference to the accompanying drawings. The embodiments shown below are examples, and the present disclosure is not limited to these embodiments. First embodiment

[0011] Fig. Figure 1 is a block diagram showing an example of the construction of an attention direction assessment device 100 according to the present embodiment. It is further assumed that the attention direction assessment device 100 is installed in a vehicle.

[0012] As in Fig. As shown in Figure 1, the attention direction assessment device 100 comprises an image acquisition section 1, a facial feature tracking section 2, a head position tracking section 3, a body part tracking section 4, a physique tracking section 5, a facial direction detection section 6, an attention direction assessment section 7, and an assessment result output section 8. Furthermore, the image acquisition section 1 is connected to a camera 9. The camera 9 is installed in a vehicle. The assessment result output section 8 is connected to a control section 10. The control section 10 receives an assessment result output by the assessment result output section 8 and, based on the assessment result, controls onboard devices, etc. Onboard devices include, for example, interior lighting, a navigation system, an audio device, etc.

[0013] Image acquisition section 1 acquires an image of occupants in a vehicle, captured by camera 9. The image is captured by camera 9 in such a way that the occupants sitting in the driver's seat, the front passenger seat, and the rear seat are included. Fig. Figure 2 is a view showing an example image of an occupant sitting in the front passenger seat. In the example in Fig. Figure 2 illustrates the case where an occupant is sitting in the front passenger seat, although there is no restriction to this. An image captured by camera 9 may also include an occupant sitting in the driver's seat or an occupant sitting in the rear seat.

[0014] Facial Feature Inference Section 2 infers facial features of an inmate based on the image acquired through Image Acquisition Section 1. Specifically, Facial Feature Inference Section 2 compares the image acquired through Image Acquisition Section 1 with a prepared learning dictionary (not shown) and infers facial features that indicate the position of the face, eyes, nose, etc., of the inmate depicted in the image acquired through Image Acquisition Section 1. For example, Facial Feature Inference Section 2 infers the position and size of the pupils, the distance between the centers of the respective pupils, the position of the eyebrows, the eyes, the tip of the nose, the mouth, etc., as facial features.of the occupant, although there is no restriction on this and other facial features can also be inferred. The learning dictionary contains the facial features of multiple people. If the image contains multiple occupants, facial feature inference section 2 infers the facial features of the respective occupants.

[0015] Fig. Figure 3 is a view showing an example of an inmate's facial features. The one in Fig. The third depicted occupant is sitting in the passenger seat. Fig. 3. A line encircling an inmate's face in a square represents the position of the inmate's face. A circle represented by a dashed line in the center of the inmate's eyes represents the position and size of the inmate's pupil. Multiple points on the inmate's face represent the position of both ends of the eyebrows, the position of both ends of the eyes, the position of the tip of the nose, and the position of both ends of the mouth. Furthermore, the eye width is the distance between the inmate's two eyes, specifically the distance between the centers of the respective pupils. In facial feature inference section 2, there is no restriction to the features listed in Fig. The 3 facial features shown are not the only ones that can be inferred; other facial features can also be deduced.

[0016] The head position inference section 3 infers the relative position of the occupant's head to camera 9 based on the facial features inferred by the facial feature inference section 2. Specifically, the head position inference section 3 infers the physical relative position of the occupant's head to camera 9 based on the distance between the two eyes or the size of the occupant's pupils, which were inferred by the facial feature inference section 2. That is, the head position inference section 3 infers the distance between camera 9 and the occupant's head.

[0017] In the present embodiment, the relative position of the occupant's head to camera 9 is inferred based on the distance between the two eyes or the size of the occupant's pupils, where the influence of physical differences is minimal. Consequently, the relative position of the occupant's head to camera 9 can be precisely inferred regardless of the occupant's physical position. Furthermore, even if the occupant's seating position changes, the relative position of the occupant's head to camera 9 can be precisely inferred accordingly.

[0018] Body Part Inference Section 4 infers the size of multiple body parts of the occupant in the image, based on the image acquired through Image Acquisition Section 1. Specifically, Body Part Inference Section 4 compares the image acquired through Image Acquisition Section 1 with a prepared learning lexicon (not shown) and infers the size and position of the respective body parts, such as neck, shoulders, arms, hands, torso, etc., of the occupant depicted in the image acquired through Image Acquisition Section 1. The learning lexicon contains the size and position of the respective body parts of multiple people in the image. If the image contains multiple occupants, Body Part Inference Section 4 infers the size of multiple body parts of the respective occupants. The unit of size of the respective body parts of the occupants in the image, which is inferred by body part inference section 4, is pix (pixel).

[0019] Fig. 4 to Fig. 7 are views showing examples of the size and position of a plurality of body parts of the occupants in the image acquired through image acquisition section 1. Fig. 4 to Fig. Six are views showing the size and position of the respective body parts of inmates with different physiques. Fig. 4 to Fig. 7. The lines connecting the respective points correspond to the body parts of the occupant, with the unit of size of the respective body parts being pix (pixels). For example, the one in Fig. The double arrow shown in 7 represents the shoulder width (n-pixels) of the occupant.

[0020] The physique inference section 5 infers the occupant's physique and the direction in which their hand is extended based on the relative position of the occupant's head to camera 9, inferred by the head position inference section 3, and the size of the respective body parts of the occupant in the image, inferred by the body part inference section 4. Specifically, the physique inference section 5 first converts the size of the respective body parts of the occupant in the image, inferred by body part inference section 4, into the actual size of the respective body parts of the occupant. For multiple occupants, the physique inference section 5 infers the actual size of the respective body parts of each individual occupant.

[0021] The procedure by which Physis Inference Section 5 converts the shoulder width of an occupant in the image, inferred by Body Part Inference Section 4, into the actual shoulder width will now be explained. Physis Inference Section 5 calculates an actual shoulder width d using the following equation (1). d=Mn / {2×tan(θ / 2}

[0022] In equation (1), n ​​represents the shoulder width in the image. M indicates the image width and is determined by the image angle θ of camera 9. x represents the distance between camera 9 and the occupant's head and corresponds to the relative position of the occupant's head to camera 9, inferred by the head position inference section 3. In this way, the physique inference section 5 converts the size of the respective body parts in the image, inferred by the body part inference section 4, into the actual size of the respective body parts of the occupant by correcting this using the relative position of the occupant's head, inferred by the head position inference section 3.

[0023] As equation (1) shows, the physique inference section 5 calculates the actual size of the respective body parts of the occupant using the relative position of the occupant's head to camera 9, which is inferred by the head position inference section 3. Consequently, the physique inference section 5 can also correctly calculate the actual size of the respective body parts of the occupant even if the occupant's sitting position changes. The calculation of an occupant's shoulder width has been explained above; however, a similar calculation can be performed for other body parts of an occupant. For example, the physique inference section 5 can also calculate the occupant's height while seated, the size of their face, and the length of their arms.

[0024] Converting the size of an inmate's body parts in the image to their actual size is possible because it can be assumed that all of the inmate's body parts are located at the relative position of the inmate's head to camera 9. While it is conceivable that the shoulders, spine, and face are essentially in the same position relative to the inmate's head to camera 9, this is not a limitation since the arms can move extensively around the shoulders. With regard to the arms, if a change in the chronological sequence of the image has resulted in the arms reaching their maximum size, the body part inference section 4 can infer the size of the inmate's arms at that maximum size.In this case, the physis inference section 5 converts the size of the occupant's arms in the image, inferred by the body part inference section 4, into the actual size of the occupant's arms.

[0025] Next, the Physics Inference Section 5 infers the occupant's physique based on the actual size of the respective body parts. Specifically, Physics Inference Section 5 inputs the actual size of the respective body parts of the occupant into a (not shown) learning device that has completed a learning process involving mechanical learning to infer the occupant's physique. By performing computational processing on the learning device, the section then obtains an inferred result regarding the occupant's physique. For multiple occupants, Physics Inference Section 5 infers the physique of each individual occupant.

[0026] The learning device contains a learning dictionary (not shown) in which the actual size of various body parts of a majority of people is recorded and correlated with the physique of those individuals. For example, if the physique inference section 5 inputs seat height, shoulder width, face size, and arm length into the learning device, the device compares the actual size of the entered body parts with the learning dictionary and infers the physique of the occupant with the highest probability based on the entered body parts. In this way, a correlation exists between the actual size of the respective body parts of the occupant and the occupant's physique. The physique inference section 5 acquires the inference result regarding the occupant's physique from the learning device.

[0027] Next, the physis inference section estimates the direction in which the occupant's hand is extended based on the inferred physis of the occupant. For example, by detecting the occupant's hand within a predefined physis detection area, the physis inference section 5 determines the direction in which the occupant's hand is extended. Fig. Figure 8 is a view showing an example of detecting the direction in which an inmate's hand is extended in the image acquired through image acquisition section 1.

[0028] At the in Fig. In example 8, the occupant is sitting in the passenger seat. Fig. The areas enclosed by a dashed line in section 8 represent the respective pre-set physical detection zones. In the example in Fig. 8. The areas designated as physical detection zones are the adjacent seat area in relation to the occupant being detected, the rear seat area in relation to the occupant being detected, and a door handle area for the occupant being detected entering and exiting the vehicle. In the case of the Fig. The example shown in Figure 8 illustrates the case where the occupant is seated in the front passenger seat, with the settings adjusted appropriately to the occupant's seating position. Furthermore, the size of each detection area is freely adjustable and will be set appropriately for the occupant being detected.

[0029] The Physis Inference Section 5 concludes that the occupant's hand is extended in the direction of the adjacent seat if the occupant's hand is in the Fig. The example shown in section 8 was demonstrated in a physis detection area A. The physis inference section 5 concludes that the occupant's hand is extended in the rearward direction if the occupant's hand is in the Fig. The example shown in section 8 was demonstrated in a physical detection area B. Physical deduction section 5 concludes that the occupant's hand is extended in the direction of the door handle when the occupant's hand is in the Fig. 8. If the occupant's hand was detected in a physis detection area C, as shown in the example. If the occupant's hand was detected in an area other than the preset physis detection areas, or if the occupant's hand was detected in a plurality of physis detection areas, the physis inference section 5 assesses that the direction of the occupant's hand cannot be inferred. Fig. Figure 8 is an example where the occupant to be verified is sitting in the front passenger seat, whereby the occupant to be verified is not limited to an occupant sitting in the front passenger seat.

[0030] The face direction detection section 6 detects the face direction based on the positional relationship between the face, eyes, nose, etc., as facial features of the occupant, which were inferred by the face feature inference section 2. For example, face direction detection section 6 extracts the features corresponding to the facial parts from the occupant's portrait and calculates the occupant's face direction angle based on the positional relationship of the facial parts represented by these features. The face direction angle can be detected, for example, using the POSIT (Pose from Orthography and Scaling with Iterations) algorithm, which is a well-established technology.

[0031] The face direction detection section 6 detects the occupant's face direction based on the detected face direction angle. For example, the face direction detection section 6 detects whether the occupant is facing the direction of the adjacent seat, whether the occupant is facing the direction of the rear seat, or whether the occupant is facing the direction of the door handle in order to enter or exit the vehicle. Fig. Figure 9 is a view showing examples of the occupant's face direction, depicting a state seen from above. Fig. Figure 9 shows a dashed line indicating the direction of the occupant's face, where a face direction angle of 0 degrees is given when the occupant's face direction coincides with a direction of travel Y of the vehicle.

[0032] Next, we will use the following methods: Fig. Nine detection examples of the occupant's face direction using face direction detection section 6 are explained. In the following examples... Fig. In 9 of the examples shown, the occupant is sitting in the passenger seat.

[0033] As in Fig. As shown in Figure 9 (a), the face direction detection section 6 assesses, in the case where the occupant's face direction is turned to the right from the direction of travel Y of the vehicle, that the occupant is facing in the direction of the adjacent seat if the face direction angle starting from 0 degrees is less than or equal to a certain angle α.

[0034] As in Fig. As shown in Figure 9 (b), the face direction detection section 6 assesses, in the case where the occupant's face direction is turned to the left from the direction of travel Y of the vehicle, that the occupant is facing in the direction of the door handle if the face direction angle starting from 0 degrees is less than or equal to a certain angle β.

[0035] As in Fig. As shown in Figure 9 (c), the face direction detection section 6 assesses, in the case that the face direction of the occupant exceeds the specified angle α from the direction of travel Y of the vehicle to the right, that the occupant is facing in the rear seat direction.

[0036] As in Fig. As shown in Figure 9 (d), the face direction detection section 6 assesses, in the case that the face direction of the occupant from the direction of travel Y of the vehicle to the left exceeds the specified angle β, that the occupant is facing in the rear seat direction.

[0037] In this process, specific angles α and β are set as angles that can be measured in the direction of the rear seat. For example, specific angles α and β are set to 50 degrees. Specific angles α and β are arbitrarily adjustable, and can be set to different values. Fig. Figure 9 shows examples where the occupant to be identified is sitting in the front passenger seat, although the occupant to be identified is not limited to an occupant sitting in the front passenger seat. Furthermore, the occupant's facial direction, which is assessed using facial direction detection section 6, is determined according to the seating position of the occupant to be identified.

[0038] The face direction detection section 6 can also be configured to detect the occupant's field of vision or gaze direction based on the features inferred by the face feature inference section 2. That is, the face direction detection section 6 can also be configured to calculate the occupant's face direction angle, field of vision, or gaze direction based on the occupant's features inferred by the face feature inference section 2, and then detect the occupant's face direction based on the calculated face direction angle, field of vision, or gaze direction.For example, the face direction detection section 6 detects the visual field or gaze direction based on the position and size of the pupils inferred by the face feature inference section 2, the distance between the centers of the respective pupils of the two eyes, the position of the two ends of the eyebrows, and the position of the two ends of the occupant's eyes. The visual field or gaze direction can be detected using a generally known gaze direction detection processing method.

[0039] The face direction detection section 6 detects a field of view direction angle as the field of vision or gaze direction. The field of view direction angle is an angle representing the direction of the occupant's field of vision relative to the occupant's face direction angle. In a setup where the face direction detection section 6 detects the field of view direction angle, the detection of the face direction is achieved by the face direction detection section 6 calculating the face direction angle, taking the occupant's field of view direction angle into account.For example, if the direction of rotation of the occupant's face direction angle to the vehicle's direction of travel Y coincides with the direction of rotation of the occupant's field of vision angle to the occupant's face direction angle, the face direction section 6 calculates a value that results from adding the occupant's field of vision angle to the occupant's face direction angle, as the face direction angle taking into account the occupant's field of vision angle.On the other hand, if the direction of rotation of the occupant's face direction angle relative to the vehicle's direction of travel Y differs from the direction of rotation of the occupant's field of vision angle relative to the occupant's face direction angle, the face direction detection section 6 calculates a value that results from subtracting the occupant's field of vision angle from the occupant's face direction angle, as the face direction angle taking the occupant's field of vision angle into account. That is, face direction detection section 6 detects the face direction based on the face direction angle that considers the occupant's field of vision angle, which, in addition to the occupant's face angle, was also calculated using the occupant's field of vision angle.

[0040] Fig. Figure 10 is a view showing an example of the occupant's face direction, taking into account the occupant's field of vision angle, and depicts a state seen from above. Fig. In Figure 10, the occupant's facing direction is shown by a dash-dotted line, and the occupant's field of vision angle is shown by a dashed line. If the occupant's facing direction coincides with the vehicle's direction of travel Y, the field of vision angle is 0 degrees. Furthermore, if the occupant's facing direction coincides with the occupant's field of vision angle, the field of vision angle is also 0 degrees.

[0041] Next, we will use the following as an example: Fig. 10. A detection example of the occupant's face direction, taking into account the occupant's field of vision angle, is explained using face direction detection section 6. In the case described in Fig. In example 10, the occupant is sitting in the passenger seat. In the example shown... Fig. In the example shown in Figure 10, the occupant's viewing angle is A degrees to the right of the vehicle's direction of travel Y. Furthermore, the occupant's field of vision angle is a degrees to the left of the vehicle's direction of travel Y. That is, in the Fig. In the example shown, the face direction detection section 6 calculates the face direction angle of the occupant as face direction angle A degrees - a degrees, taking into account the occupant's field of vision direction angle, and assesses that if the face direction angle A degrees - a degrees, which takes into account the occupant's field of vision direction angle, is less than or equal to the determined angle α, the occupant is facing the direction of the adjacent seat.

[0042] This means that the face direction detection section 6 determines that, if the face direction angle, taking into account the occupant's field of vision angle of 0 degrees, is less than or equal to the specified angle α, the occupant is facing towards the adjacent seat direction, and if the face direction angle, taking into account the occupant's field of vision angle, exceeds the specified angle α, the occupant is facing towards the rear seat direction. Furthermore, if the face direction angle, taking into account the occupant's field of vision angle of 0 degrees, is less than or equal to the specified angle β, the face direction detection section 6 determines that the occupant is facing towards the door handle direction, and if the face direction angle, taking into account the occupant's field of vision angle, exceeds the specified angle β, the occupant is facing towards the rear seat direction.

[0043] The attention direction assessment section 7 assesses the direction in which the occupant's attention is directed, based on the direction inferred by the physical inference section 5 regarding the occupant's outstretched hand, and the occupant's facial direction inferred by the face direction detection section 6. Furthermore, the attention direction assessment section 7 outputs the assessed result to the assessment result output section 8.For example, if the direction inferred by the physique inference section 5 in which the occupant's hand is extended matches the face direction inferred by the face direction detection section 6, the attention direction assessment section 7 judges that the direction in which the occupant's attention is directed is the direction inferred by the physique inference section 5 and the face direction detection section 6.On the other hand, the attention direction assessment section 7 assesses that the direction of attention of the occupant cannot be assessed if the direction inferred by the physique inference section 5, in which the occupant's hand is extended, does not correspond to the occupant's face direction inferred by the face direction detection section 6, if the direction in which the occupant's hand is extended, inferred by the physique inference section 5, cannot be inferred, and if the occupant's face direction, inferred by the face direction detection section 6, cannot be inferred.

[0044] The assessment result output section 8 outputs the assessment result assessed by the attention direction assessment section 7 to the control section 10. The assessment result assessed by the attention direction assessment section 7 also includes the assessment result of an unassessable attention direction assessed by the attention direction assessment section 7.

[0045] Control section 10 receives the assessment result output by assessment result output section 8 and executes corresponding controls on onboard devices, etc. For example, if control section 10 receives the assessment result that the direction in which the occupant's attention is directed is towards the adjacent seat, it executes a control to activate lighting that illuminates the adjacent seat relative to the occupant being assessed. Furthermore, if control section 10 receives the assessment result that the direction in which the occupant's attention is directed is towards the rear seat, it executes a control to activate lighting that illuminates the rear seat relative to the occupant being assessed.Furthermore, if control section 10 receives the assessment result that the direction in which the occupant's attention is directed is towards the door handle, it will activate lighting that illuminates the footwell of the occupant being assessed. Control section 10 will not activate any onboard devices, etc., if the received result indicates an unassessable direction of attention.

[0046] Fig. 11 is a flowchart that illustrates the working methods of the in Fig. 1 shows attention direction assessment device 100.

[0047] In step S101, image acquisition section 1 acquires an image of the occupants of a vehicle, captured by camera 9. In step S102, facial feature inference section 2 infers facial features of an occupant based on the image acquired by image acquisition section 1.

[0048] In step S103, the head position inference section 3 infers the relative position of the occupant's head to camera 9 based on the facial features inferred by the facial feature inference section 2. In S104, the body part inference section 4 infers the size of a plurality of the occupant's body parts in the image based on the image acquired by the image acquisition section 1.

[0049] In step S105, the physis inference section 5 infers the occupant's physis based on the relative position of the occupant's head inferred by the head position inference section 3 and the size of the respective body parts inferred by the body part inference section 4. In step S106, the physis inference section 5 assesses whether the inferred physis is detectable in the physis detection areas and infers the direction in which the occupant's hand is extended.

[0050] In step S107, the face direction detection section 6 detects the face direction of the occupant based on the positional relationship between the face, eyes, nose, etc. as facial features of the occupant, which were inferred by the face feature inference section 2.

[0051] In step S108, the attention direction assessment section 7 assesses the direction in which the occupant's hand is extended, based on the direction inferred by the physique inference section 5 and the face direction of the occupant inferred by the face direction detection section 6.

[0052] In step S109, the assessment result output section 8 outputs the assessment result assessed by the attention direction assessment section 7. In the example in Fig. 1. The assessment result output section 8 outputs the direction in which the occupant's attention is directed, as assessed by the attention direction assessment section 7, to the control section 10.

[0053] The attention direction assessment device according to the present first embodiment comprises an image acquisition section for acquiring an image of occupants in a vehicle, captured by a camera; a facial feature inference section, which infers facial features of an occupant based on the image acquired by the image acquisition section; a physical inference section, which infers the direction in which the occupant's hand is extended based on the image acquired by the image acquisition section; a facial direction detection section, which detects the facial direction of the occupant based on the facial features inferred by the facial feature inference section; and an attention direction assessment section, which, based on the direction inferred by the physical inference section regarding the direction in which the occupant's hand is extended,and the direction in which the occupant's attention is directed is assessed based on the occupant's facial direction detected by the facial direction detection section.

[0054] Furthermore, in the present first embodiment, the attention direction assessment section of the attention direction assessment device assesses, in the event that the direction inferred by the physis inference section in which the occupant's hand is extended coincides with the face direction of the occupant detected by the face direction detection section, the direction in which the occupant's attention is directed.

[0055] Furthermore, the attention direction assessment section of the attention direction assessment device according to the present first embodiment assesses, in the case that the direction inferred by the physis inference section in which the occupant's hand is extended is the neighboring seat direction, and the face direction of the occupant detected by the face direction detection section is the neighboring seat direction, that the direction in which the occupant's attention is directed is the neighboring seat direction.

[0056] Furthermore, according to the present first embodiment, the attention direction assessment section of the attention direction assessment device assesses, in the case that the direction inferred by the physique inference section in which the occupant's hand is extended is the rear seat direction, and the face direction of the occupant detected by the face direction detection section is the rear seat direction, that the direction in which the occupant's attention is directed is the rear seat direction.

[0057] Furthermore, the attention direction assessment section of the attention direction assessment device according to the present first embodiment assesses, in the case that the direction inferred by the physique inference section in which the occupant's hand is extended is the door handle direction, and the face direction of the occupant detected by the face direction detection section is the door handle direction, that the direction in which the occupant's attention is directed is the door handle direction.

[0058] Furthermore, the attention direction assessment section of the attention direction assessment device according to the present first embodiment comprises a head position inference section which infers the relative position of the occupant's head to the camera based on facial features inferred by the facial feature inference section, and a body part inference section which infers the size of a plurality of body parts of the occupant in the image based on the image acquired by the image acquisition section, wherein the physique inference section infers the occupant's physique based on the relative position of the head inferred by the head position inference section and the size of the respective body parts inferred by the body part inference section, and infers the direction in which the occupant's hand is extended based on the inferred physique inference result of the occupant.

[0059] Furthermore, the attention direction assessment device according to the present first embodiment is characterized in that it also includes an assessment result output section for outputting the direction in which the occupant's attention is directed, as assessed by the attention direction assessment section.

[0060] According to the above setup, the attention direction assessment device 100 of the first embodiment can assess the direction in which the occupant's attention is directed. Furthermore, the face direction detection section 6 detects the face direction based on the positional relationship between the face, eyes, nose, etc., as facial features of the occupant, which were inferred by the face feature inference section 2, so that the attention direction assessment device 100 can also assess the direction in which the occupant's attention is directed even if the occupant is wearing sunglasses.

[0061] The attention direction assessment method according to the first embodiment comprises a step to acquire an image of occupants in a vehicle, captured by a camera; a step in which facial features of an occupant are inferred from the image; a step in which the direction in which the occupant's hand is extended is inferred from the image; a step in which the occupant's face direction is detected based on the facial features; and a step in which the direction in which the occupant's hand is extended and the occupant's face direction is assessed based on the direction in which the occupant's attention is directed.

[0062] According to the above setup, the direction in which the occupant's attention is directed can be assessed using the attention direction assessment method according to the first embodiment. Second embodiment

[0063] The construction of an attention direction assessment device 200 according to a second embodiment of the present disclosure is explained. Structures identical to or corresponding to those of the first embodiment are omitted, and only differing parts of the construction are explained.

[0064] Fig. Figure 12 is a block diagram showing an example of the construction of the attention direction assessment device 200 according to the second embodiment. As in Fig. As shown in Figure 12, the attention assessment device 200 comprises a seat information acquisition section 21 and a vehicle information acquisition section 22.

[0065] The seat information acquisition section 21 acquires occupant seating information from the vehicle based on an image captured by camera 9. This seating information indicates which seat an occupant is occupying. The acquisition of occupant seating information by seat information acquisition section 21 is not limited to configurations where the acquisition is based on an image of the occupants captured by camera 9. For example, if the vehicle is equipped with a seat occupancy sensor, the configuration can also be such that seat information acquisition section 21 acquires occupant seating information from the seat occupancy sensor.

[0066] Vehicle Information Acquisition Section 22 acquires vehicle information from the vehicle. This vehicle information includes, for example, information regarding when the vehicle stopped. Vehicle Information Acquisition Section 22 acquires vehicle information, for example, from a vehicle sensor (not shown) such as a speed sensor, etc.

[0067] The attention direction assessment section 7 according to the second embodiment of the present disclosure assesses, based on the direction in which the occupant's hand is extended as inferred by the physique inference section 5, the face direction of the occupant as inferred by the face direction detection section 6, and at least either the seat information acquired by the seat information acquisition section 21 or the vehicle information acquired by the vehicle information acquisition section 22, the direction in which the occupant's attention is directed.

[0068] Next, an example of an assessment is explained for the case in which the attention direction assessment section 7 assesses the direction in which the occupant's attention is directed based on the direction inferred by the physique inference section 5 in which the occupant's hand is extended, the direction of the occupant's face inferred by the face direction detection section 6, and the seat information acquired by the seat information acquisition section 21.

[0069] Attention Direction Assessment Section 7 acquires, in the event that the direction inferred by Physical Inference Section 5, in which the occupant's hand is extended, is the direction of the adjacent seat, and the occupant's face direction inferred by Face Direction Detection Section 6 is also the direction of the adjacent seat, additionally acquires adjacent seat information based on the seat information acquired by Seat Information Acquisition Section 21 and assesses the direction in which the occupant's attention is directed. The adjacent seat information is information indicating whether an occupant is sitting in the seat adjacent to the occupant being identified.This means that the attention direction assessment section 7 can also be structured in such a way that, in the event that the direction inferred by the physique inference section 5 in which the occupant's hand is extended is the adjacent seat direction, and the face direction of the occupant inferred by the face direction detection section 6 is the adjacent seat direction, and furthermore the seat information acquired by the seat information acquisition section 21 shows that no occupant is sitting in the adjacent seat with respect to the occupant to be identified, it assesses that the direction in which the occupant's attention is directed is the adjacent seat direction.On the other hand, in the case where the seat information acquired through the seat information acquisition section 21 shows that an occupant is sitting in the adjacent seat in relation to the occupant to be identified, the attention direction assessment section 7 determines that the attention direction cannot be assessed.

[0070] Attention Direction Assessment Section 7 acquires rear seat information based on seat information acquired by Seat Information Acquisition Section 21 if the direction inferred by Physical Inference Section 5 regarding the occupant's outstretched hand is the rear seat direction, and the occupant's face direction inferred by Face Direction Detection Section 6 is also the rear seat direction. This section then assesses the direction in which the occupant's attention is directed. The rear seat information indicates whether an occupant is seated in the rear seat relative to the occupant being identified.This means that the attention direction assessment section 7 can also be structured in such a way that, in the event that the direction inferred by the physique inference section 5 in which the occupant's hand is extended is the rear seat direction, and the face direction of the occupant inferred by the face direction detection section 6 is the rear seat direction, and furthermore the seat information acquired by the seat information acquisition section 21 shows that no occupant is sitting in the rear seat in relation to the occupant to be identified, it assesses that the direction in which the occupant's attention is directed is the rear seat direction.On the other hand, in the case where the seat information acquired through the seat information acquisition section 21 shows that an occupant is sitting in the back seat in relation to the occupant to be verified, the attention direction assessment section 7 determines that the attention direction cannot be assessed.

[0071] Next, an assessment example is explained for the case in which the attention direction assessment section 7 assesses the direction in which the occupant's attention is directed based on the direction inferred by the physique inference section 5 in which the occupant's hand is extended, the direction of the occupant's face inferred by the face direction detection section 6, and the vehicle information acquired by the vehicle information acquisition section 22.

[0072] The attention direction assessment section 7 assesses, in the event that the direction inferred by the physical inference section 5, in which the occupant's hand is extended, is the direction towards the door handle, and the occupant's face direction inferred by the face direction detection section 6 is also in the direction towards the door handle, the direction in which the occupant's attention is directed, using the vehicle information acquired by the vehicle information acquisition section 22. This vehicle information pertains to information regarding the vehicle stopping.This means that the attention direction assessment section 7 can also be structured such that, if the direction inferred by the physical inference section 5, in which the occupant's hand is extended, is the direction of the door handle, and the occupant's face direction, inferred by the face direction detection section 6, is also the direction of the door handle, and furthermore, the vehicle information acquired by the vehicle information acquisition section 22 shows that the vehicle has stopped, then it assesses that the direction in which the occupant's attention is directed is the direction of the door handle. Conversely, if the vehicle information acquired by the vehicle information acquisition section 22 shows that the vehicle has not stopped, then the attention direction assessment section 7 assesses that the direction of attention cannot be assessed.

[0073] Furthermore, Attention Direction Assessment Section 7 can also be structured such that, if the occupant being assessed is a driver's seat, it acquires information regarding whether the vehicle stopped based on the vehicle information acquired by Vehicle Information Acquisition Section 22, and, if the vehicle stopped, assesses the direction in which the occupant's attention was directed. That is to say, Attention Direction Assessment Section 7 can also be structured such that, if the occupant being assessed is a driver's seat and, moreover, did not stop the vehicle, it assesses that the direction of attention cannot be assessed.

[0074] Furthermore, a setup is also possible in which the face direction detection section 6 changes the value of a certain angle used for detecting the face direction of the occupant, or the physis inference section 5 changes the size of the respective physis detection areas used for inferring the direction in which the occupant's hand is extended, according to the seat information acquired by the seat information acquisition section 21 or the vehicle information acquired by the vehicle information acquisition section 22.For example, the physical detection section 5 can be structured such that, in the event that the vehicle information acquired through the vehicle information acquisition section 22 indicates a parking gear position of the vehicle or a switched-off ignition, the respective physical detection areas are formed larger than the respective physical detection areas used when driving the vehicle.

[0075] By acquiring seat information and considering information relevant to multiple occupants, the attention direction assessment section 7 can reduce misjudgments of attention direction. Furthermore, by acquiring vehicle information and considering the vehicle's condition, the attention direction assessment section 7 can reduce misjudgments of attention direction.

[0076] If the setup involves the attention direction assessment section 7 not using seat and vehicle information when assessing the direction in which the occupant's attention is directed, a setup is also possible in which the control section 10 performs control of onboard devices, etc., based on the assessment result output by the assessment result output section 8 and at least one piece of information from either the seat information acquired by the seat information acquisition section 21 or the vehicle information acquired by the vehicle information acquisition section 22.

[0077] Next, an embodiment will be explained in the case where, according to the second embodiment of the present disclosure, the control section 10 performs control of onboard devices, etc., based on the assessment result output by the assessment result output section 8 and the seat information acquired by the seat information acquisition section 21.

[0078] If Control Section 10 receives the assessment result that the direction in which the occupant's attention is directed is towards the adjacent seat, it acquires the adjacent seat information based on the seat information acquired by Seat Information Acquisition Section 21. If the seat information acquired by Control Section 10 indicates that there is no occupant in the seat adjacent to the occupant being investigated, Control Section 10 initiates a control action to activate lighting that illuminates the adjacent seat in relation to the occupant being investigated.On the other hand, if the seat information acquired by Control Section 10 indicates that an occupant is seated in the adjacent seat in relation to the occupant to be verified, Control Section 10 will not initiate any control to switch on lighting that illuminates the adjacent seat in relation to the occupant to be verified.

[0079] If Control Section 10 receives the assessment result that the direction in which the occupant's attention is directed is towards the rear seat, it acquires rear seat information based on the seat information acquired by Seat Information Acquisition Section 21. If the seat information acquired by Control Section 10 indicates that there is no occupant in the rear seat relative to the occupant being investigated, Control Section 10 initiates a control action to activate lighting that illuminates the rear seat relative to the occupant being investigated.On the other hand, if the seat information acquired by Control Section 10 indicates that there is an occupant in the rear seat in relation to the occupant to be verified, Control Section 10 will not initiate any control to switch on lighting that illuminates the rear seat in relation to the occupant to be verified.

[0080] Next, an embodiment is described in the case where, according to the second embodiment of the present disclosure, the control section 10 performs control of onboard devices, etc., based on the assessment result output by the assessment result output section 8 and the vehicle information acquired by the vehicle information acquisition section 22.

[0081] If Control Section 10 receives the assessment result that the direction in which the occupant's attention is directed is towards the door handle, it acquires vehicle information obtained through Vehicle Information Acquisition Section 22. If the vehicle information acquired by Control Section 10 indicates that the vehicle has stopped, Control Section 10 initiates a control action to activate lighting in the footwell in relation to the occupant being investigated. Conversely, if the vehicle information acquired by Control Section 10 indicates that the vehicle has not stopped, Control Section 10 does not initiate a control action to activate lighting in the footwell in relation to the occupant being investigated.

[0082] Furthermore, a configuration is also possible in which, if the configuration involves the attention direction assessment section 7 not using seat and vehicle information when assessing the direction in which the occupant's attention is directed, and if the occupant for whom the direction in which attention was directed was assessed by the attention direction assessment section 7 is an occupant sitting in the driver's seat, the control section 10 performs control of onboard devices, etc., according to the assessment result, if the vehicle information acquired by the vehicle information acquisition section 22 shows that the vehicle has stopped.This means that it can also be a setup in which the control section 10, in the event that the occupant for whom the direction of attention was assessed by the attention direction assessment section 7 is an occupant sitting in the driver's seat, and furthermore the vehicle information acquired by the vehicle information acquisition section 22 shows that the vehicle has not stopped, is not controlling any onboard devices, etc.

[0083] By using seat information when controlling onboard devices, control section 10 allows onboard devices to be controlled based on information regarding multiple occupants, thus reducing the risk of incorrect device operation. Furthermore, by using vehicle information when controlling onboard devices, control section 10 allows onboard devices to be controlled taking the vehicle's state into account, thus reducing the risk of incorrect device operation.

[0084] Fig. 13 is a flowchart that illustrates the working methods of the in Fig. The attention direction assessment device 200 shown in step 12 is shown. Step S201 to step S207 in Fig. 13 correspond to steps S101 to S107 in Fig. 11, and step S211 in Fig. 13 corresponds to step S109 in Fig. 11, so an explanation is omitted here. Steps S208 to S210 are explained below.

[0085] In step S208, the seat information acquisition section 21 acquires seat information of the occupants in a vehicle. In step S209, the vehicle information acquisition section 22 acquires vehicle information.

[0086] In step S210, the attention direction assessment section 7 uses the direction in which the occupant's hand is extended, inferred by the physique inference section 5, the occupant's face direction detected by the face direction detection section 6, and at least either the seat information acquired by the seat information acquisition section 21 or the vehicle information acquired by the vehicle information acquisition section 22, to assess the direction in which the occupant's attention is directed.

[0087] The attention direction assessment device according to the present second embodiment has a seat information acquisition section for acquiring seat information of the occupants in a vehicle, and the attention direction assessment section assesses the direction in which the occupant's attention is directed based on the direction inferred by the physique inference section in which the occupant's hand is extended, the face direction of the occupant detected by the face direction detection section, and the seat information acquired by the seat information acquisition section.

[0088] Furthermore, according to the present second embodiment, the attention direction assessment device has a vehicle information acquisition section for acquiring vehicle information, and the attention direction assessment section assesses the direction in which the occupant's hand is extended, based on the direction inferred by the physical inference section, the occupant's face direction detected by the face direction detection section, and the vehicle information acquired by the vehicle information acquisition section.

[0089] Furthermore, according to the second embodiment, the attention direction assessment section of the attention direction assessment device assesses, in the case that the direction inferred by the physique inference section in which the occupant's hand is extended is the adjacent seat direction, and the face direction of the occupant detected by the face direction detection section is the adjacent seat direction, and furthermore, the seat information acquired by the seat information acquisition section shows that no occupant is sitting in the adjacent seat, that the direction in which the occupant's attention is directed is the adjacent seat direction.

[0090] Furthermore, according to the second embodiment, the attention direction assessment section of the attention direction assessment device assesses, in the case that the direction inferred by the physique inference section in which the occupant's hand is extended is the rear seat direction, and the face direction of the occupant detected by the face direction detection section is the rear seat direction, and furthermore, the seat information acquired by the seat information acquisition section shows that no occupant is sitting in the rear seat, that the direction in which the occupant's attention is directed is the rear seat direction.

[0091] Furthermore, according to the second embodiment, the attention direction assessment section of the attention direction assessment device assesses, in the event that the direction inferred by the physical inference section in which the occupant's hand is extended is the door handle direction, and the face direction of the occupant detected by the face direction detection section is the door handle direction, and furthermore, the vehicle information acquired by the vehicle information acquisition section shows that the vehicle has stopped, that the direction in which the occupant's attention is directed is the door handle direction.

[0092] The aforementioned design allows the attention direction assessment device 200, according to the second embodiment, to assess the direction in which the occupant's attention is directed. Furthermore, by acquiring seat information and assessing the direction in which the occupant's attention is directed based on information concerning multiple occupants, the attention direction assessment device 200, according to the second embodiment, can reduce misjudgments of the attention direction. Additionally, by acquiring vehicle information, the attention direction assessment device 200, according to the second embodiment, can assess the direction in which the occupant's attention is directed based on the vehicle's condition, thus reducing misjudgments of the attention direction.

[0093] The respective functions of the image acquisition section 1, the facial feature tracking section 2, the head position tracking section 3, the body part tracking section 4, the physical tracking section 5, the face direction detection section 6, the attention direction assessment section 7, the assessment result output section 8, the seat information acquisition section 21 and the vehicle information acquisition section 22 of the attention direction assessment device 100, 200 described above are implemented by means of a processing circuit.That is, the attention direction assessment device 100, 200 comprises a processing circuit for acquiring an image of the occupants in a vehicle, which was recorded by a camera, for inferring the facial features of the occupants, for inferring the relative position of the head of the occupants to the camera, for inferring the size of a plurality of body parts of the occupants in the image, for inferring the physique of the occupants and the direction in which a hand is extended, for detecting the face direction of the occupants, for assessing the direction in which the attention of the occupants is directed, and for outputting the assessment results.The processing circuit can be special hardware, but it can also be formed by means of a processor (also called CPU (Central Processing Unit), processing device, computing device, microprocessor, microcomputer, DSP (Digital Signal Processor)) which executes programs stored in a memory.

[0094] If the processing circuit is a special piece of hardware, a processing circuit 31 corresponds to the one in Fig. Figure 14 shows, for example, a simple circuit, a complex circuit, a programmed processor, parallel programmed processors, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof. The respective functions of the image acquisition section 1, the facial feature tracking section 2, the head position tracking section 3, the body part tracking section 4, the physique tracking section 5, the face direction detection section 6, the attention direction assessment section 7, the assessment result output section 8, the seat information acquisition section 21, and the vehicle information acquisition section 22 can each be implemented by a processing circuit 31, but the respective functions can also be implemented in their entirety by a single processing circuit 31.

[0095] Is the processing circuit 31 a processor 32, as in Fig. As shown in Figure 15, the respective functions of the image acquisition section 1, the facial feature tracking section 2, the head position tracking section 3, the body part tracking section 4, the physique tracking section 5, the face direction detection section 6, the attention direction assessment section 7, the assessment result output section 8, the seat information acquisition section 21, and the vehicle information acquisition section 22 are implemented by software, firmware, or a combination of software and firmware. The software or firmware is represented as a program and stored in a memory 33. The processor 32 reads programs stored in the memory 33 and executes them, thereby implementing the respective functions. That is, the attention direction assessment device 100, 200 includes a memory 33 for storing programs that are ultimately executed in the respective Fig. 11 steps shown from step S101 to step S109 or in the respective in Fig.The 13 steps shown, from step S201 to S211, are executed. It can further be said that these programs are the sequence or procedure for executing image acquisition section 1, facial feature following section 2, head position following section 3, body part following section 4, physis following section 5, face direction detection section 6, attention direction assessment section 7, assessment result output section 8, seat information acquisition section 21, and vehicle information acquisition section 22 on a computer. The memory can be a non-volatile or volatile semiconductor memory such as... B. a RAM (Random Access Memory), a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable Read Only Memory), an EEPROM (Electrically Erasable Programmable Read Only Memory), etc., a magnetic disk, a floppy disk, an optical disk, a compact disc, a DVD (Digital Versatile Disc), etc., and also various storage media used in the future.

[0096] Regarding the respective functions of the image acquisition section 1, the facial feature tracking section 2, the head position tracking section 3, the body part tracking section 4, the physical tracking section 5, the face direction detection section 6, the attention direction assessment section 7, the assessment result output section 8, the seat information acquisition section 21 and the vehicle information acquisition section 22, it is also possible that some of the functions are implemented using special hardware, and other functions are implemented using software or firmware.

[0097] This allows the processing circuit to implement the respective functions described above through hardware, software, firmware, or a combination thereof.

[0098] The attention direction assessment device described above can be applied not only to an onboard navigation device, i.e., a car navigation system, but also to a navigation system that combines a portable navigation device (PND) that can be installed in a vehicle with a server located outside the vehicle, etc., in a suitable manner, or to a device other than a navigation system. This allows the same effects to be achieved even in a configuration where the respective functions of the attention direction assessment device are distributed among the various functions comprising the system.

[0099] In the present disclosure, any combination of the respective embodiments is possible within the scope of the disclosure, and the respective embodiments can also be modified or omitted in a suitable manner.

[0100] The present revelation has been explained in detail, the preceding explanation being in all aspects an exemplary representation, and the present revelation is not limited to it. Countless modified examples, not presented as examples, are also conceivable without deviation from the scope of the present revelation. [Explanation of reference symbols] 100, 200 Attention Direction Assessment Device 1. Image Acquisition Section 2 Facial Feature Follow-Up Section 3 Head position follow-up section 4 Body Part Follow-Up Section 5. Physical Follow-Up Section 6 Face direction detection section 7 Attention Direction Assessment Section 8 Assessment Result Output Section 9 Camera 10 Tax Section 21 Seat Information - Purchase Section 22 Vehicle Information - Acquisition Section 31 Processing circuit 32 processor 33 storage

Claims

[1] Attention direction assessment device (200) comprising an image acquisition section (1) for acquiring an image of occupants in a vehicle taken by a camera (9), a facial feature inference section (2) which infers facial features of an inmate based on the image acquired through the image acquisition section (1), a physis inference section (5) which, based on the image acquired by the image acquisition section (1), infers a direction in which an inmate's hand is extended, a face direction detection section (6) which detects a face direction of the occupant based on the facial features inferred by the face feature inference section (2), a seat information acquisition section (21) for acquiring seat information of the occupants in the vehicle, and an attention direction assessment section (7) which, based on the direction in which the occupant's hand is extended as inferred by the physis inference section (5), the face direction of the occupant detected by the face direction detection section (6) and seat information acquired by the seat information acquisition section (21), assesses the direction in which the occupant's attention is directed, where, if the direction inferred by the physique inference section (5) in which the occupant's hand is extended is a neighboring seat direction, the direction of the occupant's face detected by the face direction detection section (6) is the neighboring seat direction, and furthermore, the seat information acquired by the seat information acquisition section (21) shows that there is no occupant in the neighboring seat, the attention direction assessment section (7) judges that the direction in which the occupant's attention is directed is the neighboring seat direction. [2] Attention direction assessment device (200) comprising an image acquisition section (1) for acquiring an image of occupants in a vehicle taken by a camera (9), a facial feature inference section (2) which infers facial features of an inmate based on the image acquired through the image acquisition section (1), a physis inference section (5) which, based on the image acquired by the image acquisition section (1), infers a direction in which an inmate's hand is extended, a face direction detection section (6) which detects a face direction of the occupant based on the facial features inferred by the face feature inference section (2), a seat information acquisition section (21) for acquiring seat information of the occupants in the vehicle, and an attention direction assessment section (7) which, based on the direction in which the occupant's hand is extended as inferred by the physis inference section (5), the face direction of the occupant detected by the face direction detection section (6) and seat information acquired by the seat information acquisition section (21), assesses the direction in which the occupant's attention is directed, where, if the direction inferred by the physique inference section (5) in which the occupant's hand is extended is a rear seat direction, the face direction of the occupant detected by the face direction detection section (6) is the rear seat direction, and furthermore, the seat information acquired by the seat information acquisition section (21) shows that there is no occupant in the rear seat, the attention direction assessment section (7) judges that the direction in which the occupant's attention is directed is the rear seat direction. [3] Attention direction assessment device (200) according to claim 1 or 2, the one head position inference section (3), which infers a relative position of the occupant's head to the camera (9) based on facial features inferred by the facial feature inference section (2), and a body part inference section (4) which infers the size of a plurality of body parts of the inmate in the image based on the image acquired by the image acquisition section (1), wherein the physis inference section (5) infers a physis of the occupant based on the relative position of the occupant's head inferred by the head position inference section (3) and the size of the respective body parts inferred by the body part inference section (4), and infers the direction in which the occupant's hand is extended based on the inferred physis inference result of the occupant. [4] Attention direction assessment device (200) according to one of claims 1 to 3, which further comprises an assessment result output section (8) for outputting the direction in which the occupant's attention is directed as assessed by the attention direction assessment section (7). [5] Attention direction assessment procedure, which a step towards acquiring an image of occupants in a vehicle, captured by a camera (9), a step in which facial features of an occupant are inferred based on the image, a step in which, based on the image, a direction is inferred in which an occupant's hand is extended; a step in which, based on facial features, a facial direction of the occupant is detected. a step in which seating information of the vehicle's occupants is acquired, and a step in which, based on the direction in which the occupant's hand is extended, the direction of the occupant's face and the seating information, the direction in which the occupant's attention is directed is assessed, where, if the direction inferred by a physis inference section (5) in which the occupant's hand is extended is a neighboring seat direction, the direction of the occupant's face detected by a face direction detection section (6) is the neighboring seat direction, and furthermore, the seat information acquired by a seat information acquisition section (21) shows that there is no occupant in the neighboring seat, an attention direction assessment section (7) judges that the direction in which the occupant's attention is directed is the neighboring seat direction. [6] Attention direction assessment procedure, which a step towards acquiring an image of occupants in a vehicle, captured by a camera (9), a step in which facial features of an occupant are inferred based on the image, a step in which, based on the image, a direction is inferred in which one of the occupant's hands is extended, a step in which, based on facial features, a facial direction of the occupant is detected, a step in which seating information of the vehicle's occupants is acquired, and a step in which, based on the direction in which the occupant's hand is extended, the direction of the occupant's face and the seating information, the direction in which the occupant's attention is directed is assessed, where if the direction inferred by a physique inference section (5) in which the occupant's hand is extended is a rear seat direction, the face direction of the occupant detected by a face direction detection section (6) is the rear seat direction, and furthermore the seat information acquired by a seat information acquisition section (21) shows that there is no occupant in the rear seat, an attention direction assessment section (7) judges that the direction in which the occupant's attention is directed is the rear seat direction.

Images