Face information registration support device

Abstract

The objective of the present disclosure is to reduce a load on a vehicle occupant in a facial information registration process used for face authentication of the occupant. A facial information registration assistance device (101) is provided with: an occupant information processing unit (12) for calculating, from a facial image of an occupant, occupant information that includes at least a line-of-sight direction and position information about the face of the occupant; a control information calculation unit (15) for calculating, from the occupant information, control information that includes the effective field of view range of the occupant; a control information determination unit (16) for determining whether or not a display device (2) is in the effective field of view range; and a notification control unit (18) for providing a notification for prompting the occupant to move the face. The notification includes a voice notification which outputs a voice for prompting the occupant to look at the display device (2), and a display notification which causes the display device (2) to display guidance information for guiding the occupant to move the face. The notification control unit (18) provides the voice notification when the display device (2) does not exist in the effective field of view range, and provides the display notification when the display device (2) exists in the effective field of view range.

Description

[Technical field]

[0001] The present disclosure relates to assistance in registering face information used for face authentication of vehicle occupants. [Background technology]

[0002] In recent years, facial recognition has come to be used as a biometric authentication method for vehicle occupants. This facial recognition is performed not only for the driver but also for other occupants in the vehicle in order to provide services optimized for each occupant.

[0003] Facial recognition is performed by, for example, comparing an image of the face of a vehicle occupant with a facial image registered in advance (hereinafter also referred to as a "registered image"). Therefore, it is necessary to register the facial image of the occupant as a preliminary step to facial recognition.

[0004] In the second embodiment of Patent Document 1, a configuration is disclosed in which, in a facial image registration process, if the position or orientation of an occupant's face is inappropriate, the position or orientation of the face is guided by an image or sound. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2019-028959 A Summary of the Invention [Problem to be solved by the invention]

[0006] When the occupant's face is guided by an image, it is difficult for the occupant to adjust the position or direction of the face while checking the guidance information displayed on the screen. Also, when the occupant's face is guided by voice, it is difficult for the occupant to correctly understand the destination of the position or direction of the face by voice alone.

[0007] In order to improve the accuracy of face recognition, it is desirable that the registered images are images of the occupant's face photographed from various angles. In order to register the images of the variations, the face must be guided multiple times, which places a heavy burden on the occupant when registering the face image. Here, the registered image is given as an example, but the image used for face recognition is not limited to a face image, and face information representing the amount of facial features may be used.

[0008] The present disclosure has been made to solve the above problems, and aims to reduce the burden on vehicle occupants in the registration process of face information used for face authentication of vehicle occupants. [Means for solving the problem]

[0009] The face information registration support device disclosed herein is a face information registration support device that supports the registration of face information for performing face authentication of a vehicle occupant, and includes an acquisition unit that acquires a face image of the occupant captured by an imaging device provided in the vehicle, an occupant information processing unit that calculates occupant information including at least position information and gaze direction of the occupant's face from the face image of the occupant, a control information calculation unit that calculates control information including an effective visual field range of the occupant from the occupant information, a control information determination unit that determines whether a display device provided in the vehicle is present within the effective visual field range, and a notification control unit that issues a notification to the occupant to encourage him or her to move their face, the notification including an audio notification that causes an audio output device provided in the vehicle to output an audio that encourages the occupant to view the display device, and guidance information that guides the occupant to move at least one of the position and orientation of their face to be appropriate for registering the face information. ,The target is prompted to be visually recognized by the voice notification. and a display notification to be displayed on the display device, and the notification control unit The target is prompted to be visually recognized by the sound of the voice notification. If the display device is not within the effective field of view, an audio notification is given. The target is prompted to be visually recognized by the sound of the voice notification. When the display device is within the effective visual field range, a display notification is given. Effect of the Invention

[0010] According to the face information registration support device of the present disclosure, it is possible to reduce the burden on the vehicle occupant in the registration process of face information used for face authentication of the vehicle occupant. Objects, features, aspects, and advantages of the present disclosure will become more apparent from the following detailed description and the accompanying drawings. [Brief description of the drawings]

[0011] [Figure 1] 1 is a block diagram showing a configuration of a face information registration support device according to a first embodiment. [Diagram 2] FIG. 2 is a diagram showing the arrangement of an imaging device, a display device, and an audio output device in a vehicle cabin. [Diagram 3] 11 is a perspective view showing a case where the display device is not present within the effective visual field of an occupant; FIG. [Figure 4] FIG. 11 is a perspective view showing a case where a display device is present within the effective visual field range of an occupant. [Diagram 5] FIG. 2 is a perspective view showing the effective visual field range of an occupant and the coordinates of a display device. [Figure 6] FIG. 2 is an xz plane view showing the effective visual field range of an occupant and the coordinates of a display device. [Figure 7] FIG. 2 is a diagram showing the facial positions and facial orientations of an occupant at times t1 and t2. [Figure 8] 4 is a flowchart showing an operation of the face information registration support device of the first embodiment. [Figure 9] FIG. 11 is a block diagram showing a configuration of a face information registration support device according to a second embodiment. [Figure 10] FIG. 2 is a perspective view showing the viewing direction of an occupant and the coordinates of a display device. [Figure 11] FIG. 2 is an xz plan view showing the viewing direction of an occupant and the coordinates of a display device. [Figure 12] FIG. 2 is a yz plane view showing the viewing direction of an occupant and the coordinates of a display device. [Figure 13] 11 is a flowchart showing the operation of the face information registration support device of the embodiment 2. [Figure 14] 1 is a diagram illustrating a hardware configuration of a face information registration support device according to a first embodiment. [Figure 15]This is a diagram showing the hardware configuration of the face information registration support device according to Embodiment 1.

Embodiments for Carrying Out the Invention

[0012] <A. Embodiment 1> <A-1. Configuration> FIG. 1 is a diagram showing the configuration of a face information registration support device 101 according to Embodiment 1. The face information registration support device 101 is connected to an imaging device 1, a display device 2, and an audio output device 3, and is configured to be able to use these. The face information registration support device 101 is a device that supports the registration of face information used for face authentication of vehicle occupants. The imaging device 1, the display device 2, and the audio output device 3 are all mounted in the interior of the vehicle (hereinafter referred to as the “passenger compartment”). In this specification, the vehicle refers to the vehicle in which the occupant who is the target for the face information registration support device in each embodiment boards. Also, in this specification, the occupant refers to the occupant who is the target for the face information registration support device in each embodiment. The face information registration support device 101 is also typically mounted on the vehicle, but a part of its configuration may be constituted by a server existing outside the vehicle, a portable terminal carried by the occupant and brought into the vehicle, or the like.

[0013] FIG. 2 is a diagram showing the arrangement of the imaging device 1, the display device 2, and the audio output device 3 in the passenger compartment.

[0014] In FIG. 2, cameras 51 and 52 are an example of the imaging device 1. The camera 51 is installed at a position in the passenger compartment where it can photograph the occupant from the front and above. The camera 52 is installed below the camera 51 and above the CID (Center Information Display) 21, and photographs the occupant from the front. Although a plurality of cameras 51 and 52 are shown in FIG. 2, the number of cameras constituting the imaging device 1 may be single or plural. The face images of the occupant photographed by the cameras 51 and 52 are used for face information registration in the face information registration support device 101.

[0015] 2, the CID 21, the meter cluster panel 22, the passenger seat display 23, and the HUD (Head Up Display) 24 are examples of the display device 2. The CID 21 is installed in front between the driver's seat 41 and the passenger seat 42. The meter cluster panel 22 is installed in front of the steering wheel 40. The passenger seat display 23 is installed in front of the passenger seat 42. The HUD 24 is installed above the meter cluster panel 22. In addition, a windshield 25 turned into a display may be included in the display device 2. When the imaging device 1 captures an image of the passenger's face, guidance information is displayed on the display device 2 to guide the position and orientation of the passenger's face in a specific direction.

[0016] 2, speakers 31 and 32 are examples of the audio output device 3. The speaker 31 is installed near the driver's seat 41, and the speaker 32 is installed near the passenger seat 42. The audio output device 3 outputs a sound that prompts the occupant to view the display device 2.

[0017] The configuration of the face information registration support device 101 will be described with reference to Fig. 1. The face information registration support device 101 includes an acquisition unit 11, an occupant information processing unit 12, a registration information determination unit 13, an occupant information storage unit 14, a control information calculation unit 15, a control information determination unit 16, an output information generation unit 17, a notification control unit 18, a unique information storage unit 19, and a registration information storage unit 20.

[0018] The acquisition unit 11 acquires, from the imaging device 1, the facial image of the occupant captured by the imaging device 1.

[0019] The occupant information processing unit 12 acquires the face image of the occupant from the acquisition unit 11, and acquires occupant information from the face image of the occupant. The occupant information includes at least any one of the face position, face direction, line of sight direction, and eye opening degree of the occupant.

[0020] The registration information determination unit 13 acquires a facial image of the occupant from the occupant information processing unit 12, and determines whether or not the facial image of the occupant satisfies a predetermined registration condition. The registration information determination unit 13 may be configured by a DMS (Driver Monitoring System), which is an existing face information registration system.

[0021] The occupant information storage unit 14 acquires and stores occupant information from the occupant information processing unit 12. The occupant information storage unit 14 may store the occupant information acquired by the occupant information processing unit 12 in one processing each time, or may collectively store the occupant information acquired in multiple processings.

[0022] The control information calculation unit 15 acquires the occupant information from the occupant information storage unit 14 and calculates the control information from the occupant information. The control information includes the amount of face movement and the effective visual field range of the occupant. The method of calculating the control information will be described in detail later.

[0023] The control information determination unit 16 acquires the control information from the control information calculation unit 15, and determines whether or not the acquired control information satisfies a predetermined condition.

[0024] The output information generating unit 17 generates output information according to the determination result of the control information determining unit 16. The output information includes either audio information that prompts the user to view the display device 2 or guidance information that guides the position and orientation of the face to a predetermined position and orientation.

[0025] The notification control unit 18 acquires output information from the output information generation unit 17, and provides the output information to either the display device 2 or the audio output device 3 according to the determination result of the control information determination unit 16. That is, the notification control unit 18 controls whether to notify the occupant by display or audio according to the determination result of the control information determination unit 16.

[0026] The unique information storage unit 19 stores unique information used by the face information registration assistance device 101. The unique information may include, for example, the position coordinates of the display device 2 or a threshold value for the amount of face movement.

[0027] When the registration information determination unit 13 determines that the face image satisfies the registration conditions, the registration information storage unit 20 stores, that is, registers, the face information regarding the face image. The face information is information used for identifying the face of the occupant in the face authentication process, and may be, in addition to the face image itself, a score obtained by evaluating the face image with a machine learning model. Note that the registered face information is also referred to as registration information.

[0028] <A-2. Operation> FIGS. 3 and 4 show the positional relationship between the effective visual field range of the occupant and the display device. In these figures, the CID 21 and the meter cluster panel 22 correspond to the display device 2. The control information calculation unit 15 sets the effective visual field range A2 in consideration of the effective visual field of a human, based on the line-of-sight direction A1 of the occupant P included in the occupant information. The effective visual field range A2 is represented as a range of a visual angle θ1 centered on the line-of-sight direction A1.

[0029] In the example of FIG. 3, neither the CID 21 nor the meter cluster panel 22 exists within the effective visual field range A2. For example, the control information determination unit 16 identifies that the CID 21 is not within the effective visual field range A2 by comparing the effective visual field range A2 with the position A3 of the CID 21. Similarly, the control information determination unit 16 identifies that the meter cluster panel 22 is not within the effective visual field range A2 by comparing the effective visual field range A2 with the position A3 of the meter cluster panel 22. Even if guidance information for guiding the position and orientation of the occupant's face is displayed on the CID 21 or the meter cluster panel 22 in this state, it will not be visually recognized by the occupant. Therefore, the face information registration support device 101 prompts the occupant to visually recognize the display device 2 by voice. For example, as shown in FIG. 3, a voice "Please move your face to the right" is output from the speaker 31. When the occupant who hears this voice moves the face to the right, the effective visual field range A2 of the occupant moves to the right and overlaps with the CID 21 as shown in FIG. 4. In the above example, the movement of the face orientation is instructed by voice, but the movement of the face position may be instructed together with, or instead of, the face orientation.

[0030] 4 illustrates a case where CID 21 is present in the effective visual field range A2. In this state, the face information registration support device 101 displays guidance information A4 for guiding the position and orientation of the occupant's face on CID 21. The occupant can visually recognize the guidance information A4 without changing the position and orientation of the face, and then change the position and orientation of the face according to the guidance information A4.

[0031] Fig. 5 and Fig. 6 show the position coordinates of the occupant information and the control information in the xyz coordinate system. In Fig. 5, the horizontal and vertical directions of the display surface of CID 21 are the x-axis and y-axis directions, respectively, and the direction perpendicular to the display surface of CID 21 is the z-axis direction. Fig. 6 is an xz plane view.

[0032] The position coordinates of CID21 are (x1, y1, z1). The position coordinates of the occupant's right eye are (rx1, ry1, rz1), and the position coordinates of the left eye are (lx1, ly1, lz1). The position coordinates of the intersection between the xy plane parallel to the display surface of CID21 and the line of sight A1r of the right eye are (rx2, ry2, rz2). Similarly, the position coordinates of the intersection between the xy plane parallel to the display surface of CID21 and the line of sight A1l of the left eye are (lx2, ly2, lz2). If the effective visual field range A2r of the right eye is a circular area of ​​radius r1 from the line of sight A1r of the right eye, the radius r1 of the effective visual field range A2r is calculated by formula (1).

[0033]

number

[0034] The effective visual field range of the left eye is calculated in the same manner. Then, the effective visual field range A2r of the right eye and the effective visual field range of the left eye are combined to calculate the effective visual field range A2.

[0035] The face information registration support device 101 registers multiple facial images of the occupant by changing the position or orientation of the face. Therefore, the guiding of the occupant's face and the photographing and registration of the face image are repeated multiple times. Therefore, the face information registration support device 101 calculates the amount of facial movement of the occupant in the control information calculation unit 15 so as not to perform the next guidance while the occupant is moving his face in accordance with the guidance, and determines whether the occupant has completed the movement of his face from the amount of facial movement in the control information determination unit 16.

[0036] Fig. 7 shows facial images of occupant P at two times t1 and t2. The facial movement amount m1 between times t1 and t2 is expressed as the sum of the movement amount of the facial position between times t1 and t2 and the movement amount of the facial direction between times t1 and t2. If the facial position coordinates at time t1 are (x2, y2, z2) and the facial direction are (θx1, θy1), and the facial position coordinates at time t2 are (x3, y3, z3) and the facial direction are (θx2, θy2), the facial movement amount m1 is calculated by the following formula (2).

[0037]

number

[0038] 8 is a flowchart showing the operation of the face information registration support device 101. The operation of the face information registration support device 101 will be described below with reference to the flowchart in FIG.

[0039] When the face information registration process by the face information registration support device 101 starts, the imaging device 1 captures a face image of the occupant, and the acquisition unit 11 acquires the face image of the occupant from the imaging device 1 (step S101). Next, the occupant information processing unit 12 acquires the face image of the occupant from the acquisition unit 11 and calculates occupant information from the face image of the occupant (step S102). Here, the occupant information includes at least one of the face position, face direction, gaze direction, and eye opening degree of the occupant. Note that a method for calculating the face position and direction is disclosed in Patent Document 1, a method for calculating the gaze direction is disclosed in Japanese Patent Application Laid-Open No. 2019-152957, and a method for calculating the eye opening degree is disclosed in Japanese Patent Application Laid-Open No. 2020-517707. Therefore, detailed explanations of these are omitted. The occupant information calculated by the occupant information processing unit 12 is stored in the occupant information storage unit 14.

[0040] After step S102, the registration information determination unit 13 determines whether or not face information needs to be registered (step S103). Specifically, the registration information determination unit 13 acquires registration information from the registration information storage unit 20, and determines that face information needs to be registered when occupant information is not included in the registration information. The registration information determination unit 13 may also determine that face information needs to be registered when an occupant performs a specific input operation on a display device such as a touch panel. There may be one or more occupants whose face information is to be registered.

[0041] If the registration information determination unit 13 determines in step S102 that registration of occupant information is not necessary, the process of the face information registration support device 101 returns to step S101. If the registration information determination unit 13 determines in step S102 that registration of occupant information is necessary, the process of the face information registration support device 101 proceeds to step S104 and subsequent steps. In step S104, the control information calculation unit 15 calculates the effective visual field range of the occupant from the occupant information calculated in step S102. Specifically, the control information calculation unit 15 sets a range having a certain angle with respect to the line of sight from the face position or eye position included in the occupant information calculated in step S102 as the effective visual field range visible to the occupant. The certain angle is appropriately set in consideration of the effective visual field of a human being with respect to the line of sight, and is generally 4 degrees to 20 degrees. The detailed calculation method of the effective visual field range is as described with reference to FIG. 4 to FIG. 6. In addition to the above method, the control information calculation unit 15 may set the effective visual field range using information such as the degree of eye opening.

[0042] After step S104, the control information determination unit 16 determines whether or not the display device 2 is present within the effective field of view range (step S105). Specifically, the control information determination unit 16 determines whether or not the display device 2 is present within the effective field of view range by comparing the effective field of view range with the position coordinates of the display device 2. If the display device 2 is not present within the effective field of view range, the process of the face information registration support device 101 proceeds to step S106, and if the display device 2 is present within the effective field of view range, the process of the face information registration support device 101 proceeds to step S107.

[0043] In step S106, the face information registration support device 101 prompts the occupant to view the display device 2 by audio. Specifically, the output information generation unit 17 creates audio information prompting the occupant to view the display device 2, and the notification control unit 18 provides the audio information to the audio output device 3. As a result, audio prompting the occupant to view the display device 2 is output from the audio output device 3. For example, audio such as "Please look at the display device as your face information will be registered" is output from the audio output device 3. After step S106, the processing of the face information registration support device 101 proceeds to step S101.

[0044] In step S107, the face information registration support device 101 guides the position and orientation of the occupant's face by displaying it. Specifically, the output information generation unit 17 creates guidance information for guiding the movement of the position and orientation of the face. Then, the notification control unit 18 provides the guidance information to the display device 2. As a result, a display is performed on the display device 2 to guide the position and orientation of the occupant's face. This guidance display is also referred to as a display notification. Here, for example, an image in which an animation indicating the movement direction is added to the face image of the occupant captured by the imaging device 1 is displayed. Also, a face model created with a 3D mesh may be used instead of a face image.

[0045] In step S108, the registration information determination unit 13 determines whether or not the registration of the variation images required for the face recognition processing has been completed. The variation images are, for example, images of the face facing forward, up, down, left, and right. If the registration of the variation images has been completed, the processing of the face information registration support device 101 ends. On the other hand, if the registration of the variation images has not been completed, the processing of the face information registration support device 101 proceeds to step S109.

[0046] In step S109, the imaging device 1 captures a facial image of the occupant, and the acquisition unit 11 acquires the facial image of the occupant from the imaging device 1. Thereafter, the occupant information processing unit 12 calculates occupant information from the facial image of the occupant acquired in step S109 (step S110). Here, the occupant information includes at least one of the occupant's facial position, facial direction, line of sight, and eye opening degree. The occupant information calculated by the occupant information processing unit 12 is stored in the occupant information storage unit 14.

[0047] After step S110, the control information calculation unit 15 calculates the amount of face movement as control information to grasp the operating status of the occupant (step S111). Here, the occupant information calculated in step S110, i.e., the current occupant information, is defined as the first occupant information, and the previous occupant information stored in the occupant information storage unit 14 is defined as the second occupant information. The control information calculation unit 15 calculates the sum of squares of the difference in each axial component between the face position of the first occupant information and the face position of the second occupant information as the amount of movement of the face position. In addition, the control information calculation unit 15 calculates the sum of squares of the difference in each axial component between the face direction of the first occupant information and the face direction of the second occupant information as the amount of movement of the face direction. Then, the control information calculation unit 15 calculates the sum of the amount of movement of the face position and the amount of movement of the face direction as the amount of movement of the face. A detailed calculation method of the amount of movement of the face movement is as follows: <a-1>As explained in.

[0048] After step S111, the control information determination unit 16 determines whether the face movement is completed (step S112). The control information determination unit 16 acquires a predetermined face movement amount threshold from the unique information storage unit 19, and determines that the face movement is completed if the face movement amount calculated in step S111 exceeds the face movement amount threshold.

[0049] If the face movement is not complete in step S112, the process of the face information registration support device 101 returns to step S109. If the face movement is complete in step S112, the process of the face information registration support device 101 proceeds to step S113.

[0050] In step S113, the control information calculation unit 15 calculates the effective visual field range of the occupant from the occupant information acquired in step S110. Then, the control information determination unit 16 determines whether or not any display device 2 is present in the effective visual field range (step S114).

[0051] If the display device 2 is not present within the effective field of view in step S114, the process of the face information registration support device 101 proceeds to step S115. If the display device 2 is present within the effective field of view in step S114, the process of the face information registration support device 101 proceeds to step S107, where guidance information for guiding the face to the next position and orientation is displayed.

[0052] In step S115, the face information registration support device 101 prompts the occupant to visually recognize the arbitrary display device 2 by voice. Specifically, the output information generation unit 17 creates voice information for prompting the visual recognition of the display device 2, and the notification control unit 18 provides the voice information to the voice output device 3. As a result, a voice for prompting the visual recognition of the display device 2 is output from the voice output device 3. This voice is, for example, "Please move your face position to the right" or the like. The output information generation unit 17 may create voice information in advance for each pattern with different face movement directions such as up, down, left, right, and diagonal, and apply the nearest neighbor method to the current face position and orientation of the occupant and the expected value to select the voice information. Since the nearest neighbor method is well-known, a detailed description thereof is omitted.

[0053] In the case where the display device 2 is a front windshield formed into a display or an in-pan el integrated display, and the display range of the display device 2 is extremely large compared to the effective visual field range, the face information registration support device 101 may change the display position of the guidance information within the effective visual field range instead of prompting the occupant to visually recognize the display device 2 by voice.

[0054] <A-3. Effect> The face information registration support device 101 according to Embodiment 1 supports the registration of face information for performing face authentication of an occupant boarding a vehicle. The face information registration support device 101 includes an acquisition unit 11 that acquires a face image of the occupant captured by an imaging device 1 provided in the vehicle, an occupant information processing unit 12 that calculates occupant information including at least the position information and the line-of-sight direction of the occupant's face from the face image of the occupant, a control information calculation unit 15 that calculates control information including the effective visual field range of the occupant from the occupant information, a control information determination unit 16 that determines whether a display device 2 provided in the vehicle exists within the effective visual field range, and a notification control unit 18 that gives a notification prompting the occupant to move the face. The notification by the notification control unit 18 includes an audio notification that causes an audio output device 3 provided in the vehicle to output an audio prompting the occupant to visually recognize the display device 2, and a display notification that causes the display device 2 to display guidance information for guiding the occupant to move at least one of the position and the orientation of the face. The notification control unit 18 gives an audio notification when the display device 2 does not exist within the effective visual field range, and gives a display notification when the display device 2 exists within the effective visual field range. With the above configuration, the face information registration support device 101 performs face guidance only by display when the occupant is visually recognizing the display device 2, and prompts the occupant to visually recognize the display device 2 by voice if the occupant is not visually recognizing the display device 2. Therefore, after the occupant visually recognizes the display device 2 according to the voice, and moves the face according to the information displayed on the display device 2, the face information registration process can be smoothly performed, and the burden on the occupant in the face information registration process is reduced.

[0055] Further, in the face information registration support device 101, the control information includes the amount of face movement of the occupant calculated from the occupant information calculated from the face images acquired before and after the display notification, the control information determination unit 16 compares the amount of face movement with a predetermined threshold value, and the notification control unit 18 may give the next display notification when the amount of face movement is equal to or greater than the threshold value. Thereby, the face information registration support device 101 can give the next display notification to the occupant at the timing when the movement of the occupant's face is completed according to the previous display notification.

[0056] <B. Embodiment 2> <B-1. Configuration> In the second embodiment, the same components as those in the first embodiment are given the same reference numerals, and the description thereof will be omitted as appropriate. Fig. 9 is a diagram showing the configuration of a face information registration support device 102 according to the second embodiment. The face information registration support device 102 is connected to an imaging device 1, a display device 2, and an audio output device 3, and is configured to be able to use these. The face information registration support device 102 is a device that supports the registration of face information used for face authentication of vehicle occupants.

[0057] The face information registration support device 102 comprises an acquisition unit 11, an occupant information processing unit 12, a control information calculation unit 15, a control information determination unit 16, an output information generation unit 17, a notification control unit 18, a unique information storage unit 19, and a display device control unit 20A. The display device control unit 20A is attached to the display device 2 and has a mechanism for adjusting the position and orientation of the display device 2. Note that a method for adjusting the position and orientation of the display device is disclosed in Japanese Patent Application Laid-Open No. 20007-78172, and therefore a detailed description thereof will be omitted.

[0058] The control information calculation unit 15 acquires the occupant information from the occupant information processing unit 12, and calculates the control information from the occupant information. Here, the control information includes the effective visual field range of the occupant and the amount of movement of the display device. The amount of movement of the display device is the amount of movement of the position and angle of the display device 2 to the position and angle at which the display surface becomes perpendicular to the line of sight of the occupant.

[0059] The control information determination unit 16 acquires the effective visual field range of the occupant from the control information calculation unit 15, and determines whether or not the effective visual field range satisfies a predetermined condition.

[0060] The output information generating unit 17 generates output information according to the determination result of the control information determining unit 16 .

[0061] Based on the judgment result of the control information judgment unit 16, the notification control unit 18 switches whether to output the output information as a display or as an audio, and provides the output information generated by the output information generation unit 17 to either the display device 2 or the audio output device 3.

[0062] The unique information storage unit 19 stores unique information used by the face information registration support device 101. Examples of the unique information include the position coordinates of the display device 2.

[0063] The display device control unit 20A acquires the display direction of the display device 2 from the control information calculation unit 15 via the notification control unit 18, and acquires the determination result of the control information determination unit 16 via the notification control unit 18. Then, the display device control unit 20A changes the position and display direction of the display device 2 based on the determination result of the control information determination unit 16. Details will be described in detail later.

[0064] <B-2. Operation> Figures 10 to 12 show an example of a method for controlling the display device 2 by the face information registration support device 102. In these figures, the control of the position and display direction of the CID21 is described as an example of the display device 2. Figure 10 is a perspective view showing the positional relationship between the occupant P and the CID21. In Figure 10, the horizontal and vertical directions of the display surface of the CID21 are defined as the x-axis direction and the y-axis direction, respectively, and the direction perpendicular to the display surface of the CID21 is defined as the z-axis direction. Figure 11 is an xz plane view showing the positional relationship between the occupant P and the CID21, and Figure 12 is a yz plane view showing the positional relationship between the occupant P and the CID21. Although the change in the display direction of the CID21 by the display device control unit 20A is shown in Figures 11 and 12, the x, y, and z axes in these figures are set based on the CID21 before the change.

[0065] Let the absolute position coordinates of the position A3 of CID21 before adjustment in the passenger compartment be (x1, y1, z1), and let the absolute position coordinates of the position A3a of CID21 after adjustment in the passenger compartment be (x1, y1a, z1). Let the position coordinates of the midpoint A5 of the straight line connecting the left and right eyes of the occupant be (x3, y3, z3). Let the straight line connecting the position A3 of CID21 before adjustment and the midpoint A5 be A6. Let the angle formed by the straight line A6 and the y-axis direction be θ2. Let the straight line connecting the position A3a of CID21 after adjustment and the midpoint A5 be A6a. Let the angle formed by the straight line A6a and the vertical direction of the display surface of CID21 after adjustment be θ2a. Let the angle formed by the straight line A6a and the horizontal direction of the display surface of CID21 after adjustment be θ3. Let the angular movement amount (rotation angle) of CID21 in the yz plane be θ4. Note that 0 < θ4 < maxθ4, and maxθ4 is the maximum value of the rotation angle of CID21 in the yz plane. θ5 is the angle formed by the horizontal direction of the display screen of CID21 after adjustment and the x-axis, in other words, it is the angular movement amount (rotation angle) of CID21 in the xz plane. By adjusting the position and angle of CID21 in the direction where the angle θ2a formed by the straight line A6a and the y-axis direction, and the angle θ3 formed by the straight line A6a and the x-axis direction are each 90 degrees, CID21 moves in the front direction of the occupant.

[0066] For example, the amount of position movement m2 of the display device 2 in the y-axis direction is calculated by Equation (3). Also, the angular movement amount θ4 of the display device 2 in the yz plane is calculated by Equation (4). Note that 0 < m2 < maxm2, and maxm2 is the maximum value of the amount of position movement m2 of the display device 2 in the y-axis direction.

[0067]

Number

[0068]

Number

[0069] Figure 13 is a flowchart showing the operation of the face information registration support device 102. Hereinafter, the operation of the face information registration support device 102 will be described according to the flowchart of Figure 13.

[0070] The face information registration support device 102 starts the face information registration process in step S201. The face information registration process is started when, for example, a specific input operation is performed by an occupant on a display device such as a touch panel. The number of occupants whose occupant information is to be registered may be one or more.

[0071] Next, the imaging device 1 captures a facial image of the occupant, and the acquisition unit 11 acquires the facial image of the occupant from the imaging device 1 (step S202). Next, the occupant information processing unit 12 acquires the facial image of the occupant from the acquisition unit 11, and calculates occupant information from the facial image of the occupant (step S203). Here, the occupant information includes at least one of the occupant's facial position, facial direction, line of sight, and eye opening degree. The occupant information calculated by the occupant information processing unit 12 is stored in the occupant information storage unit 14.

[0072] Next, the face information registration support device 102 notifies the occupant by voice that the display device 2 will move, and prompts the occupant to view the display device 2 (step S204). Specifically, the output information generation unit 17 creates voice information prompting the occupant to view the display device 2, and the notification control unit 18 provides the voice information to the voice output device 3. As a result, a voice prompting the occupant to view the display device 2 is output from the voice output device 3. For example, a voice such as "The display device will be moved to a position where it is easy to see, please look at the display device" is output from the voice output device 3. Here, the voice output from the voice output device 3 may be any voice that notifies the occupant that the display device 2 will move and prompts the occupant to view the display device 2.

[0073] Next, the control information calculation unit 15 calculates the effective visual field range of the occupant from the occupant information calculated in step S203 (step S205). After that, the control information determination unit 16 determines whether the display device 2 is present within the effective visual field range (step S206). If the display device 2 is not present within the effective visual field range in step S206, the process of the face information registration support device 102 returns to step S204.

[0074] When the display device 2 exists within the effective visual field range in step S206, the process of the face information registration support device 102 proceeds to step S207. In step S207, the control information calculation unit 15 calculates the position and orientation of the display device 2 that are most visible, that is, optimal, for the occupant, and calculates the amount of movement of the position and orientation of the display device 2 to the said position and orientation (step S207). Specifically, the control information calculation unit 15 acquires the position information of the occupant's eyes from the occupant information, and calculates, as the display device movement amount, the amount of movement to the position and angle at which the display surface of the display device 2 becomes perpendicular to the straight line connecting the midpoint of the positions of both eyes of the occupant and the position of the display device 2. The detailed calculation method of the display device movement amount is as described with reference to FIGS. 10 to 12.

[0075] Next, the display device control unit 20A moves the display device 2 by the amount of movement of the display device acquired from the control information calculation unit 15 (step S208).

[0076] Thereafter, the face information registration support device 102 guides the position and orientation of the occupant's face by display (step S209). This step is the same as step S107 in FIG. 8.

[0077] <B-3. Effect> The face information registration support device 102 according to the second embodiment includes a display device control unit 20A that rotates the display device so that the display surface of the display device becomes perpendicular to the line of sight direction of the occupant when the display device 2 exists within the effective visual field range. Then, the notification control unit 18 performs a display notification after the rotation of the display device 2. Therefore, the visibility of the display notification by the occupant is improved.

[0078] <C. Hardware Configuration> Next, the hardware configuration of the main parts of the face information registration support devices 101 and 102 will be described. The face information registration support devices 101 and 102 are configured by a computer, and the computer has a processor 81 and memories 82 and 83 as shown in FIG. 14. The memory 82 stores a program for causing the computer to function as the acquisition unit 11, the occupant information processing unit 12, the registration information determination unit 13, the control information calculation unit 15, the control information determination unit 16, the output information generation unit 17, the notification control unit 18, and the unique information storage unit 19 (hereinafter also referred to as the "acquisition unit 11, etc."). The processor 81 reads out and executes the program stored in the memory 82, thereby realizing the functions of the acquisition unit 11, etc. The memory 83 also realizes the functions of the occupant information storage unit 14 and the registration information storage unit 20.

[0079] Alternatively, the face information registration support devices 101 and 102 may include a memory 83 and a processing circuit 84. In this case, the functions of the acquisition unit 11 and the like may be realized by the processing circuit 84.

[0080] Alternatively, the face information registration support devices 101, 102 may have a processor 81, memories 82, 83, and a processing circuit 84. In this case, some of the functions of the acquisition unit 11 and the like may be realized by the processor 81 and the memory 82, and the remaining functions may be realized by the processing circuit 84.

[0081] The processor 81 may be, for example, a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor, a microcontroller, or a digital signal processor (DSP).

[0082] For example, a semiconductor memory or a magnetic disk is used as the memories 82 and 83. More specifically, the memory 82 is a random access memory (RAM), a read only memory (ROM), a flash memory, an erasable programmable read only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a solid state drive (SSD), a hard disk drive (HDD), or the like.

[0083] The processing circuit 84 may be implemented using, for example, an Application Specific Integrated Circuit (ASIC), a Programmable Logic Device (PLD), a Field-Programmable Gate Array (FPGA), a System-on-a-Chip (SoC), or a system Large-Scale Integration (LSI).

[0084] In addition, each embodiment can be freely combined, and each embodiment can be appropriately modified or omitted. The above description is an example in all respects. It is understood that countless modified examples not illustrated can be assumed. [Explanation of symbols]

[0085] 1 imaging device, 2 display device, 3 audio output device, 11 acquisition unit, 12 occupant information processing unit, 13 registration information determination unit, 14 occupant information storage unit, 15 control information calculation unit, 16 control information determination unit, 17 output information generation unit, 18 notification control unit, 19 unique information storage unit, 20 registration information storage unit, 20A display device control unit, 21 CID, 22 meter cluster panel, 23 passenger seat display, 25 windshield, 31, 32 speakers, 40 steering wheel, 41 driver's seat, 42 passenger seat, 51, 52 cameras, 81 processor, 82, 83 memory, 84 processing circuit, 101, 102 face information registration support device.

Claims

1. A face information registration support device that supports registration of face information for performing face authentication of a passenger in a vehicle, comprising: an acquisition unit that acquires a facial image of the occupant captured by an imaging device provided in the vehicle; an occupant information processing unit that calculates occupant information including at least position information and a line of sight direction of the occupant's face from the face image of the occupant; a control information calculation unit that calculates control information including an effective visual field range of the occupant from the occupant information; a control information determination unit that determines whether a display device provided in the vehicle is present within the effective visual field; A notification control unit that notifies the occupant to move his / her face, The notification includes a voice notification that outputs a voice from a voice output device provided in the vehicle to prompt the occupant to visually confirm the display device, and a display notification that displays guidance information on the display device that is the target of the visual notification, the guidance information instructing the occupant to move at least one of the position and the orientation of the face so as to be appropriate for registering the face information, by the voice of the voice notification. The notification control unit performs the audio notification when the display device to be prompted to be viewed by the audio of the audio notification is not within the effective field of view, and performs the display notification when the display device to be prompted to be viewed by the audio of the audio notification is within the effective field of view. Facial information registration support device.

2. the control information includes a facial movement amount of the occupant calculated from the occupant information calculated from the facial images acquired before and after the display notification, the control information determination unit compares the amount of face movement with a predetermined threshold; The notification control unit performs the next display notification when the face movement amount is equal to or greater than the threshold value. The face information registration support device according to claim 1 .

3. The occupant information includes an eye opening degree of the occupant. The face information registration support device according to claim 1 .

4. The vehicle further includes a display device control unit that, when the display device that is to be prompted to be viewed by the sound of the voice notification is within the effective visual field, rotates the display device so that a display surface of the display device that is to be prompted to be viewed by the sound of the voice notification is perpendicular to the line of sight of the occupant, The notification control unit performs the display notification after rotating the display device of the target that is prompted to be visually recognized by the sound of the audio notification. The face information registration support device according to claim 1 .

Images