Image processing apparatus and image processing method

Abstract

An image processing apparatus includes a camera that captures an image of a user, a controller that registers or authenticates face data of the user using the image captured by the camera, and a display that displays a moving image. The controller performs a process of registering or authenticating the face data of the user using a plurality of face images of faces whose orientations are within a first angle and whose orientations are different from each other, the first angle being an angle with reference to a state in which the user faces the camera head-on, and a process of displaying, on the display, a moving image that changes the orientation of a face of an image simulating a human face until the registration or authentication is completed, the second angle being an angle formed by a direction in which the user faces the camera from a seated position and a direction in which the user faces the display from the seated position, the second angle being smaller than the first angle.

Description

Technical Field

[0001] This invention relates to an image processing apparatus and an image processing method. Background Technology

[0002] A technique for recognizing the face of a passenger in a car has been known since the past (Patent Document 1). In the invention described in Patent Document 1, a camera installed in the car takes a picture of the user, and the user's face is recognized using facial feature information from the captured facial image and pre-registered facial feature information.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2008-017227 Summary of the Invention

[0006] The problem the invention aims to solve

[0007] However, when a camera is filming a user, the user may not be facing the correct direction. In such cases, registering or authenticating the user's facial data can take time.

[0008] The present invention was made in view of the above-mentioned problems, and its object is to provide an image processing apparatus and an image processing method that can shorten the time required for the registration or authentication of a user's facial data.

[0009] Solution for solving the problem

[0010] One aspect of the present invention involves an image processing apparatus that uses multiple facial images captured by a camera, each with a different facial orientation, within a first angle relative to a user facing the camera. The apparatus registers or authenticates the user's facial data and displays motion images simulating human faces showing changes in facial orientation on a display screen until registration or authentication is complete. A second angle, smaller than the first angle, is the angle formed by the user's orientation from their seated position towards the camera and the user's orientation from their seated position towards the display screen.

[0011] The effects of the invention

[0012] According to the present invention, the time required for the registration or authentication of a user's facial data is shortened. Attached Figure Description

[0013] Figure 1 This is a structural diagram of the image processing apparatus 1 according to an embodiment of the present invention.

[0014] Figure 2This is a diagram illustrating the positions of camera 11 and monitor 12.

[0015] Figure 3 This is a diagram illustrating the motion of a CG image simulating a human face, showing the changing orientation of the face.

[0016] Figure 4 This is a diagram illustrating an example of an angle related to the orientation of the face.

[0017] Figure 5 It is a diagram illustrating a CG image that simulates a human face.

[0018] Figure 6 This is a diagram illustrating an example of the positional relationship between camera 11 and display 12.

[0019] Figure 7 This is a diagram illustrating an example of the positional relationship between camera 11 and display 12.

[0020] Figure 8 This is a diagram illustrating an example of the positional relationship between camera 11 and display 12.

[0021] Figure 9 This is a diagram illustrating an example of the positional relationship between camera 11 and display 12.

[0022] Figure 10 This is a flowchart illustrating an example of the operation of the image processing apparatus 1 according to an embodiment of the present invention.

[0023] Figure 11 This is a flowchart illustrating an example of the operation of the image processing apparatus 1 according to an embodiment of the present invention.

[0024] Figure 12 This is an illustration of an example of how facial ID is used. Detailed Implementation

[0025] Embodiments of the present invention will now be described with reference to the accompanying drawings. In the drawings, the same reference numerals are used to denote the same parts, and descriptions are omitted.

[0026] Reference Figures 1-2 Let me illustrate the structure of image processing device 1 using an example. For instance... Figure 1 As shown, the image processing device 1 includes an input I / F 10, a camera 11, a controller 20, a display 12, a speaker 13, and a storage device 14.

[0027] The image processing apparatus 1 in this embodiment is an apparatus that performs processing related to facial authentication of a user (mainly a driver). Facial authentication includes two stages. One is the stage of registering facial data for a new user. The other is the stage of effectively utilizing previously registered facial data. The stage of effectively utilizing previously registered facial data refers to, for example, authenticating facial data by matching camera images with previously registered facial data.

[0028] Input I / F 10 is a virtual button displayed on display 12, or a mechanical switch located near display 12. Input I / F 10 is used when a user registers new facial data.

[0029] Camera 11 has imaging elements such as CCD (charge-coupled device) and CMOS (complementary metal oxide semiconductor). Camera 11 is installed inside the vehicle. For example, as Figure 2 As shown, camera 11 is positioned in the center of the steering wheel. Camera 11 primarily captures the face of the user seated in the driver's seat. The images captured by camera 11 are output to controller 20. Figure 2 The reference numeral 30 shown is a CG image (CG: Computer Graphics) simulating a human face. Hereinafter, reference numeral 30 will be referred to as face image 30.

[0030] like Figure 2 As shown, the display 12 is located in the central part of the dashboard. The motion image reproduction unit 23, described later, reproduces motion images on the display 12. In this embodiment, the motion image is an image that changes the orientation of the face in the facial image 30. In other words, the motion image is an image that shows the face image 30 moving its neck. Figure 2 As shown, the camera 11 and the display 12 are arranged along the width of the vehicle.

[0031] The controller 20 is a general-purpose microcomputer equipped with a CPU (Central Processing Unit), memory, and input / output units. A computer program is installed in the microcomputer to enable it to function as the image processing device 1. The microcomputer executes the computer program to function as multiple information processing circuits included in the image processing device 1. Furthermore, an example of implementing multiple information processing circuits included in the image processing device 1 in software is shown here; however, dedicated hardware for performing the information processing described below can also be prepared to construct the information processing circuits. Alternatively, multiple information processing circuits can be constructed using separate hardware. The controller 20 includes an input receiving unit 21, a status determination unit 22, a moving image reproduction unit 23, a feature extraction unit 24, an ID registration unit 25, a feature comparison unit 26, an ID authentication unit 27, and a notification unit 28 as multiple information processing circuits.

[0032] Next, refer to Figure 3 The following describes the process of registering a new user's facial data. As a prerequisite, the user is seated in the driver's seat of the vehicle. When registering facial data, the user presses the registration button (Input I / F 10). This registration button can be a virtual button displayed on the display 12 or a mechanical switch located near the display 12. The user's input is sent to the input receiving unit 21. Upon receiving input from the user, the input receiving unit 21 sends a signal to the status determination unit 22 indicating that the input from the user has been received. Upon receiving the signal from the input receiving unit 21, the status determination unit 22 determines the vehicle's status. Specifically, the status determination unit 22 determines whether the vehicle's power state is "IGN-ON". "IGN-ON" is defined as the state where the power switch inside the vehicle is turned on and all electrical components, including the instrument panel and blower motor, are operating. Additionally, the status determination unit 22 determines whether the shift position is "P". In this embodiment, as a condition for registering facial ID, the vehicle's power state must be "IGN-ON" and the shift position must be "P". Alternatively, the power switch being turned on can be used as a condition instead of "IGN-ON". In other words, the condition for registering facial ID can be that the vehicle's power switch is on and the gear is in "P".

[0033] When the status determination unit 22 determines that the conditions for registering a face ID are met, it sends a signal indicating the determination result to the moving image reproduction unit 23. "The conditions for registering a face ID are met" means that the vehicle's power state is "IGN-ON" and the gear is "P". Upon receiving the signal from the status determination unit 22, the moving image reproduction unit 23 reproduces a moving image on the display 12. (See reference...) Figure 3The motion image reproduced by the motion image reproduction unit 23 will be explained. The motion image is stored in the storage device 14. The storage device 14 is composed of HDD (Hard Disk Drive), SSD (Solid State Drive), etc. The motion image reproduction unit 23 reproduces the motion image stored in the storage device 14. Figure 3 As shown, in the motion image, the orientation of the face in the facial image is changed. In other words, the facial image rotates or swings its neck. Facial image 31 faces to the right. Facial image 32 faces to the left. Facial image 33 faces upward. Facial image 34 faces downward. Furthermore, the motion image does not necessarily need to be stored in storage device 14. For example, the motion image can also be stored on a cloud server (a so-called cloud server). When the motion image is stored on a cloud server, the controller 20 can access the cloud server to reproduce the motion image. Thus, the latest motion image can always be utilized.

[0034] Reference Figure 4 Here's an example illustrating the angle (neck angle) related to the orientation of the face. For example... Figure 4 As shown, the vertical angle of the neck is +20 degrees to -20 degrees. The horizontal angle of the neck is +30 degrees to -30 degrees. When the user is facing the camera 11 (reference numeral 37), the vertical and horizontal directions are defined as 0 degrees. This state serves as the reference for the angles. The numerical values ​​of the angles described above represent changes relative to this reference. Regarding the sign of the angles, the vertical and left directions are positive, and the vertical and right directions are negative. In this embodiment, motion images that change the orientation of a CG image simulating a human face are repeatedly reproduced. The angle of the neck in the CG image simulating a human face is the angle of the face orientation required for capturing an image of facial ID registration or authentication.

[0035] The user rotates their face (neck) in a manner that aligns their own face with the orientation of the face in the moving image. Camera 11 captures multiple facial images with different orientations as the user rotates their face. An example of "multiple facial images with different orientations" is... Figure 4Facial images 35-39 are shown. Facial image 35 is oriented 20 degrees upwards and 30 degrees to the right. Facial image 36 is oriented 20 degrees downwards and 20 degrees to the right. Facial image 37 is oriented frontally (0 degrees in both the vertical and horizontal directions). Facial image 38 is oriented 20 degrees upwards and 20 degrees to the left. Facial image 39 is oriented 20 degrees downwards and 30 degrees to the left. Camera 11 captures five facial images (facial images 35-39) with different facial orientations. Five images are considered one example; multiple images are acceptable, including three or four. Furthermore, as long as the facial orientations are different, a frontal image like facial image 37 is not required.

[0036] Multiple facial images 35-39 captured by camera 11 are sent to feature extraction unit 24. Feature extraction unit 24 uses the multiple facial images 35-39 acquired from camera 11 to extract features of the facial data. As an example, feature extraction unit 24 uses a convolutional neural network (CNN) to extract features of the facial data. However, it is not limited to this, and other machine learning algorithms can also be used. Furthermore, feature extraction methods using machine learning algorithms are well-known techniques, so detailed descriptions are omitted. In this embodiment, the reason for capturing multiple facial images with different facial orientations is to extract facial data features with high accuracy.

[0037] The feature extraction unit 24 sends the features of the extracted facial data to the ID registration unit 25. The ID registration unit 25 uses the features of the facial data obtained from the feature extraction unit 24 to register a facial ID. The registered facial ID is stored in the storage device 14. Thus, the registration of the facial ID is completed. The ID registration unit 25 sends a signal indicating that the registration of the facial ID is complete to the motion picture reproduction unit 23 and the notification unit 28. When the motion picture reproduction unit 23 receives the signal, it stops the motion picture, and the notification unit 28 notifies the user that the registration is complete. As a notification method, the notification unit 28 may display "The registration of the facial ID is complete" on the display 12 or use sound. In other words, the motion picture reproduction unit 23 reproduces the motion picture until the registration of the facial ID is complete. The user rotates their face in the same way as the motion picture until the registration of the facial ID is complete. Thus, multiple facial images with different orientations can be captured in a short time, thereby shortening the time required for facial data registration.

[0038] In a moving image, the orientation of the face in the facial image is changed (by swaying the neck), but there are parts that rotate and parts that do not. For example... Figure 5As shown, in the facial image 31 displayed on the monitor 12, the area 40 above the neck rotates, while the area 41 below the neck does not rotate. Area 40 includes the face and neck itself. Area 41 includes the upper body portion from the chest to the neck.

[0039] like Figure 6 As shown, the position of the monitor 12 is offset from the position of the camera 11 (see reference for positional relationship). Figure 2 ). Figure 6 In the attached figure, reference numeral 50 represents the user, and reference numeral 51 represents the user's nose (the orientation of the user's face). Figure 6 The upper part of the diagram shows the user facing the camera 11. Reference numeral 52 indicates a predetermined angle. The predetermined angle is defined as the angle relative to the user's facing position towards the camera 11, such that the facial image captured by the camera 11 contains a significant amount of facial data. That is, the predetermined angle 52 is the range of angles in which the face of the image is oriented, required for facial ID registration or authentication. In the moving image displayed on the monitor 12, an image simulating a human face is made to change the orientation of the face (swaying the neck) within the same angular range as the predetermined angle 52. The orientation of the user's face is preferably within the range of the predetermined angle 52. Furthermore, the predetermined angle 52 is determined through experiments and simulations. The predetermined angle 52 is composed of first angles 61 and 62 in the left and right directions relative to the user's facing position towards the camera 11. An example of the first angles 61 and 62 is 30 degrees to the left and 30 degrees to the right relative to the user's facing position towards the camera 11 (0 degrees in the vertical and horizontal directions). In this case, the range of the predetermined angle 52 is 60 degrees.

[0040] In this embodiment, the user 50 turns their face while looking at the moving image displayed on the display 12. Therefore, the user 50's gaze will inevitably be directed towards the display 12. Figure 6 (The image below). Figure 6 Reference numeral 63a in the attached figure represents the second angle formed by the direction in which user 50 faces camera 11 from his seated position and the direction in which user 50 faces monitor 12 from his seated position. "The direction in which user 50 faces camera 11 from his seated position" can also be expressed as "the direction of user 50's gaze when user 50 is looking at camera 11". Similarly, "The direction in which user 50 faces monitor 12 from his seated position" can also be expressed as "the direction of user 50's gaze when user 50 is looking at monitor 12". Figure 6Reference numeral 64a indicates the range of angles by which the user 50 rotates their face to the left or right while looking at the moving image. When the user 50 rotates their face in such a way that their own face is oriented in the same direction as the face in the moving image, the size of range 64a is the same as the size of the specified angle 52.

[0041] like Figure 6 As shown in the figure below, when the display 12 is positioned at a second angle 63a greater than the predetermined angle 52, the range 64a does not overlap with the range of the predetermined angle 52. In this case, the orientation of the face in the facial image captured by the camera 11 when the user 50 turns their face left or right while looking at the moving image is not included within the range of the predetermined angle 52. That is, even if the user 50 turns their face in the same way as the moving image while looking at it, it is sometimes difficult to obtain an image of the facial orientation required for facial ID registration or authentication.

[0042] To quickly obtain an image of the face orientation required for facial ID registration or authentication, it is preferable that the range of angles at which the user rotates their face left and right while looking at the moving image overlaps with the range of a predetermined angle 52. Therefore, in this embodiment, the camera 11 and the display 12 are positioned such that the range of angles at which the user 50 rotates their face left and right while looking at the moving image overlaps with the range of the predetermined angle 52. If the camera 11 and the display 12 are positioned close together, the angle of the second angle 63a becomes smaller, so that the range 64a of the angle at which the user 50 rotates their face left and right while looking at the moving image approaches the range of the predetermined angle 52.

[0043] In this embodiment, for example, such as Figure 7 As shown, the camera 11 and the display 12 are positioned at a second angle 63b that is smaller than the predetermined angle 52 formed by the first angles 61 and 62. The range 64b is the range obtained by shifting the range 64a clockwise. A portion of the range 64b overlaps with the range of the predetermined angle 52, thus enabling the capture of the orientation of the user 50's face within the range of the predetermined angle 52.

[0044] like Figure 8 As shown, the camera 11 and the display 12 can also be positioned such that the second angle 63b is smaller than the first angles 61 and 62. Range 64c is the range obtained by further clockwise offset of range 64b. Figure 8 The second angle 63c is less than Figure 7The second angle is 63b. In this case, compared to range 64b, the overlap between range 64c and the range of the predetermined angle 52 becomes larger, enabling the capture of the left and right orientation of the user 50's face within the range of the predetermined angle 52. Thus, it is preferable that the distance between the camera 11 and the display 12 is short. Alternatively, instead of adjusting the positional relationship between the camera 11 and the display 12, the range of the predetermined angle 52 can be adjusted in a manner that makes the range of the predetermined angle 52 larger than the second angle.

[0045] Alternatively, instead of adjusting the positional relationship between the camera 11 and the display 12 or the range of the specified angle 52, the angle of the neck movement in the image simulating a human face can be adjusted so that the range of angles at which the user 50 rotates their face left and right while watching the moving image overlaps with the range of the specified angle 52. For example, as... Figure 9 As shown, the angle of the neck movement in the simulated human face image can be adjusted such that the range 65 of the user 50 turning their face to the right while looking at the moving image is greater than the second angle 63d, based on the user 50 facing the display 12. Specifically, the moving image reproduction unit 23 reproduces a moving image in which the angle of the neck movement to the left (corresponding to the range 65 of the user turning their face to the right) is greater than the second angle 63d. Thus, even if the range of the second angle 63d is greater than the range of the predetermined angle 52, a portion of the range 65 of the user 50 turning their face to the right while looking at the moving image will overlap with the predetermined angle 52, thereby enabling the capture of the left and right orientations of the user 50's face within the range of the predetermined angle 52.

[0046] Next, refer to Figure 10 The flowchart illustrates an example of an operation of the image processing device 1.

[0047] In step S101, the input receiving unit 21 receives input from the user. Upon receiving input from the user, the input receiving unit 21 sends a signal to the status determination unit 22 indicating that input from the user has been received. The process proceeds to step S103, where the status determination unit 22 determines the vehicle's status upon receiving the signal from the input receiving unit 21. If the status determination unit 22 determines that the conditions for registering a face ID are met, it sends a signal indicating the determination result to the moving image reproduction unit 23. The process proceeds to step S105, where the moving image reproduction unit 23 reproduces a moving image (see reference) on the display 12 upon receiving the signal from the status determination unit 22. Figure 3When the notification unit 28 displays a moving image on the display 12, it notifies the user to face the camera 11 and change the orientation of their face in the same up, down, left, and right directions as the moving image, until the registration or authentication of the face ID is completed. As a notification method, there are notifications using text information or notifications using sound. As a notification using text information, the notification unit 28 only needs to display the text "Please face the camera and change the orientation of your face in the same up, down, left, and right directions as the moving image" on the display 12. As a notification using sound, the notification unit 28 only needs to notify the user via the speaker 13 to "Please face the camera and change the orientation of your face in the same up, down, left, and right directions as the moving image".

[0048] In step S107, while the user is looking at the moving image and turning their face, the camera 11 captures multiple facial images with different orientations of the face, each within a first angle 61, 62 based on the user's face being directly facing the camera 11.

[0049] The process proceeds to step S109, where the feature extraction unit 24 determines whether facial data features can be extracted. The determination method is not particularly limited; for example, if the user is detected wearing a mask, the feature extraction unit 24 determines that facial data features cannot be extracted ("No" in step S109). Conversely, if the user is not detected wearing a mask, the feature extraction unit 24 determines that facial data features can be extracted ("Yes" in step S109). If "No" is detected in step S109, the process is interrupted. At this time, the notification unit 28 can either notify "Please remove your mask and take another picture" or notify "Are you wearing anything that is obscuring your face?"

[0050] If "Yes" is received in step S109, the process proceeds to step S111, where the feature extraction unit 24 determines whether multiple images can be acquired within a specified time. If the camera 11 fails to acquire multiple images even after the specified time (if "No" is received in step S111), the process proceeds to step S113, where the notification unit 28 urges the user to retake the shot. While the display 12 shows a moving image, the user is notified to change their facial orientation in the same way as the moving image, until the registration or authentication of the face ID is completed. The notification method is the same as in step S105. After attracting the user's attention through notification, the process returns to step S101. If "Yes" is received in step S111, the process proceeds to step S115, where the feature extraction unit 24 uses the multiple facial images acquired from the camera 11 to extract facial data features. The feature extraction unit 24 sends the extracted facial data features to the ID registration unit 25. The process proceeds to step S117, where the ID registration unit 25 uses the facial data features acquired from the feature extraction unit 24 to register the face ID. ID registration unit 25 sends a signal indicating that facial ID registration is complete to moving image reproduction unit 23 and notification unit 28. The process proceeds to step S119, where moving image reproduction unit 23 stops the moving image upon receiving the signal, and notification unit 28 notifies the user that registration is complete.

[0051] Next, refer to Figure 11The flowchart below illustrates the stages of facial data authentication. In step S201, the input receiving unit 21 determines whether the trigger for authentication is activated. "Trigger activation for authentication" refers to any one of "accessory activation," "user pressing the authentication button (input I / F 10)," or "receiving a signal indicating that the door has been opened and then closed." In this embodiment, "accessory activation" is defined as a state in which the power switch installed in the vehicle is turned on and all electrical components except the instrument panel and blower motor are operating. However, the definition of "accessory activation" is not limited to this. When the user presses the unlock switch of the smart key before entering the vehicle, a predetermined signal is sent to the controller 20. Upon receiving the predetermined signal, the controller 20 changes the power position from power off to a predetermined state. This state can also be defined as "accessory activation." The smart key is used to remotely unlock and lock the door. The smart key is sometimes also referred to as a smart key or a remote key. Alternatively, instead of "receiving a signal indicating that the door has been opened" or "receiving a signal indicating that the door has been closed," the trigger can be "receiving a signal indicating that the door has been opened" or "receiving a signal indicating that the door has been closed." When the trigger for authentication is activated ("Yes" in step S201), the process proceeds to step S203, where the state determination unit 22 determines the vehicle's state. If the state determination unit 22 determines that the conditions for authenticating facial ID are met, it sends a signal indicating the determination result to the moving image reproduction unit 23. "The conditions for authenticating facial ID are met" means that the vehicle's power state is "IGN-ON" and the gear is "P". Alternatively, "The conditions for authenticating facial ID" can also be "accessory activated" and the gear is "P".

[0052] Processing in steps S205 to S215 Figure 10The processes in steps S105 to S115 are the same, so their descriptions are omitted. In step S217, the feature comparison unit 26 compares the features of the facial data extracted in step S215 with the features of the facial data stored in the storage device 14. The feature comparison unit 26 outputs the facial ID with the highest similarity as the recognition result to the ID authentication unit 27. The process proceeds to step S219, where the ID authentication unit 27 authenticates the facial ID. Furthermore, if the features of the facial data extracted in step S215 are inconsistent with the features of the facial data stored in the storage device 14, that is, if there is no facial ID that matches the facial data extracted in step S215, the ID authentication unit 27 may also authenticate the user as a "visitor". In this embodiment, a "visitor" refers to a unique user who has not registered a facial ID. When authenticating a user as a "visitor", the initial value is reflected in various settings. The ID authentication unit 27 sends a signal indicating that the facial ID authentication has been completed to the moving image reproduction unit 23 and the notification unit 28. In step S221, when the motion picture reproduction unit 23 receives the signal, it stops the motion picture and the notification unit 28 notifies the user that authentication has been completed.

[0053] (Effects)

[0054] As explained above, the image processing apparatus 1 according to this embodiment can achieve the following effects.

[0055] Image processing device 1 includes: a camera 11 for capturing images of a user; a controller 20 for registering or authenticating the user's facial data using images captured by the camera 11; and a display for displaying moving images. The controller 20 registers or authenticates the user's facial data using multiple facial images captured by the camera 11, where the orientation of the face is within first angles 61 and 62, and the orientations are different for each face. These first angles 61 and 62 are based on the user facing the camera 11. The controller 20 displays moving images simulating human faces, showing changes in the orientation of the faces, on the display 12 until registration or authentication is complete. The second angle formed by the direction the user faces from their seated position towards the camera 11 and the direction the user faces from their seated position towards the display 12 is smaller than the first angles 61 and 62.

[0056] Even if a user turns their face while looking at display 12, the range of angles of facial rotation will partially overlap with the range of the specified angle 52. This allows for the rapid acquisition of the image required for facial ID registration or authentication. By utilizing moving images to encourage the user to turn their face, multiple facial images with different orientations can be captured in a short time, reducing the time required for facial data registration. Furthermore, in these multiple facial images with different orientations, the facial orientations are all different.

[0057] Camera 11 is positioned in front of the seat inside the vehicle. The direction from which the face is to be turned includes any one of the directions (up, down, left, right), or a combination of these directions (see [reference]). Figure 4 The camera 11 and the display 12 are arranged in a manner that runs along the width of the vehicle or along the height of the vehicle (see reference). Figure 2 When the user is seated in the driver's seat, camera 11 can film the user from the front.

[0058] The controller 20 can also notify the user, while displaying a moving image on the monitor 12, to change the orientation of their face at the same speed as the image simulating a human face. The notification method can be either text-based or audio-based. For text-based notification, the controller 20 simply displays the message "Please change the orientation of your face at the same speed as the moving image" on the monitor 12. For audio-based notification, the controller 20 uses the speaker 13 to verbally notify the user to change the orientation of their face at the same speed as the moving image. This prevents the user from turning their face too quickly or too slowly.

[0059] The controller 20 can also notify the user, while displaying a moving image on the monitor 12, to change the orientation of their face in the same way as the moving image, until registration or authentication is complete. Alternatively, the controller 20 can also notify the user, while displaying a moving image on the monitor 12, to change the orientation of their face towards the camera 11 in the same up, down, left, and right directions as the moving image, until registration or authentication is complete. An example of the notification method is described in S105 above. This prevents the user from stopping turning before registration or authentication is complete. It encourages the user to change the orientation of their face based on a face-to-camera-11 orientation, making it easier for the range of angles the user can turn their face to overlap with the range of the specified angle 52. It enables faster acquisition of the image required for facial ID registration or authentication.

[0060] In this embodiment, the facial images captured by camera 11 can be in any orientation, regardless of the direction. However, for example, it is also possible to add a condition that at least one facial image facing up, down, left, or right is required. If facial images facing up, down, or right are acquired under this condition, but a facial image facing left is not acquired, the controller 20 can notify the user of the missing orientation (left direction). As a notification method, simply changing the color when the CG image faces left is sufficient.

[0061] The camera 11 is positioned in front of the seat inside the vehicle. The camera 11 and the display 12 can also be arranged in a front-to-back configuration along the vehicle's longitudinal direction. When the user looks at the display 12, their face is also facing the camera 11. This facilitates the capture of characteristic facial images.

[0062] The image processing device 1 may also include a sensor for detecting the user's sitting height. The controller 20 may also change the motion image displayed on the display 12 based on the sitting height detected by the sensor. The height relative to the camera 11 is different for users with lower and higher sitting heights. The controller 20 can reproduce a motion image that corresponds to the viewing angle of the user from the camera 11. Multiple motion images are stored in the storage device 14 to correspond to the sitting height.

[0063] The angle of the neck swing in one of the following directions, which simulates the image of a human face, can be greater than the second angle. By making the angle of the neck swing in the direction from the display 12 toward the camera 11 greater than the second angle, even if the range of the second angle is greater than the range of the predetermined angle 52, a portion of the range 65 in which the user turns their face while looking at the moving image will overlap with the predetermined angle 52. Therefore, it is possible to capture the left and right orientation of the user's face within the range of the predetermined angle 52.

[0064] The controller 20 can also determine whether the user's gaze is directed towards the camera 11 or the display 12 based on the image captured by the camera 11. The controller 20 can also change the motion image displayed on the display 12 based on the determination result. Thus, the controller 20 can reproduce a motion image corresponding to the user's gaze direction. Multiple motion images are stored in the storage device 14 to correspond to the user's gaze direction.

[0065] The controller 20 can also, when it determines that the user 50's gaze is directed towards the display 12, make the angle of one of the simulated human face image—the angle of the neck swinging to the right and the angle of the neck swinging to the left—greater than the second angle. Thus, when the user 50's gaze is directed towards the display 12, the controller 20 can reproduce a moving image where the angle of the neck swinging towards the camera 11 is greater than the second angle. A portion of the range 65 within which the user rotates their face while viewing the moving image overlaps with a predetermined angle 52, thus enabling the capture of the left and right orientations of the user 50's face within the predetermined angle 52.

[0066] The controller 20 can also, when it determines that the user's gaze is directed toward the display 12, make the angle opposite to one of the two directions of the simulated human face image—the angle of the neck swinging to the right and the angle of the neck swinging to the left—smaller than the second angle. By making the angle of the neck swinging from the display 12 in the direction opposite to the camera 11 greater than the second angle, the range 66 of the user's face turning while looking at the moving image is reduced, thus suppressing the capture of images with facial orientations other than the predetermined angle 52. This can suppress image processing not required for facial ID registration or authentication, reducing the processing load on the controller 20.

[0067] In a motion image that changes the orientation of a face that simulates a human face, the area 40 above the neck rotates, while the area 41 below the neck does not rotate. This allows the user to easily rotate their face in sync with the motion image.

[0068] The functions described in the above embodiments can be implemented by one or more processing circuits. The processing circuit includes a programmed processing device such as a processing apparatus containing circuitry. The processing circuit also includes an application-specific integrated circuit (ASIC), circuit components, or other devices configured to perform the described functions.

[0069] Embodiments of the invention have been described above, but should not be construed as limiting the invention by the descriptions and drawings that form part of this disclosure. Various alternative embodiments, examples, and techniques will become apparent to those skilled in the art based on this disclosure.

[0070] For example, the image processing device 1 has been described as a device mounted in a vehicle, but it is not limited to this. The image processing device 1 can also be a terminal device (e.g., a smartphone). When the image processing device 1 is a terminal device, the terminal device only needs to reproduce the moving image on the display while capturing the user's image using a camera. The processing involved in the registration or authentication of facial ID is implemented by a controller built into the terminal device.

[0071] Reference Figure 12 This is an example illustrating how facial ID is used. For example... Figure 12 As shown, there is a known technology that associates a vehicle's key ID (the ID sent from key 70) with in-vehicle devices to control these devices and functions based on the key ID. Examples of in-vehicle devices associated with the key ID include the instrument cluster 74, head-up display (HUD) 75, rearview mirror 78, seat 80, and steering wheel 81. Additionally, examples of functions associated with the key ID include air conditioning 76 and driving modes 77. There are situations where multiple people share a single vehicle, such as family sharing. In the case of sharing a vehicle, there is only one standard key. Therefore, it is impossible to configure in-vehicle devices and functions according to the preferences of multiple individuals solely based on the key ID.

[0072] Therefore, facial ID is utilized. By logging into the IVI 72 (in-vehicle infotainment system) using facial ID, a user ID associated with the facial ID can be used. The settings for in-vehicle devices and functions can then be linked to this user ID. By utilizing both the key ID and the user ID (facial ID), even when sharing a vehicle, it is possible to configure in-vehicle devices and functions according to each user's preferences.

[0073] Explanation of reference numerals in the attached figures

[0074] 1: Image processing device; 11: Camera; 12: Display; 13: Speaker; 14: Storage device; 20: Controller; 21: Input receiving unit; 22: Status determination unit; 23: Moving image reproduction unit; 24: Feature extraction unit; 25: ID registration unit; 26: Feature comparison unit; 27: ID authentication unit; 28: Notification unit.

Claims

1. An image processing apparatus, characterized in that, have: The camera, which captures images of the user; A controller that uses images captured by the camera to register or authenticate the user's facial data; and The monitor displays moving images. The controller performs the following processing: The user's facial data is registered or authenticated using multiple facial images captured by the camera, each with a different facial orientation within a first angle. The first angle is based on the user facing the camera directly. The motion image simulating a human face, showing changes in facial orientation, will be displayed on the monitor until the registration or authentication is completed. Based on the images captured by the camera, determine whether the user's gaze is directed towards the camera or towards the display; and When the controller determines that the user's gaze is directed toward the display, it adjusts the angle of the neck movement of the image simulating a human face in such a way that at least a portion of the range of angles at which the user turns their face while looking at the moving image overlaps with the first angle.

2. The image processing apparatus according to claim 1, characterized in that, The camera is positioned in front of the seats inside the vehicle. The direction in which the orientation of the face is to be changed includes any one of the directions of up, down, left, or right, or a combination of the directions of up, down, left, and right. The camera and the display are arranged in a manner that runs along the width of the vehicle or along the height of the vehicle.

3. The image processing apparatus according to claim 1 or 2, characterized in that, When the controller displays the motion image on the display, it notifies the user to change the orientation of the face at the same speed as the speed at which the orientation of the face in the image simulating a human face changes.

4. The image processing apparatus according to claim 1 or 2, characterized in that, When the controller displays the motion image on the display, it notifies the user to change the orientation of their face in the same way as the motion image, until the registration or authentication is completed.

5. The image processing apparatus according to claim 1 or 2, characterized in that, When the controller displays the motion image on the monitor, it notifies the user of the direction they are facing towards the camera and changes the orientation of their face in the same up, down, left, and right directions as the motion image, until the registration or authentication is completed.

6. The image processing apparatus according to claim 1 or 2, characterized in that, The controller informs the user of missing orientations in facial images captured by the camera, where the orientations of the faces differ.

7. The image processing apparatus according to claim 1, characterized in that, The camera is positioned in front of the seats inside the vehicle. The camera and the display are arranged in a manner that runs along the front-to-back direction of the vehicle.

8. The image processing apparatus according to claim 1 or 2, characterized in that, It also includes a sensor that detects the user's sitting height. The controller changes the motion image displayed on the display based on the sitting height detected by the sensor.

9. The image processing apparatus according to claim 1 or 2, characterized in that, When the controller determines that the user's gaze is directed towards the display, it sets one of the angles of the simulated human face image—the angle of the neck swinging to the right and the angle of the simulated human face image—to be greater than a second angle. The second angle is the angle formed by the direction the user faces from their seated position towards the camera and the direction the user faces from their seated position towards the display. The simulated image of a human face shows that the angle of the neck swinging to the right and the angle of the neck swinging to the left are both greater than the second angle.

10. The image processing apparatus according to claim 1 or 2, characterized in that, When the controller determines that the user's gaze is directed towards the display, it sets one of the angles of the simulated human face image—the angle of the neck swinging to the right and the angle of the simulated human face image—to be greater than a second angle. The second angle is the angle formed by the direction the user faces from their seated position towards the camera and the direction the user faces from their seated position towards the display. When the controller determines that the user's gaze is directed toward the display, it makes the angle opposite to one of the two directions of the simulated human face image—the angle of the neck swinging to the right and the angle of the neck swinging to the left—smaller than the second angle.

11. The image processing apparatus according to claim 1 or 2, characterized in that, In a motion image that changes the orientation of the face in the simulated human face image, the area above the neck rotates, while the area below the neck does not rotate.

12. The image processing apparatus according to claim 1 or 2, characterized in that, When the controller determines that the user's gaze is directed towards the display, it sets one of the angles of the simulated human face image—the angle of the neck swinging to the right and the angle of the simulated human face image—to be greater than a second angle. The second angle is the angle formed by the direction the user faces from their seated position towards the camera and the direction the user faces from their seated position towards the display. The second angle is greater than the first angle.

13. The image processing apparatus according to claim 1 or 2, characterized in that, The second angle formed by the direction from which the user faces the camera from their seated position and the direction from which the user faces the display from their seated position is greater than the first angle. When the controller determines that the user's gaze is directed toward the display, it sets the angle at which the user swings their neck toward the camera side from the image of the simulated human face viewed from the display to be greater than the second angle, and sets the angle at which the user swings their neck toward the opposite direction from the camera side from the image of the simulated human face viewed from the display to be less than the second angle.

14. An image processing method comprising an image processing apparatus including a camera, a controller, and a display, wherein the camera captures images of a user, the controller uses the images captured by the camera to register or authenticate the user's facial data, and the display shows moving images, characterized in that... The controller performs the following processing: The user's facial data is registered or authenticated using multiple facial images captured by the camera, each with a different facial orientation within a first angle. The first angle is based on the user's face being directly facing the camera. as well as The motion image simulating a human face, showing changes in facial orientation, will be displayed on the monitor until the registration or authentication is completed. Based on the images captured by the camera, determine whether the user's gaze is directed towards the camera or towards the display; as well as If it is determined that the user's gaze is directed toward the display, the angle of the neck movement in the image simulating a human face is adjusted such that at least a portion of the range of angles at which the user turns their face while looking at the moving image overlaps with the first angle.

15. The image processing method according to claim 14, characterized in that, When the controller determines that the user's gaze is directed towards the display, it sets one of the angles of the simulated human face image—the angle of the neck swinging to the right and the angle of the simulated human face image—to be greater than a second angle. The second angle is the angle formed by the direction the user faces from their seated position towards the camera and the direction the user faces from their seated position towards the display. The second angle is greater than the first angle.

16. The image processing method according to claim 15, characterized in that, The second angle formed by the direction from which the user faces the camera from their seated position and the direction from which the user faces the display from their seated position is greater than the first angle. When the controller determines that the user's gaze is directed toward the display, it sets the angle at which the user swings their neck toward the camera side from the image of the simulated human face viewed from the display to be greater than the second angle, and sets the angle at which the user swings their neck toward the opposite direction from the camera side from the image of the simulated human face viewed from the display to be less than the second angle.

Images