Imaging parameter adjustment method, imaging system, and program

Abstract

The present disclosure provides an imaging parameter adjustment method for adjusting a parameter related to imaging at a next imaging timing by a camera on the basis of face image data captured this time by the camera, the method comprising: an acquisition step for acquiring face image data currently captured from the camera; and an imaging parameter determination step for determining the parameter for the next imaging timing on the basis of the face image data currently captured and acquired in the acquisition step and a predetermined function designed to output the parameter that maximizes face authentication accuracy when the face image data is input thereto.

Description

Photographing parameter adjustment method, photographing system, and program

[0001] The present disclosure relates to an imaging parameter adjustment method, an imaging system, and a program.

[0002] Conventionally, face recognition processing has been performed based on a photographed image of a face (face image data) captured by a camera. In such cases, for example, if the face in the face image data is too dark or too bright depending on the shooting environment or time of day, the accuracy of face recognition decreases. As a countermeasure, for example, there is a method of manually adjusting the camera's shooting parameters (aperture value, gain, exposure time, etc.).

[0003] Patent No. 7246029

[0004] The manual parameter adjustment method described above has the problem of being time-consuming.

[0005] An object of the present disclosure is to provide a photography parameter adjustment method, photography system, and program that are capable of automatically adjusting photography-related parameters when capturing face image data used in face recognition processing.

[0006] The present disclosure provides a shooting parameter adjustment method for adjusting parameters related to the next shooting with a camera based on facial image data currently captured with the camera, and includes an acquisition step for acquiring facial image data currently captured from the camera, and a shooting parameter determination step for determining the parameters for the next shooting based on the facial image data currently captured acquired in the acquisition step and a predetermined function designed to output the parameters that maximize facial recognition accuracy when facial image data is input.

[0007] FIG. 1 is an explanatory diagram showing an overview of an embodiment. FIG. 2 is an overall configuration diagram showing an image capture system of an embodiment. FIG. 3 is a flowchart showing processing during learning in an embodiment and in conventional technology. FIG. 4 is a flowchart showing processing during face authentication by the image capture system of an embodiment. FIG. 5 is a flowchart showing processing during suggested content display by the image capture system of an embodiment. FIG. 6 is a diagram showing a first example screen of an embodiment. FIG. 7 is a diagram showing second to fourth example screens of an embodiment. FIG. 8 is a diagram showing fifth and sixth example screens of an embodiment. FIG. 9 is a diagram showing a seventh example screen of an embodiment.

[0008] Hereinafter, embodiments of the photography parameter adjustment method, photography system, and program of the present disclosure will be described in detail with reference to the accompanying drawings. Note that, hereinafter, the term "parameter" refers to a parameter related to photography when photographing with a camera, and includes, for example, at least one of aperture value, gain, exposure time, gamma value, white balance, and hue.

[0009] To facilitate understanding of the present embodiment, an overview of the embodiment will first be described with reference to Fig. 1. Fig. 1 is an explanatory diagram showing an overview of the embodiment. Consider a case where face recognition processing is performed based on an image captured by a camera. For example, if a face is captured in backlight, the face in the captured image (reference symbol I1) may be too dark, resulting in reduced face recognition accuracy.

[0010] Therefore, for example, a parameter optimization AI (Artificial Intelligence) (reference symbol P) is prepared in advance, which is trained to output parameters that maximize the accuracy of face recognition when a photographed image of a face is input. Then, when a photographed image (reference symbol I1) is input, the parameter optimization AI (reference symbol P) outputs parameters that maximize the accuracy of face recognition.

[0011] In this way, the captured image (reference symbol I2) obtained by photographing using these parameters will not have faces that are too dark or too bright from the perspective of face recognition.

[0012] In this way, even if the first captured image (I1) is unsuitable for facial recognition, the next captured image (I2) is suitable for facial recognition by using parameters obtained by inputting the captured image (I1) into the parameter optimization AI (P). Details of this technology will be described below with reference to Figure 2 and subsequent figures.

[0013] FIG. 2 is an overall configuration diagram showing an image capture system S according to an embodiment. The image capture system S is a system that executes an image capture parameter adjustment method for adjusting parameters for the next image capture with the camera 3 based on face image data captured by the camera 3. The image capture system S includes a PC (Personal Computer) 1, a display device 2, a camera 3, and a control device 4 connected to the PC 1 via a communication network N (such as the Internet) that are installed in a facility F (e.g., an office or pharmacy). Note that the configuration in FIG. 2 is merely an example and is not limiting. For example, the PC 1 may be installed on a cloud or at another location, and the display device 2 and camera 3 may be remotely controlled by the PC 1.

[0014] The PC 1 includes a storage unit 11 , an input unit 12 , a communication unit 13 , and a processing unit 14 .

[0015] The storage unit 11 is configured from, for example, a read-only memory (ROM), a random access memory (RAM), a solid-state drive (SSD), a hard disk drive (HDD), etc., and stores various types of information.

[0016] The input unit 12 is a device for a user to input information, and is, for example, a keyboard, a mouse, a touch panel, or the like.

[0017] The communication unit 13 is a communication interface for communicating with an external device (such as the control device 4).

[0018] The processing unit 14 is configured by, for example, a CPU (Central Processing Unit) and executes programs stored in a ROM to realize various information processes. For example, the processing unit 14 transmits and receives information to and from the control device 4, displays information on the display device 2, controls the camera 3, and acquires captured image data from the camera 3.

[0019] The display device 2 is a device for displaying information, and is, for example, an LCD (Liquid Crystal Display).

[0020] The camera 3 is a photographing device, and may be, for example, any of various image sensors.

[0021] The control device 4 is a computer device and includes a storage unit 41 , an input unit 42 , a display unit 43 , a communication unit 44 , and a processing unit 45 .

[0022] The storage unit 41 is composed of, for example, a ROM, a RAM, an SSD, an HDD, etc., and stores various types of information.

[0023] The input unit 42 is a device for a user to input information, and is, for example, a keyboard, a mouse, a touch panel, or the like.

[0024] The display unit 43 is a device for displaying information, and is, for example, an LCD. The input unit 42 and the display unit 43 may be integrated into a single touch panel.

[0025] The communication unit 44 is a communication interface for communicating with an external device (such as the PC 1).

[0026] The processing unit 45 is configured by, for example, a CPU and executes various information processes. For example, the CPU executes a program stored in the ROM to realize the following functional units: an acquisition unit 451, a face detection unit 452, a face determination unit 453, a shooting parameter determination unit 454, a shooting parameter determination unit 455, a face authentication unit 456, a control unit 457, and a learning processing unit 458.

[0027] The acquisition unit 451 acquires various types of information. For example, the acquisition unit 451 acquires an image (face image data) captured by the camera 3 from the PC 1.

[0028] The face detection unit 452 detects a face portion (hereinafter simply referred to as a "face") from the captured image and outputs a face image. Note that if multiple faces exist in the captured image, the face detection unit 452 may ultimately select and output one face image using a predetermined algorithm, such as selecting the face with the largest face size. Specifically, the face detection unit 452 may acquire (count) the number of pixels of each of the detected multiple faces and select and output the face with the largest number of pixels.

[0029] The face determination unit 453 uses the face image to determine whether or not the face has been photographed correctly. The purpose of providing the face determination unit 453 is to prevent erroneous face detection by the face detection unit 452. In other words, if the face image detected by the face detection unit 452 is not actually a face, using that face image in the processing performed by the shooting parameter determination unit 454 and subsequent processes will prevent appropriate parameter values ​​from being determined.

[0030] The photographing parameter determination unit 454 determines parameters for the next photographing based on the currently photographed face image data (face image output by the face detection unit 452) and a predetermined function. The predetermined function is designed in advance to output parameters that maximize face recognition accuracy when face image data is input. For example, the predetermined function is a learning model that has been machine-learned to output parameters that maximize face recognition accuracy when face image data is input.

[0031] The shooting parameter determination unit 455 determines whether the parameters for the next shooting have changed by a predetermined threshold or more from the parameters for the current shooting. If the parameters have changed by the predetermined threshold or more, the shooting parameter determination unit 455 uses one of the following (1) to (3) instead of the parameters for the next shooting. (1) Parameters for the current shooting. (2) Moving averages (simple moving averages, weighted moving averages, etc.) including the parameters for the next shooting and the parameters for the current shooting. (3) Parameters based on an existing algorithm. An example of an existing algorithm is an existing AE (Auto Exposure) that automatically determines the shutter speed, aperture value, etc.

[0032] The face authentication unit 456 performs face authentication using the face image.

[0033] The control unit 457 executes various controls. For example, if the parameters for the next shooting are outside the preset allowable range of camera parameters, the control unit 457 controls the display device 2 via the PC 1 to display suggested content about shooting according to the deviation. The suggested content includes at least one of moving the shooting location and changing the camera orientation according to the deviation.

[0034] The learning processing unit 458 executes a learning process. Here, FIG. 3 is a flowchart showing the learning process in the embodiment and the prior art. As shown in (a), in the learning process in the embodiment, first, in step S1, the learning processing unit 458 collects a data set. Here, N pairs of face images and parameters are used.

[0035] Next, in step S2, the learning processing unit 458 generates learning data. For example, the learning processing unit 458 calculates the face recognition accuracy for each pair of face image and parameters using a face recognition model (learning model).

[0036] Next, in step S3, the learning processing unit 458 designs a model so that the parameter with the highest face recognition accuracy (for example, parameter 2 of face image 2) becomes the true value.

[0037] Next, in step S4, the learning processing unit 458 performs learning so that the face recognition accuracy approaches the maximum parameter.

[0038] The learning model created in this manner is used by the shooting parameter determination unit 454.

[0039] On the other hand, as shown in (b), in the learning process of the conventional technology, first, in step S11, a data set is collected, where N pairs of face images and parameters are used.

[0040] Next, in step S12, learning data is generated. For example, the average luminance and luminance histogram are calculated for each pair of a face image and parameters.

[0041] Next, in step S13, a model is designed so that the parameter closest to the defined value (for example, parameter 2 of face image 2) becomes the true value.

[0042] Next, in step S14, learning is performed so that the average luminance and the luminance histogram approach predetermined parameters. However, although the learning model created in this way is useful for obtaining captured images that look good to the human eye, it is not necessarily useful for improving the accuracy of face recognition.

[0043] Next, FIG. 4 is a flowchart showing a process for face authentication by the photographing system S of the embodiment.

[0044] In step S21, the acquisition unit 451 of the processing unit 45 of the control device 4 acquires the image captured by the camera 3 from the PC 1.

[0045] Next, in step S22, the face detection unit 452 detects a face portion from the photographed image and outputs a face image.

[0046] Next, in step S23, the face determination unit 453 uses the face image to determine whether the face has been photographed correctly, and if the face has been photographed correctly (OK), proceeds to step S25, and if the face has not been photographed correctly (NG), proceeds to step S24.

[0047] In step S24, the shooting parameter determination unit 454 determines the parameters for the next shooting using the existing AE algorithm, and instructs the PC 1 to execute the next shooting with the camera 3 using those parameters.

[0048] In step S25, the photographing parameter determination unit 454 determines parameters for the next photographing based on the face image data photographed this time and the learning model described above.

[0049] Next, in step S26, the shooting parameter determination unit 455 determines whether the parameters for the next shooting have changed (changed suddenly) by more than a predetermined threshold value from the parameters for the current shooting, and if the answer is Yes, proceed to step S27, and if the answer is No, proceed to step S28.

[0050] In step S27, the shooting parameter determination unit 454 determines one of the above (1) to (3) as another parameter, and instructs the PC 1 to execute the next shooting with the camera 3 using that parameter.

[0051] In step S28, the shooting parameter determination unit 454 instructs the PC 1 to perform the next shooting with the camera 3 using the parameters determined in step S25.

[0052] Next, in step S29, the face authentication unit 456 performs face authentication using the face image.

[0053] Next, in step S30, the control unit 457 determines whether the face authentication result is OK or NG, and if OK, ends the process, and if No, returns to step S21.

[0054] In this way, even if the captured image obtained this time is an image that is unfavorable for facial recognition (for example, an image in which the face is too dark), if the next capture is instructed using the parameters obtained by inputting the captured image into the learning model (step S28), the next captured image obtained will be an image that is favorable for facial recognition (for example, an image in which the face is appropriately bright).

[0055] Next, FIG. 5 is a flowchart showing a process when the image capturing system S of the embodiment displays the proposed content.

[0056] In step S41, the control unit 457 determines whether the parameters used in step S28 of FIG. 4 (hereinafter also referred to as optimal parameters) are outside the preset allowable range of camera parameters, and if the answer is Yes, proceeds to step S42, and if the answer is No, return to step S41.

[0057] In step S42, the control unit 457 controls the display device 2 via the PC 1 to display suggested content related to photography (such as moving the photography location or changing the orientation of the camera 3) that corresponds to the content that is not correct. Below, examples of screens showing suggested content will be described with reference to Figures 6 to 9. After the description of Figure 6, the description of Figure 7 and subsequent figures will omit any overlapping information.

[0058] 6 is a diagram showing a first example of a screen according to an embodiment. Here, the following situation is assumed: <Scene> When the face recognition device (camera 3) is installed <Location> Indoor environment with sunlight <Subject> Installer <Situation> The face image is dark, and the optimal parameters exceed the upper limit of the camera parameters

[0059] A dark face image is displayed in the upper left of the screen. Information about the parameters and progress of face recognition accuracy is displayed in the upper right of the screen. The message at the bottom of the screen reads, "The desired face recognition accuracy cannot be achieved at this location. Please move to a brighter location."

[0060] By looking at such a screen, the user can easily recognize that the shooting location should be changed to a brighter place and can take action.

[0061] 7 shows second to fourth screen examples of the embodiment. Here, the following are assumed: <Location> Pharmacy, store, office, event venue, nursing home, school, etc. <Installer> Installer, part-time female, teacher, etc. <Other> The proposed content may be presented on the installer's terminal (smartphone, etc.) so that the installer can understand the situation when they are not present, such as during operation.

[0062] A dark facial image is shown in the upper left of the screen (a1). It is also assumed that the optimal parameters exceed the upper limit of the camera parameters. Information regarding the progress of the parameters and facial recognition accuracy, as well as the acceptable range (upper and lower limits) of the camera parameters, is displayed in the upper right of the screen. The message "The desired facial recognition accuracy cannot be achieved at this location. Please move to a brighter location" is displayed at the bottom of the screen.

[0063] By looking at such a screen, the user can easily recognize that the shooting location should be changed to a brighter place and can take action.

[0064] Next, a bright face image is shown in the upper left corner of the screen (a2). Also, assume that the optimal parameters are below the lower limit of the camera parameters. The message "The desired face recognition accuracy cannot be achieved at this position. Please move to a darker location." is displayed at the bottom of the screen.

[0065] By looking at such a screen, the user can easily recognize that the shooting location should be changed to a darker place and take appropriate action.

[0066] Next, in the upper left corner of the screen (b), a face image with abnormal color is displayed. Furthermore, the optimal parameters related to color, such as hue, are outside the allowable range of the camera parameters. The message "The desired face recognition accuracy will not be achieved from this position. Please move to a location where the current lighting is not shining on you" is displayed at the bottom of the screen.

[0067] By looking at such a screen, the user can easily recognize that the shooting location should be changed to a location that is not illuminated by the current light, and can take appropriate action.

[0068] 8A and 8B are diagrams showing fifth and sixth screen examples according to the embodiment. Here, the following locations are assumed: <Locations, etc.> Pharmacies, stores, offices, autonomous mobile robots, buses, amusement parks, stations, smartphones, etc.

[0069] The screen in (a) shows a dark facial image. It is also assumed that the optimal parameters exceed the upper limit of the camera parameters. The top right corner of the screen displays information about the parameters and the progress of facial recognition accuracy. The bottom of the screen displays the message, "Please point the camera in a different direction."

[0070] By looking at such a screen, the user can easily recognize that the camera 3 should be rotated to a different orientation and can take appropriate action.

[0071] The image in (b) shows a dark face image. The optimal parameters exceed the upper limit of the camera parameters. The message "Please move a little closer" is displayed at the bottom of the screen.

[0072] By looking at such a screen, the user can easily recognize that the person to be photographed should approach the camera 3 and take appropriate action.

[0073] 9 is a diagram showing a seventh example screen of the embodiment. Here, the following is assumed: <Time> During operation <Location, etc.> Pharmacy, store, office, autonomous mobile robot, bus, amusement park, station, smartphone, etc. <Target> Operator monitoring remotely <Situation> When facial recognition accuracy is low for a certain period of time when data is collected for one day. With the current installation location, it is difficult to achieve the desired facial recognition accuracy throughout the day just by adjusting the camera parameters.

[0074] The upper part of the screen displays information about facial images for each time period, the time, parameters, and facial recognition accuracy. Also, assume that the optimal parameters are below the lower limit of the camera parameters only at 1:00 PM. The lower part of the screen displays a message saying, "Please change the shooting position or camera direction for a few hours around 1:00 PM."

[0075] By looking at this screen, the user can easily recognize that the shooting position or the direction of the camera 3 should be changed for a few hours around 1:00 PM, and can take appropriate action.

[0076] In addition, the following Tables 1 to 3 list examples of the proposed recipient, environment, installation location, (optimal) parameters output from the AI ​​(learning model), and proposal content.

[0077]

[0078] In this way, the photographing system S of this embodiment can automatically determine (adjust) parameters for the next photographing based on the facial image data photographed this time and the above-mentioned predetermined function, thereby eliminating the need to manually adjust parameters and improving convenience.

[0079] In this case, by using, for example, a learning model that has been machine-learned as described above as the predetermined function, it is possible to obtain parameters with higher accuracy.

[0080] Therefore, even if the shooting environment is poor, such as backlit, higher face recognition accuracy can be achieved compared to conventional techniques.

[0081] Furthermore, if the parameters for the next photograph have changed suddenly from the parameters for the current photograph, it is highly likely that it is not a good idea to use those parameters as they are. Therefore, by using the other parameters ((1) to (3)) described above instead of the parameters for the next photograph, it is possible to avoid or reduce the deterioration of the facial image data for the next photograph.

[0082] Furthermore, if the parameters for the next photograph are outside the preset allowable range of camera parameters, suggestions according to the reason for the deviation (such as moving the photographing location or changing the orientation of the camera 3) are displayed. This allows the user to easily improve the situation by looking at the display.

[0083] In particular, since it is expected that face recognition technology will become widespread in the future, this is extremely effective when amateurs operate face recognition devices.

[0084] Note that the existing AE is intended to obtain a photographed image that looks good to the human eye, and is not intended to obtain a photographed image that is good from the viewpoint of face authentication.

[0085] According to at least one of the embodiments described above, it is possible to automatically adjust parameters at the time of capturing face image data used in face authentication processing.

[0086] Although the embodiments of the present disclosure have been described above, the above-described embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These novel embodiments and modifications thereof are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as defined in the claims.

[0087] For example, the predetermined function is not limited to the machine-learned learning model described above, but may also be other functions, such as an input / output function designed to output parameters that maximize facial recognition accuracy when facial image data is input based on statistical values.

[0088] Furthermore, information for allowing the user to recognize how much margin there is in the current installation location regarding the brightness of the environment (face image) may be displayed on the display device 2. Specifically, for example, if the allowable range of a parameter related to brightness is 0 to 100 and the current optimal parameter is 50, by displaying this information on the display device 2, the user who sees it can recognize that there is margin in both the dark and bright directions.

[0089] If the user is short of space, they can easily change the situation to one where they have more space by moving the shooting location or changing the camera's orientation, and then viewing the same display again. This method also makes it possible to flexibly respond to environmental changes over time, for example. In other words, it becomes easy to shoot in a more suitable environment.

[0090] In addition to the brightness of the environment (face image), parameter information can also be displayed for other elements such as the color of the face image to allow the user to recognize how much margin there is currently.

[0091] (Additional Notes) The above embodiments disclose the following technologies. (1) A shooting parameter adjustment method for adjusting parameters related to shooting with a camera for the next shooting based on face image data currently captured with the camera, the shooting parameter adjustment method including: an acquisition step for acquiring face image data currently captured from the camera; and a shooting parameter determination step for determining the parameters for the next shooting based on the face image data currently captured acquired in the acquisition step and a predetermined function designed to output the parameters that maximize face recognition accuracy when face image data is input. (2) The shooting parameter adjustment method described in (1) above, wherein the predetermined function is a learning model trained by machine learning to output the parameters that maximize face recognition accuracy when face image data is input. (3) The shooting parameter adjustment method described in (1) above, wherein, if the determined parameters for the next shooting have changed from the parameters for the current shooting by a predetermined threshold or more, the shooting parameter determination step uses the parameters for the current shooting, a moving average including the parameters for the next shooting and the parameters for the current shooting, or parameters based on an existing algorithm, instead of the parameters for the next shooting. (4) The photographing parameter adjustment method according to (1), further comprising: a display control step of controlling a display unit to display, when the determined parameters for the next photographing are outside a preset allowable range of camera parameters, suggestions regarding the photographing according to the deviation. (5) The photographing parameter adjustment method according to (4), wherein the suggestions include at least one of moving the photographing location and changing the orientation of the camera according to the deviation. (6) The photographing parameter adjustment method according to (1), wherein the parameters include at least one of aperture value, gain, exposure time, gamma value, white balance, and hue.(7) A photography system that adjusts parameters for photography with a camera for the next photography based on face image data currently photographed with the camera, the photography system including: an acquisition unit that acquires face image data currently photographed from the camera; and a photography parameter determination unit that determines the parameters for the next photography based on the face image data currently photographed acquired by the acquisition unit and a predetermined function designed to output the parameters that maximize face recognition accuracy when face image data is input. (8) A program that causes a computer to execute the photography parameter adjustment method described in (1) above. (9) A program executed by a computer, a recording medium (computer program product) on which the program described in (8) above is recorded.

[0092] 1...PC, 2...display device, 3...camera, 4...control device, 11...storage unit, 12...input unit, 13...communication unit, 14...processing unit, 41...storage unit, 42...input unit, 43...display unit, 44...communication unit, 45...processing unit, 451...acquisition unit, 452...face detection unit, 453...face determination unit, 454...shooting parameter determination unit, 455...shooting parameter determination unit, 456...face authentication unit, 457...control unit, F...facility, S...shooting system

Claims

1. A method for adjusting shooting parameters for shooting at the next shooting with a camera based on face image data captured by the camera this time, the method comprising: an acquisition step of acquiring the face image data captured by the camera this time; and a shooting parameter determination step of determining the parameters for the next shooting based on the face image data captured this time in the acquisition step and a predetermined function designed to output the parameters that maximize the face recognition accuracy when the face image data is input.

2. The method for adjusting shooting parameters according to claim 1, wherein the predetermined function is a learning model that is machine-learned to output the parameters that maximize the face recognition accuracy when the face image data is input.

3. The method for adjusting shooting parameters according to claim 1, wherein in the shooting parameter determination step, when the determined parameters for the next shooting have changed by a predetermined threshold or more from the parameters for this shooting, instead of the parameters for the next shooting, either the parameters for this shooting, a moving average including the parameters for the next shooting and the parameters for this shooting, or parameters by an existing algorithm are used.

4. The method for adjusting shooting parameters according to claim 1, further comprising a display control step of performing control to display, on a display unit, proposed content related to shooting according to the content that is out of range, when the determined parameters for the next shooting are out of the allowable range of the preset camera parameters.

5. The method for adjusting shooting parameters according to claim 4, wherein the proposed content includes at least either movement of the shooting location or the orientation of the camera according to the content that is out of range.

6. The method for adjusting shooting parameters according to claim 1, wherein the parameters include at least any one of an aperture value, a gain, an exposure time, a gamma value, a white balance, and a hue.

7. A photographing system that adjusts parameters related to photographing at the time of the next photographing with the camera based on the face image data photographed this time with the camera, the system comprising: an acquisition unit that acquires the face image data photographed this time from the camera; and a photographing parameter determination unit that determines the parameters at the time of the next photographing based on the face image data photographed this time acquired by the acquisition unit and a predetermined function designed to output the parameters that maximize the face authentication accuracy when face image data is input.

8. A program for causing a computer to execute the photographing parameter adjustment method according to claim 1.

Images