Image processing device and image processing method

Abstract

To provide a technology which can correct a captured image of a caller, when personal appearance of the caller of a video call is not ready.SOLUTION: An image processing device comprises: an extraction unit which extracts feature quantities of a user's captured image and a user's reference image; a determination unit which determines whether to validate or invalidate correction processing with respect to the user's captured image on the basis of a result of collation between the feature quantity of the user's captured image and the feature quantity of the user's reference image; a correction unit which generates a correction image of the user's captured image when it is determined that the correction processing is validated; and an output unit which outputs the correction image when it is determined that the correction processing is validated, and outputs the user's captured image which has not been corrected when it is determined that the correction processing is invalidated.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an image processing device and an image processing method. [Background technology]

[0002] In recent years, the number of companies that have introduced remote work has increased, and opportunities for online meetings or video calls have increased. During online meetings or video calls, it is desirable to enable video in order to check the facial expression of the person on the other end of the line. When enabling video, the person on the other end of the line must take the time to dress appropriately for the call, such as by putting on makeup if the person is female, or shaving if the person is male. In response to this, a technology has been proposed that corrects video for video calls to a state suitable for the call. For example, Patent Document 1 discloses a technology that, in data communication involving image data, determines the degree of fatigue of a user based on the number of blinks or the degree of bloodshot eyes, and synthesizes the image with a previously captured image if the degree of fatigue is above a certain level. Patent Document 2 discloses a technology that selects or processes video to be used for video communication from "video that is acceptable to show to the other party" that the user has previously determined, and outputs the video. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2001-016564 A [Patent Document 2] JP 2012-142925 A Summary of the Invention [Problem to be solved by the invention]

[0004] When video calling is performed by simply determining the degree of fatigue of the caller, unnecessary corrections may be made to the image even if the caller is well-groomed. In addition, when the image for communication is always selected or processed from the image that is acceptable for the other party to see, the image for communication may be corrected regardless of whether the person is well-groomed or not. Furthermore, it may be time-consuming for the user to decide in advance which image is acceptable for the other party to see, and usability may not be optimal.

[0005] An object of one aspect of the present invention is to provide a technique for correcting a captured image of a person in a video call when the person is not well-groomed. [Means for solving the problem]

[0006] In order to achieve the above object, the present invention employs the following configuration.

[0007] A first aspect of the present disclosure provides an image processing device comprising: an extraction unit that extracts features of a user's captured image and a user's reference image; a determination unit that determines whether to enable or disable a correction process for the user's captured image based on a result of comparing the features of the user's captured image with the features of the user's reference image; a correction unit that generates a corrected image of the user's captured image when it is determined that the correction process is enabled; and an output unit that outputs the corrected image when it is determined that the correction process is enabled, and outputs an uncorrected user's captured image when it is determined that the correction process is disabled.

[0008] The "reference image" is, for example, an image of a user in a neat and tidy state. The image processing device can determine whether or not to correct the captured image based on the result of comparing the captured image of the user with the reference image. Since the image processing device automatically corrects the captured image according to the determination of whether or not to correct it, the user can easily instruct the image correction when he or she is concerned about the state of his or her appearance. You can make video calls without having to do any special operations to make a call.

[0009] The feature amount may be a feature amount of a part of the user's face that is likely to change. By extracting the feature amount of the part of the user's face that is likely to change from the captured image and the reference image, the image processing device can accurately determine whether the grooming of the captured image is good or not, i.e., whether to disable the correction process, when comparing between the same user.

[0010] The feature amount may include at least one of Haar-like feature amount, color histogram, and color moment. The feature amount may be calculated by an algorithm using a learning model that has been trained on images of well-groomed and ungroomed subjects. The image processing device can use various feature amounts or a combination of these feature amounts to determine whether or not to enable the correction process.

[0011] The determination unit may calculate a degree of coincidence between the feature amount of the captured image of the user and the feature amount of the reference image of the user, and determine that the correction process is enabled when the degree of coincidence is less than a predetermined threshold, and that the correction process is disabled when the degree of coincidence is equal to or greater than the predetermined threshold. The image processing device can determine whether or not to correct the captured image based on the degree of coincidence between the captured image of the user and the reference image, and can automatically correct the captured image. This allows the user to participate in a video call without worrying about the state of their appearance or performing a special operation to instruct image correction.

[0012] The correction unit may change the amount of correction for the captured image of the user according to the degree of matching. By changing the amount of correction according to the degree of matching between the captured image and the registered image, when the user is relatively well-groomed, unnecessary correction can be suppressed.

[0013] The correction unit may generate the corrected image based on the captured image of the user and the reference image of the user. For example, the correction unit may generate the corrected image by a GAN (Generative Adversarial Network) that has learned the captured image of the user and the reference image of the user. The correction unit may also generate the corrected image by cutting out a part or the entire face of the reference image of the user and replacing the corresponding part of the captured image of the user with the cut-out image. By generating the corrected image based on the captured image of the user and the reference image, the image processing device can generate a corrected image that is closer to the reference image.

[0014] The correction unit may generate the corrected image by performing a filter process for removing noise or adjusting saturation on the captured image of the user based on the facial feature information. Since the image processing device can generate the corrected image based on various facial feature information without using a reference image, the user can save the trouble of preparing a reference image.

[0015] The determination unit may determine whether to enable or disable the correction process for each predetermined number of frames of the captured image of the user. Since the image processing device can correct the captured image even if makeup such as lipstick comes off during a video call, the user can continue the call without worrying about their appearance being ruined.

[0016] The image processing device may further include an imaging section for capturing an image of the user. The image processing device may be configured integrally with the imaging section, thereby allowing for a simple configuration.

[0017] A second aspect of the present invention includes an extraction step of extracting feature amounts of a captured image of a user and a reference image of the user, and a result of comparing the feature amounts of the captured image of the user with the feature amounts of the reference image of the user. The present invention provides a human body detection method comprising: a determination step of determining whether to enable or disable a correction process for a user's captured image based on the result of the determination; a correction step of generating a corrected image of the user's captured image when it is determined that the correction process is enabled; and an output step of outputting the corrected image when it is determined that the correction process is enabled, and outputting an uncorrected captured image of the user when it is determined that the correction process is disabled. Effect of the Invention

[0018] According to the present invention, when a caller in a video call is not well groomed, a captured image of the caller can be corrected. [Brief description of the drawings]

[0019] [Figure 1] FIG. 1 is a diagram illustrating an application example of an image processing device according to an embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of a functional configuration of the image processing device. [Diagram 3] FIG. 3 is a flowchart illustrating an example of the image correction process. [Figure 4] FIG. 4 is a diagram showing a first example of extracting facial feature amounts. [Diagram 5] FIG. 5 is a diagram showing a second example of extracting facial feature amounts. [Figure 6] FIG. 6 is a diagram illustrating an example of a correction process for a captured image. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an embodiment according to one aspect of the present invention will be described with reference to the drawings.

[0021] <Application Examples> 1 is a diagram for explaining an application example of an image processing device according to an embodiment. The image processing device acquires a camera image (captured image) input from a camera and a registered image (reference image) registered in advance in a DB (database), and extracts features from each image. The registered image is, for example, an image of a well-groomed user, and serves as a reference image for determining whether or not to correct the captured image.

[0022] The image processing device compares the feature amounts of the camera image and the registered image and evaluates the degree of match. If the degree of match is equal to or greater than a predetermined threshold, the image processing device determines that the user is well-groomed and disables correction processing for the camera image. If the degree of match is less than the predetermined threshold, the image processing device determines that the user is not well-groomed and enables correction processing. In this way, the image processing device determines whether to enable or disable correction processing for the camera image based on the result of comparing the feature amounts of the camera image and the registered image.

[0023] When the correction process is enabled, the image processing device corrects the user's camera image to generate a display image (corrected image), and transmits the corrected image to another computer used by the call partner for display. The corrected image may also be displayed on the display of the image processing device. The display image may be generated based on a registered image of the user in a well-groomed state. This allows the user to participate in a video call without worrying about the state of their appearance, since the corrected image is displayed without the user having to perform any special operations when their appearance is not good.

[0024] <Embodiment> (Device configuration) An example of the functional configuration of the image processing device 1 will be described with reference to Fig. 2. Fig. 2 is a diagram illustrating an example of the functional configuration of the image processing device 1. The image processing device 1 includes an imaging unit 10, a registered image database 11, a feature extraction unit 12, a correction determination unit 13, a correction processing unit 14, and an output unit 15.

[0025] The image capturing unit 10 captures an image of a user who is making a call. The registered image database 11 stores a registered image (reference image) that serves as a reference for determining whether or not to correct a captured image. The registered image database 11 may store a plurality of registered images for each user. The registered image may be, for example, an image captured when the user makes a call for the first time using the image processing device 1. The registered image may also be an image selected by the user while checking an image displayed on the display device. Note that the registered image database 11 is not limited to registered images of users, and may also store information on feature amounts of a well-groomed user.

[0026] The feature extraction unit 12 (extraction unit) extracts features from the captured image of the user and the registered image. The features are, for example, Haar-like features, color histograms, and color moments. The feature extraction unit 12 may also use a combination of these features as the feature for determining whether to enable or disable the correction process.

[0027] The correction determination unit 13 (determination unit) determines whether or not to enable correction processing for the captured image based on the feature amount extracted by the feature extraction unit 12. Specifically, the correction determination unit 13 compares the feature amount of the captured image of the user with the feature amount of the registered image of the user, and calculates the degree of match.

[0028] For example, the correction determination unit 13 can calculate the Haar-like feature amount around the boundary between the eyebrows and the skin on the forehead for each of the captured image and the registered image, and calculate the degree of agreement as "(feature amount of registered image - difference between each feature amount) / feature amount of registered image." The correction determination unit 13 may calculate Haar-like feature amounts for multiple sites other than the boundary between the eyebrows and the skin on the forehead, and calculate the average value of these as the degree of agreement.

[0029] The correction determination unit 13 can determine that the correction process for the user's captured image is disabled if the degree of similarity is greater than or equal to a predetermined threshold (e.g., 80%), and can determine that the correction process is enabled if the degree of similarity is less than the predetermined threshold.

[0030] The correction processing unit 14 (correction unit) corrects the captured image of the user when the correction determination unit 13 determines that the correction process is enabled. The correction processing unit 14 can generate the corrected image by, for example, a GAN (Generative Adversarial Network) that has been trained on images of the user in which the user is well-groomed and images in which the user is not well-groomed. The correction processing unit 14 can also generate the corrected image by cutting out a part of the registered image or the entire face, and replacing the corresponding part of the captured image of the user with the cut-out image and synthesizing the image. Furthermore, the correction processing unit 14 can also generate the corrected image by performing a filter process for removing noise or a saturation adjustment on the captured image of the user.

[0031] The output unit 15 outputs an image of the user who is the caller. The image output by the output unit 15 is transmitted to another computer. The image output by the output unit 15 may be displayed on the display device of the image processing device 1. When the correction determination unit 13 determines that the correction process for the captured image of the user is enabled, the output unit 15 outputs the captured image of the user after correction (corrected image), and when the correction determination unit 13 determines that the correction process is disabled, the output unit 15 outputs the captured image of the user without correction. When the correction determination unit 13 determines that the correction process is enabled, the output unit 15 transmits the captured image of the user after correction to an electronic device such as a tablet terminal used by the call partner for the call to display it.

[0032] The image processing device 1 of this embodiment may be a general-purpose computer such as a personal computer, a server computer, a tablet terminal, or a smartphone, or may be an embedded computer such as an on-board computer. The image processing device 1 has a CPU (processor), a RAM (memory), non-volatile storage (HDD, SSD, etc.), an input device (touch panel, etc.), and a communication device (wired or wireless LAN module, etc.). The image processing device 1 also has a lens and an imaging element (image sensor such as CCD or CMOS). The image capturing device includes a display device (such as a liquid crystal monitor), and other hardware resources.

[0033] The processor deploys a program stored in the storage in the RAM and executes it to realize the functions of each functional unit described in Fig. 2. Note that the method of realizing the image processing device 1 is not limited to this. For example, the image processing device 1 may be realized by distributed computing using a plurality of computer devices, and some of the functional units may be realized by a cloud server. In addition, some of the functional units of the image processing device 1 may be realized by a dedicated hardware device such as an FPGA or an ASIC.

[0034] (Image correction processing) The overall flow of the image correction process will be described with reference to Fig. 3. Fig. 3 is a flowchart illustrating the image correction process. The image correction process is started, for example, when a user starts an application used for calling on the image processing device 1. Note that the image correction process shown in Fig. 3 is a process executed for each frame of a camera image (captured image).

[0035] In S101, the feature extraction unit 12 acquires a camera image of the user captured by the imaging unit 10. The feature extraction unit 12 proceeds to processing in S102 for each frame of the camera image. In the following explanation of each process, the camera image will be explained as an image of one frame of data received from the imaging unit 10.

[0036] In S102, the feature extraction unit 12 judges whether the camera image is an image at the timing for evaluating the feature amount. The timing for evaluating the feature amount may be, for example, the timing when the feature extraction unit 12 first recognizes a human face in each frame. Note that the timing for evaluating the feature amount is not limited to the timing when the human face is first recognized, and may be a predetermined interval such as every predetermined number of frames (e.g., 30 frames) or every predetermined time (e.g., 5 minutes). If the camera image is an image at the timing for evaluating the feature amount (S102: Yes), the process proceeds to S103. If the camera image is not an image at the timing for evaluating the feature amount (S102: No), the process proceeds to S108.

[0037] In S103, the feature extraction unit 12 extracts the feature amount of the camera image. Also, the feature extraction unit 12 acquires a registered image of the user from the registered image database 11 and extracts the feature amount of the registered image. Here, the extraction of the feature amount will be described with reference to Figs. 4 and 5.

[0038] First example of feature extraction Fig. 4 is a diagram showing a first example of extracting facial features. The example in Fig. 4 is an example in which feature points that are unique to the user and do not change easily with time are selected, and Haar-like features are extracted around the feature points. The circles shown in the camera image in Fig. 4(A) and the registered image in Fig. 4(B) indicate feature points that are unique to the user and do not change easily with time.

[0039] FIG. 4(A) shows the results of Haar-like feature calculations in the area around the user's left eye, along with a camera image. FIG. 4(A) shows an example of Haar-like feature calculations using a rectangular pattern for identifying edges (a filter that divides the area vertically or horizontally into two parts, one side being white and the other being black). The calculated Haar-like feature is represented by a rectangle with a shade that corresponds to the calculated value. Haar-like feature calculations for the outer corner 401a of the user's left eye The feature amount is indicated in gray 401b.

[0040] FIG. 4B shows the result of calculating the Haar-like feature in the area around the left eye of the user together with the registered image. The Haar-like feature is calculated in the same manner as in FIG. 4A. The calculated Haar-like feature is represented by a rectangle with a shade corresponding to the calculated value. The Haar-like feature at the outer corner 402a of the left eye of the user is shown in FIG. 4(A) is shown as a darker gray 402b than 401b in FIG. 4(A). In this way, in the registered image, the Haar-like feature amount is larger than that in the camera image due to the difference in shading caused by makeup such as eyeliner, and therefore 402b is darker than 401b.

[0041] In the first example shown in Fig. 4, the degree of match between the camera image and the registered image can be calculated based on the degree of match of the Haar-like feature at each of the feature points indicated by circles. The degree of match between the camera image and the registered image may be the sum or average value of the degrees of match at each feature point. The degree of match at each feature point can be calculated as, for example, (X-|XY|) / X (where X is the Haar-like feature at the feature point of the registered image, and Y is the Haar-like feature at the corresponding feature point of the camera image).

[0042] The method of matching features using feature points that are unique to a user and do not easily change over time can also be realized using a face recognition algorithm. In the determination processes of S104 and S105, the correction determination unit 13 calculates a score indicating whether or not the person is the person himself / herself using a face recognition algorithm, and can use the score as the degree of match between the camera image and the registered image.

[0043] Second example of feature extraction Fig. 5 is a diagram showing a second example of extracting facial features. The example in Fig. 5 is an example in which feature points that are easy to capture changes between the same user are selected, and Haar-like features are extracted around the feature points. The circles shown in the camera image in Fig. 5(A) and the registered image in Fig. 5(B) indicate feature points that are easy to capture changes between the same user.

[0044] FIG. 5(A) shows the results of Haar-like feature calculations in the area around the user's left cheek, along with a camera image. FIG. 5(A) shows an example of Haar-like feature calculations using a rectangular pattern for identifying edges (a filter dividing the area vertically or horizontally into two parts, one side being white and the other being black). The calculated Haar-like feature is represented by a rectangle with a shade corresponding to the calculated value. Haar-like feature calculations in the center 501a of the user's left cheek The feature amount is shown in white 501b.

[0045] FIG. 5B shows the results of calculating Haar-like features in the area around the user's left cheek together with the registered image. The Haar-like features are calculated in the same manner as in FIG. 5A. The calculated Haar-like features are represented by rectangles with different shadings according to the calculated values. The Haar-like features in the center 502a of the user's left cheek are shown as a gradient. In this way, in the registered image, the Haar-like feature amount is larger than in the camera image due to the difference in shading caused by makeup such as blush, and so 502b is darker than 401b.

[0046] In the second example shown in Fig. 5, the degree of match between the camera image and the registered image can be calculated, for example, based on the degree of match of the Haar-like features at each of the feature points indicated by circles, in the same manner as in the first example of Fig. 4. Note that the feature extracted from a part whose color changes before and after grooming, such as a woman's cheek or a man's beard, is not limited to the Haar-like feature, and may be a feature such as a color histogram or a color moment, or a combination of these.

[0047] In the second example, since feature amounts at feature points that are easy to detect changes between the same user are compared, the correction determination unit 13 can accurately determine whether or not the grooming is neat. Feature points that are easy to detect changes between the same user may be selected from areas such as eyebrows, corners of the eyes, cheeks, and mouth for women, and areas such as around the mouth where beard grows for men. By changing the areas of the feature points to be compared depending on gender, it becomes possible to more accurately determine whether or not the grooming is neat.

[0048] Third example of feature extraction As a third example of extracting facial features, an example using a learning model generated by deep learning such as CNN (Convolution Neural Network) will be described. The learning model used in the third example is, for example, a model generated by having CNN learn images of well-groomed people and images of poorly-groomed people. The images to be learned by CNN may be images of people other than the user himself / herself, and may include images of the user himself / herself. The feature extraction unit 12 uses the generated learning model to extract the registered image (image of well-groomed people) and the score of the camera image as features by the algorithm of CNN. The correction determination unit 13 can determine whether to enable or disable the correction process for the camera image based on the degree of agreement between the score of the registered image and the score of the camera image in the determination process of S104 and S105.

[0049] 3, in S104, the correction determination unit 13 compares the feature amount of the camera image extracted in S103 with the feature amount of the registered image. Specifically, the correction determination unit 13 calculates the degree of match between the camera image and the registered image based on the feature amount of the camera image and the feature amount of the registered image. As described in the first to third examples above, the degree of match between the camera image and the registered image is calculated by a method according to the type of feature amount extracted from each image.

[0050] In S105, the correction determination unit 13 determines whether the degree of match between the camera image and the registered image calculated in S104 is equal to or greater than a predetermined threshold. If the degree of match is equal to or greater than the predetermined threshold (S105: Yes), the process proceeds to S107. If the degree of match is less than the predetermined threshold (S105: No), the process proceeds to S106.

[0051] In S106, since the degree of match between the camera image and the registered image is less than a predetermined threshold, the correction determination unit 13 sets the correction flag to ON and enables the correction process of the camera image. In S107, since the degree of match between the camera image and the registered image is equal to or greater than a predetermined threshold, the correction determination unit 13 sets the correction flag to OFF and disables the correction process of the camera image.

[0052] The correction flag is a flag for determining whether or not the correction processing unit 14 executes a correction process on the captured image of the user. When the correction flag is set to ON in S106, the correction process is valid until the correction flag is set to OFF in S107 for a subsequent frame. In the case where the feature amount is evaluated only at the beginning of a call, when the correction flag is initially set to ON, the correction processing unit 14 continues the correction process of the camera image until the call ends.

[0053] On the other hand, if the correction flag is set to OFF in S107, the correction process is disabled until the correction flag is set to ON for a subsequent frame in S106. In the case where the feature amount is evaluated only at the beginning of a call, if the correction flag is initially set to OFF, the camera image is displayed on the display device without being corrected.

[0054] Furthermore, if the timing for evaluating the feature amount is every predetermined number of frames, when the correction flag is set to ON in S106, the correction process of the camera image is enabled until the next timing for evaluating the feature amount. Conversely, when the correction flag is set to OFF in S107, the correction process of the camera image is disabled until the next timing for evaluating the feature amount, and the camera image is displayed on the display device without correction.

[0055] In S108, the correction processing unit 14 determines whether the correction flag is on (ON). If the correction flag is on (S108: Yes), the process proceeds to S109. If the correction flag is off (S108: No), the correction processing unit 14 outputs the camera image to the output unit 15 without correcting it, and the process proceeds to S110.

[0056] In S109, the correction processing unit 14 corrects the camera image of the user to generate a corrected image. Here, three methods by which the correction processing unit 14 generates a corrected image will be described. The first and second methods are methods for generating a corrected image based on a camera image and a registered image. The third method is a method for generating a corrected image by correcting the camera image based on facial feature information prepared in advance.

[0057] The first method is a method of generating a corrected image by a GAN that has been trained with images of a user in which the user is well-groomed and images in which the user is not well-groomed. For example, the GAN can learn a camera image in which the correction flag is off as data of an image in which the user is well-groomed. Also, the GAN can learn a camera image in which the correction flag is on as data of an image in which the user is not well-groomed. The correction processing unit 14 can generate a corrected image in which the user is well-groomed by the trained GAN.

[0058] The second method is to cut out a part of the user's registered image or the entire face, and replace the corresponding part of the user's captured image with the image cut out from the registered image to generate a corrected image. The second method will be specifically described with reference to FIG. 1. In the example of FIG. 1, the correction processing unit 14 cuts out images of the eyebrows, eyes, and mouth of the registered image. The correction processing unit 14 can replace the eyebrows, eyes, and mouth in the camera image with the images of the eyebrows, eyes, and mouth cut out from the registered image to generate a corrected image.

[0059] When replacing a part of the camera image with an image cut out from a registered image and synthesizing these images, the correction amount (here, the synthesis ratio) may be changed according to the degree of match between the camera image and the registered image. For example, the correction processing unit 14 may decrease the correction amount as the degree of match increases and increase the correction amount as the degree of match decreases.

[0060] The user may be allowed to set whether to cut out and replace the entire face of the registered image or to cut out and replace only a part of the face. When cutting out a part of the registered image, the user may be allowed to set which part of the face to cut out and replace.

[0061] The third method is to correct the camera image based on facial feature information stored in advance in the registered image database 11 or the like, without using a registered image, to generate a corrected image. The facial feature information is, for example, information on eyebrows, mouth, cheeks, skin color or brightness when makeup is applied. For the facial feature information, for example, multiple patterns may be prepared according to the scene of the video call, such as for business use or private use. This saves the user the trouble of preparing a registered image according to the scene of the video call.

[0062] The third method will be specifically described with reference to Fig. 6. In the example of Fig. 6, the correction processing unit 14 removes blemishes on the cheeks in the camera image by filtering to remove noise. Furthermore, the correction processing unit 14 can generate a corrected image (display image) by adjusting the saturation of the eyebrows, mouth, cheeks, and skin based on the facial feature information. When performing filtering or saturation adjustment, the amount of correction may be changed according to the degree of match between the camera image and the registered image.

[0063] Returning to FIG. 3, in S110, the output unit 15 outputs the video output from the correction processing unit 14. That is, when the correction flag is set to on, the output unit 15 outputs the corrected image generated by the correction processing unit 14. Also, when the correction flag is set to off, the output unit 15 outputs the uncorrected camera image of the user. The video output by the output unit 15 is transmitted to and displayed on another computer. Also, the video output by the output unit 15 is displayed on a display device.

[0064] The image processing device 1 repeats the above process for each frame until the user ends the call. When the user ends the call, the camera image is no longer acquired in S101, and the image correction process shown in FIG.

[0065] (Action and effect) In the above embodiment, the image processing device 1 acquires a camera image (captured image) and a registered image of a well-groomed person, extracts features from each image, and evaluates the degree of match. If the degree of match is equal to or greater than a predetermined threshold, the image processing device 1 determines that the person is well-groomed and disables correction processing. If the degree of match is less than the predetermined threshold, the image processing device 1 determines that the person is not well-groomed and enables correction processing. This allows the caller (user) to participate in a video call without worrying about the state of their appearance or performing a special operation to instruct image correction.

[0066] <Other> The above-described embodiment merely describes an exemplary configuration of the present invention. The present invention is not limited to the above-described specific embodiment, and various modifications are possible within the scope of the technical concept thereof.

[0067] For example, in the above embodiment, the correction determination unit 13 compares the feature amounts of the camera image and the registered image and evaluates the degree of match, but is not limited to this. The correction determination unit 13 may evaluate the difference between the feature amounts of the camera image and the registered image, and enable the correction process when the difference is equal to or greater than a predetermined threshold, and disable the correction process when the difference is less than the predetermined threshold.

[0068] Also, for example, in the above embodiment, the correction determination unit 13 disables the correction process when the degree of match between the camera image and the image of a well-groomed state is equal to or greater than a predetermined threshold, but this is not limited to the above. An image of an ungroomed state may be stored in the registered image database 11 as a reference image of the user. In this case, the correction determination unit 13 may disable the correction process when the degree of match between the camera image and the image of an ungroomed state is less than a predetermined threshold, and enable the correction process when the degree of match is equal to or greater than the predetermined threshold.

[0069] <Appendix 1> (1) an extraction unit (12) that extracts features of a captured image of a user and a reference image of the user; a determination unit (13) that determines whether to enable or disable a correction process for the captured image of the user based on a result of comparing a feature amount of the captured image of the user with a feature amount of a reference image of the user; a correction unit (14) that generates a corrected image of the captured image of the user when it is determined that the correction process is enabled; an output unit (15) that outputs the corrected image when it is determined that the correction process is enabled, and outputs an uncorrected captured image of the user when it is determined that the correction process is disabled; An image processing device (1) comprising:

[0070] (2) extracting features of a captured image of a user and a reference image of the user (S103); a determination step of determining whether to enable or disable a correction process for the captured image of the user based on a result of comparing the feature amount of the captured image of the user with the feature amount of a reference image of the user (S104 to S107); a correction step of generating a corrected image of the captured image of the user when it is determined that the correction process is enabled (S108, S109); an output step of outputting the corrected image when it is determined that the correction process is to be enabled, and outputting an uncorrected captured image of the user when it is determined that the correction process is to be disabled (S110); 13. An image processing method comprising: [Explanation of symbols]

[0071] 1: image processing device, 10: imaging section, 11: registered image database, 12: feature extraction section, 13: correction determination section, 14: correction processing section, 15: output section

Claims

1. an extraction unit that extracts feature amounts of a captured image of a user's face and a reference image of the user's face; a determination unit that determines whether to enable or disable a correction process for the captured image of the user's face based on a result of comparing a feature amount of the captured image of the user's face with a feature amount of a reference image of the user's face; a correction unit that generates a corrected image of the captured image of the face of the user when it is determined that the correction process is enabled; an output unit that outputs the corrected image when it is determined that the correction process is enabled, and outputs an uncorrected captured image of the user's face when it is determined that the correction process is disabled; Equipped with The image processing device according to claim 1, wherein the reference image of the user's face is an image of the user in an ungroomed state.

2. The feature amount is a feature amount of at least one of eyebrows and cheeks of the user.

2. The image processing device according to claim 1,

3. The feature amount includes at least one of a Haar-like feature amount, a color histogram, and a color moment.

3. The image processing device according to claim 1, wherein the first and second inputs are input to the image processing device.

4. The feature amount is calculated by an algorithm using a learning model that has been trained on images of the user in a well-groomed state and images of the user in an ungroomed state.

4. The image processing device according to claim 1, wherein the first and second inputs are input to the image processing apparatus.

5. The determination unit calculates a degree of coincidence between a feature amount of a captured image of the user's face and a feature amount of a reference image of the user's face, and determines to disable the correction process when the degree of coincidence is less than a predetermined threshold, and to enable the correction process when the degree of coincidence is equal to or greater than a predetermined threshold.

5. The image processing device according to claim 1,

6. The image processing apparatus according to claim 5 , wherein the correction unit changes an amount of correction applied to the captured image of the user's face depending on the degree of coincidence.

7. The correction unit generates the corrected image based on a captured image of the user's face and a reference image of the user's face.

7. The image processing device according to claim 1,

8. The correction unit generates the corrected image using a Generative Adversarial Network (GAN) that has learned a captured image of the user's face and a reference image of the user's face.

8. The image processing device according to claim 7,

9. The correction unit generates the corrected image by cutting out eyebrows, eyes, a mouth, or the entire face of the reference image of the user's face and replacing the corresponding part of the captured image of the user's face with the cut-out image.

8. The image processing device according to claim 7,

10. The correction unit performs a filter process for removing noise or a saturation adjustment on the captured image of the user's face based on facial feature information to generate the corrected image.

7. The image processing device according to claim 1,

11. The determination unit determines whether to enable or disable the correction process for each predetermined number of frames of a captured image of the user's face.

11. The image processing device according to claim 1,

12. The image capturing unit captures an image of the face of the user.

12. The image processing device according to claim 1,

13. An extraction step of extracting feature amounts of a captured image of a user's face and a reference image of the user's face; a determination step of determining whether to enable or disable a correction process for the captured image of the user's face based on a result of comparing a feature amount of the captured image of the user's face with a feature amount of a reference image of the user's face; a correction step of generating a corrected image of the captured image of the face of the user when it is determined that the correction process is enabled; an output step of outputting the corrected image when it is determined that the correction process is to be enabled, and outputting an uncorrected captured image of the user's face when it is determined that the correction process is to be disabled; Including, An image processing method, characterized in that the reference image of the user's face is an image of the user in an ungroomed state.

14. A program for causing a computer to execute each step of the method according to claim 13.

Images