CI generation method, generation device, and computer program

Abstract

To appropriately execute a CI method by increasing a change from a base image even when an image with a strong visual feature such as a face image of a specific individual is used as the base image.SOLUTION: The disclosed method is a method of generating a CI from a base image by a classificationimage (hereinafter referred to as "CI") method, and includes performing, on a first face image, preprocessing of removing a high spatial frequency component of an image by a spatial filter to obtain a second face image obtained by removing the high spatial frequency component of the first face image, and using the second face image as the base image to obtain the CI by the CI method.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a method, a generating device, and a computer program for generating a CI. [Background technology]

[0002] Non-Patent Document 1 discloses the generation of CIs by a classification image (CI) method. The CI method is a technique for visualizing mental images present in a person's brain as CIs.

[0003] In Non-Patent Document 1, a subject repeatedly selects one of two images created by overlaying randomly generated noise on the same face image. CIs are generated from all the selected images. In this CI method, the face image onto which noise is overlaid is called a base image.

[0004] In the experiment described in Non-Patent Document 1, a neutral male face image was used as a base image. From this base image, a face image that the participants of the experiment thought to have the appearance of a typical trustworthy person was generated as a CI.

[0005] In this experiment, participants were presented with two images of a neutral male face (base image) overlaid with random noise to generate CIs, and were asked to choose the image that more closely resembled the face of a trusted person.

[0006] The average noise in the image of the person the participant chose as trustworthy was superimposed on the base image, resulting in the CI, an approximation of what the participant thought a typical trustworthy face looked like (the participant's mental image of a trustworthy person). [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Japanese Patent Application Laid-Open No. 2010-92439 [Non-patent literature]

[0008] [Non-Patent Document 1] Ron Dotsch, and Alexander Todorov, “Reverse correlating social face perception”, Social Psychological and Personality Science, 3(5), 562-571, 2012. Summary of the Invention

[0009] In Non-Patent Document 1, an image of a neutral male face is used as the base image. The neutral face image is obtained by averaging the face images of many people. In this way, in the conventional CI method, an average face image of many people (hereinafter referred to as "average face of many people") is used as the base image.

[0010] However, conventional CI methods use an average face image of many people as a base image, making it difficult to generate a face image related to a specific individual's face as a CI. If a neutral average face image of many people is used as a base image, only CIs that are far removed from the face of a specific individual can be generated.

[0011] For example, even if an individual attempts to generate a CI based on what they consider to be their ideal face using a neutral multi-average face image as a base image, they will only be able to generate a CI that is far removed from their own face. This is because it is difficult to approximate their own face by overlaying noise on a neutral face that is different from their own face.

[0012] Therefore, the inventors came up with the idea of ​​using a facial image of a specific individual as a base image, rather than an average facial image of many people.

[0013] However, according to the inventors' experiments, even if the CI method is performed using a facial image of a specific individual as a base image, the resulting CI retains strong characteristics of the base image and changes little from the base image, making it difficult to effectively approximate the mental image.

[0014] Furthermore, the inventors have discovered that this problem occurs not only when a facial image of a specific individual is used as a base image, but also when an average image of a relatively small number of faces (hereinafter referred to as a "small number of people's average facial image") is used, which is an image of a face that maintains more specific features than an average facial image of a large number of people (a neutral facial image).

[0015] Therefore, even when an image that may cause the above-mentioned problems, such as a facial image of a specific individual, is used as a base image, a technology is desired that can increase the change from the base image and perform the CI method appropriately.

[0016] One aspect of the present disclosure is a method for generating a classification image (CI) from a base image using a CI algorithm. The disclosed method includes performing preprocessing on a first face image using a spatial filter to remove high spatial frequency components of the image, obtaining a second face image from which the high spatial frequency components of the first face image have been removed, and then using the second face image as the base image to obtain the CI using the CI algorithm.

[0017] Another aspect of the present disclosure is an apparatus. The disclosed apparatus is an apparatus for generating a CI from a base image by a classification image (CI) method. The disclosed apparatus is configured to execute a process including: performing preprocessing on a first face image by using a spatial filter to remove high spatial frequency components of the image, obtaining a second face image from which the high spatial frequency components of the first face image have been removed, and obtaining the CI by the CI method using the second face image as the base image.

[0018] Another aspect of the present disclosure is a computer program. The disclosed computer program causes a computer to execute a process for generating a CI from a base image using a classification image (CI) method. The process includes performing preprocessing on a first face image to remove high spatial frequency components of the image using a spatial filter, obtaining a second face image from which the high spatial frequency components of the first face image have been removed, and obtaining the CI using the second face image as the base image using the CI method.

[0019] Further details will be described in the following embodiments. [Brief explanation of the drawings]

[0020] [Figure 1] FIG. 1 is a flow chart illustrating a method according to an embodiment. [Figure 2] FIG. 2 is a configuration diagram showing a generating device according to an embodiment. [Figure 3] FIG. 3 is a flowchart and an explanatory diagram of the preprocessing. [Figure 4] Figure 4 shows the power spectrum showing the reduction in high frequency components due to preprocessing. [Figure 5] FIG. 5 is a flowchart showing the procedure of the CI method. [Figure 6] FIG. 6 is an explanatory diagram of the CI method. [Figure 7] FIG. 7 is a diagram showing the experimental results of the example. [Figure 8] FIG. 8 is a diagram showing the experimental results of the comparative example. [Figure 9] FIG. 9 is a flowchart illustrating a variation of the method according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0021] <1. Overview of CI generation method, generation device, and computer program>

[0022] (1) A method according to an embodiment may be a method for generating a CI from a base image by a classification image (hereinafter referred to as "CI") method. The method according to an embodiment includes performing preprocessing on a first face image to remove high spatial frequency components of the image using a spatial filter, obtaining a second face image from which the high spatial frequency components of the first face image have been removed, and then using the second face image as the base image to obtain the CI by the CI method. By removing the high spatial frequency components through preprocessing, even when a face image of a specific individual is used as the base image, the CI method can be appropriately performed by increasing the change from the base image. Note that the first face image may be an average face image of a small number of people, in addition to the face image of a specific individual.

[0023] (2) The first facial image is preferably a facial image of a specific individual.

[0024] (3) It is preferable that the first facial image is generated based on a plurality of facial images of a specific individual taken under different photographing conditions, in which case a stable facial image can be obtained.

[0025] (4) The method according to the embodiment may further include repeatedly performing the CI method using the CI obtained by the CI method as a base image. By repeating the CI method, mental images can be extracted more appropriately.

[0026] (5) The method according to the embodiment may further include performing post-processing to remove noise from the CI obtained by the CI method, thereby enabling facial features indicated by the CI to be more clearly expressed.

[0027] (6) An apparatus according to an embodiment may be a generating apparatus that generates a CI from a base image by a classification image (CI) method. The apparatus according to an embodiment may be configured to execute a process that includes performing preprocessing on a first face image by using a spatial filter to remove high spatial frequency components of the image, obtaining a second face image from which the high spatial frequency components of the first face image have been removed, and obtaining the CI by the CI method using the second face image as the base image.

[0028] (7) A computer program according to an embodiment may be a computer program for causing a computer to execute a process for generating a CI from a base image by a classification image (CI) method. The process may include performing preprocessing on a first face image by using a spatial filter to remove high spatial frequency components of the image, obtaining a second face image from which the high spatial frequency components of the first face image have been removed, and obtaining the CI by the CI method using the second face image as the base image.

[0029] 2. Examples of CI generating method, generating device, and computer program

[0030] Hereinafter, the embodiments will be described in more detail with reference to the drawings.

[0031] 1 shows an example of the procedure of a method according to an embodiment. The method according to an embodiment is executed by a CI generation device 10, for example.

[0032] In step S101, the generating device 10 acquires a facial image of a specific individual. Hereinafter, the acquired facial image itself will also be referred to as an “initial facial image.” The initial facial image is an image before preprocessing, which will be described later, is performed.

[0033] The specific individual may be the user using the generation device 10, or an individual other than the user. Hereinafter, the user's own facial image will be referred to as a "self-face image," and the facial image of an individual other than the user will be referred to as an "other-face image." Furthermore, the facial image of the specific individual will be referred to as an "individual-face image." The self-face image and the other-face image are each an example of an individual-face image.

[0034] The generation device 10 stores the acquired individual facial image. The generation device 10 generates a CI using the acquired individual facial image. By generating a CI based on the facial image of a specific individual, it is possible to avoid generating a CI that is far removed from the face of the specific individual, and to obtain a CI in which the face of the specific individual is changed according to a mental image.

[0035] Hereinafter, the mental image face approximated by the CI will be referred to as the "target face." The CI approximates the target face.

[0036] The target face is, for example, the user's own ideal face. An ideal face is a face that is considered ideal in a mental image. The CI of the user's own ideal face is generated from the user's own facial image. By visually showing the user's own ideal face in the CI, the user can easily and accurately convey their ideal face when applying makeup or cosmetic surgery.

[0037] It should be noted that the target face is not limited to an ideal face. For example, the target face may be the user's own face as imagined by the user. There are individual differences in how people perceive themselves, and it is predicted that the self-face CI of an individual who perceives themselves positively will be highly attractive, while the CI of an individual who perceives themselves negatively will be less attractive. In other words, it is possible to infer an individual's self-affirmation from the attractiveness of the self-face CI.

[0038] We also predict that the self-face CI reflects mood fluctuations within an individual. When a person is in a good mood, a positive facial expression such as a smile is seen in the self-face CI, and when a person is in a bad mood, a positive facial expression such as a smile is seen in the self-face CI. Therefore, it is possible to estimate the user's mood from the self-face CI.

[0039] Similarly, when a face image of another person is used as an image of a specific individual, the face of the other person can be impression-modulated according to the user's mental image to obtain a face (target face) as a CI.

[0040] However, experiments by the present inventors have revealed that it is difficult to effectively approximate a mental image in the CI method when a facial image of a specific individual is used as is. Therefore, in step S102, the generation device 10 performs preprocessing on the acquired initial facial image. The preprocessed image is used as a base image for the CI method. By performing preprocessing on the facial image of a specific individual, it becomes easier to effectively approximate a mental image even when the facial image of a specific individual is used. Details of the preprocessing will be described later.

[0041] In step S103, the CI method is executed based on the base image obtained in the preprocessing. The generating device 10 generates a CI that approximates the user's mental image from the base image obtained in the preprocessing by the CI method.

[0042] If necessary, in step S103A, the generating device 10 removes unnecessary noise contained in the CI as post-processing. Noise removal will be described later. In step S104, the generating device 10 outputs the CI from which noise has been removed to a display device or the like. This allows a user or the like to observe the CI. Note that noise removal does not have to be performed. This is because, depending on the cognitive ability of the image observer, an image (CI) with noise remaining may be able to more appropriately grasp the atmosphere represented by the image (CI) than a clear image from which noise has been removed. If noise removal is not performed, the CI with noise remaining will be output to a display device or the like.

[0043] Noise removal is performed, for example, by a noise remover. The generation device 10 may function as a noise remover. The noise remover, for example, removes unnecessary noise contained in the CI. The unnecessary noise contained in the CI refers to noise added to the base image other than the noise that contributes to changing the base image to approximate the target face. The unnecessary noise reduces the appearance of the CI. Noise removal can more clearly express the facial features indicated by the CI.

[0044] The denoiser may be, for example, a neural network configured to remove unwanted noise contained in the CI. For example, the neural network configured to remove unwanted noise contained in the CI is a denoising autoencoder (DAE) configured to remove such noise. The denoising autoencoder is a neural network that has been machine-learned to remove noise.

[0045] The generation device 10 is configured with one or more computers. The one or more computers may be one or more servers connected to a network such as the Internet, or may be smartphones, tablets, or personal computers. The generation device 10 may be configured as a server-client system including a server and a client such as a smartphone.

[0046] The computer constituting the generation device 10 may include a processor 20 and a storage device 30 connected to the processor 20. The computer may also include an interface 40. The interface 40 may be, for example, a communication interface for communicating with the outside, an input interface for an input device, or an output interface for an output device such as a display device. The computer may acquire a first image from the outside via the communication interface, for example. The computer may output a CI to a display device via the output interface, for example.

[0047] The processor 20 is, for example, a CPU.

[0048] The storage device 30 includes, for example, a primary storage device and a secondary storage device. The primary storage device is, for example, RAM. The secondary storage device is, for example, a hard disk drive (HDD) or a solid state drive (SSD). The storage device 30 includes a computer program 31 executed by the processor 20. The processor 20 reads and executes the computer program 31 stored in the storage device 30. The computer program has program code representing instructions for causing a computer to execute the generation process 21 according to the embodiment.

[0049] The storage device 30 may store the acquired initial face image 32. The storage device 30 may also store a first face image 33 and a second face image 34 generated in preprocessing from the first face image. The storage device 30 may also store a CI 35 generated by the generation process 21.

[0050] Fig. 3(A) shows an example of the procedure of pre-processing S102 shown in Fig. 1. It is assumed here that the personal face images acquired in step S101 prior to pre-processing S102 in Fig. 1 are multiple personal face images of the same person.

[0051] Each of the multiple personal face images is a photograph of the same face of the same person taken under different shooting conditions. The same face refers to a face in which the facial orientation, facial placement, facial expression, etc. in the image are substantially the same. Different shooting conditions refer to external factors other than the face that may affect the appearance of the face in the image when the same face of the same person is photographed. External factors include, for example, the environment at the time of shooting or factors related to the camera used for shooting.

[0052] 3(B), for example, it is assumed that four personal face images A1, A2, A3, and A4 are acquired as initial face images in step S101. These personal face images A1, A2, A3, and A4 are images of the same face of the same person, photographed under different photographing conditions a1, a2, a3, and a4, respectively.

[0053] As shown in FIGS. 3A and 3B, in step S301 of preprocessing S102, the generating device 10 generates an average face image 33 of multiple individual face images A1, A2, A3, and A4. Hereinafter, this average face image will be referred to as the "first face image." The first face image 33 is stored in the storage device 30. The average face image 33 is used as the first face image in order to obtain a stable individual face image. Although the appearance of the initial face image itself varies depending on the shooting conditions, a stable individual face image can be obtained by averaging face images of the same person. Note that step S301 may be omitted and one initial face image 32 may be used as the first face image 33.

[0054] 3(A) and 3(B), in step S302, the generating device 10 applies a spatial filter to the first facial image 33 to generate a second facial image 34 (base image). The second facial image 34 is stored in the storage device 30. This second facial image 34 is used as a base image for the CI method.

[0055] The spatial filter used in step S302 is for removing high spatial frequency components from the image. By using a personal face image from which high spatial frequency components have been removed as the base image, the CI method can effectively approximate the mental image even when using the face image of a specific individual.

[0056] Personal face images generally strongly represent the visual characteristics of each individual. These individual visual characteristics are important for generating CIs of personal faces. However, if these visual characteristics remain strong in the base image, they are too strong and cannot be appropriately modulated by adding noise to the base image. If the base image cannot be appropriately modulated, the resulting CI will retain the characteristics of the base image strongly, and the mental image cannot be effectively extracted.

[0057] Therefore, if we use a spatial filter to reduce overly strong visual features in the base image while preserving the visual features of an individual's face, we can appropriately modulate those visual features by overlaying noise on the base image. By appropriately modulating the base image, we can increase the change from the base image. Therefore, it becomes possible to present the user with an image that is relatively significantly different from the base image, and the CI method can effectively extract mental images.

[0058] It should be noted that the "removal" of high spatial frequency components does not necessarily mean that the high spatial frequency components are completely removed, but rather that the high spatial frequency components are removed so as to be reduced.

[0059] Furthermore, "high spatial frequency components" refers to frequency components in a spatial frequency region higher than the low spatial frequency components in an image where the spatial frequency components are not reduced by a spatial filter (low spatial frequency components). For example, Figure 4 shows the change in the power spectrum with and without preprocessing. In Figure 4, high spatial frequency information is reduced at frequencies above 11 [cycles / image]. Therefore, in the example of Figure 5, frequencies above 11 [cycles / image] are high spatial frequency components, and frequencies below that are low spatial frequency components.

[0060] The range of high spatial frequency components depends on the characteristics of the spatial filter. In any case, if a filter maintains low spatial frequency components while reducing higher spatial frequency components, it is a "spatial filter for removing high spatial frequency components."

[0061] The range of high spatial frequencies removed by the spatial filter preferably includes frequencies that represent the shapes of characteristic facial features such as the eyes, nose, and mouth in a face image, because the shapes of characteristic features such as the eyes, nose, and mouth strongly represent the visual characteristics of each individual.

[0062] As an example of a spatial filter, the imgaussfilt function of the programming language MATLAB can be used. For example, if the size of the face image is 2000 x 2000 pixels, it is preferable to set the value of the sigma variable in the imgaussfilt function to approximately 6.5 to 8. It is preferable to change the size of the face image that has been subjected to the spatial filter to 512 x 512 pixels for the final base image.

[0063] Fig. 5 shows the procedure of the CI method in step S101 of Fig. 1. The procedure shown in Fig. 5 conforms to the method of Non-Patent Document 1 and is executed by the generating device 10. In the CI method shown in Fig. 5, steps S502 to S504 are repeatedly executed multiple times (steps S501A and S501B). The number of repetitions is, for example, several tens to several hundreds of times.

[0064] In step S502, the generation device 10 applies an anti-noise pattern to the base image 34 to generate an image pair to be presented to the user. As shown in FIG. 6, the anti-noise pattern consists of a first noise pattern and a second noise pattern. The second noise pattern has the polarity of the first noise pattern reversed (a noise pattern that is mathematically negative of the first noise pattern). Therefore, the second noise pattern changes the base image 34 in the opposite direction to the first noise pattern.

[0065] 6, a first presentation image is generated by applying a first noise pattern to a base image, and a second presentation image is generated by applying a second noise pattern to the base image. The pair of the first presentation image and the second presentation image is the image pair to be presented to the user.

[0066] 5, in step S503, the generating device 10 presents the generated image pair to the user by displaying them side by side on a display device. The user selects the image from the presented image pair that the user considers to be closer to a target face (e.g., the user's own ideal face) using an appropriate input device (not shown). Note that three or more images may be presented to the user simultaneously.

[0067] In step S504, the generating device 10 accepts the user's selection and stores the image selected by the user (or the noise pattern applied to the image selected by the user).

[0068] The above steps S502 to S504 are executed repeatedly. In Fig. 6, this repetition is shown as "trial." In Fig. 6, the repetition is performed X times, from "trial 1" to "trial X." Therefore, X selected images (or selected noise patterns) are obtained.

[0069] The noise pattern applied to the base image is generated randomly for each trial. Therefore, the noise pattern applied to the base image may be different for each trial. Therefore, variously changed images are presented to the user for each trial. For each trial, the user selects the image that they consider to be closer to the target face (e.g., the user's own ideal face), and thus multiple images (or their noise patterns) that the user considers to be similar to the target face are obtained.

[0070] In step S505 after repeating steps S502 to S504, the generating device generates a CI based on the user's selection. The CI can be generated, for example, as an average image of multiple images selected by the user. Alternatively, the CI can be generated by applying to a base image the average noise pattern of the noise patterns applied to the images selected by the user. Regardless of the generation method, a CI that approximates the user's mental image can be obtained based on the user's selection.

[0071] A method for generating the random noise pattern to be applied to the base image is described in Figure 2 of Non-Patent Document 1. The generating device 10 according to the embodiment can also generate a noise pattern in a similar manner to the method described in Non-Patent Document 1. In Non-Patent Document 1, the noise pattern is generated by superimposing 12 sine waves per cycle (6 directions x 2 phases) for each spatial frequency (2, 4, 8, 16, and 32 cycles / image). Each sine wave has a random parameter that indicates its contrast.

[0072] 7 and 8 show experimental results. FIG. 7 shows the results when the generating device 10 according to the embodiment is used (with preprocessing). FIG. 8 shows the results without preprocessing. In FIG. 8, an initial face image without preprocessing is used as the base image. Even when noise is added to the initial face image, high spatial frequency information is hardly modulated, and as shown in FIG. 8, the appearance of the face in the image with noise added is almost unchanged from the initial face image. Therefore, even when a CI is obtained by implementing the CI method, the CI changes little from the original initial face image.

[0073] On the other hand, if a preprocessed personal face image (second face image) is used as the base image, adding noise will significantly change the appearance of the face. Therefore, the CI obtained by implementing the CI method will be significantly different from the initial face image. Moreover, since the base image is a personal face image, the CI also retains the individual's characteristics.

[0074] In this way, according to the embodiment, it is possible to extract the mentally imagined facial image of an individual with unprecedented accuracy using an individual facial image as a base image.

[0075] Furthermore, visualization of one's ideal face is particularly useful in the fields of cosmetic surgery and plastic surgery. By visualizing one's ideal face using the CI method, one can easily and accurately convey one's ideal when applying makeup or cosmetic surgery. In addition, it is possible to visualize facial images of specific faces other than one's own, with various impressions added.

[0076] The preprocessing according to the embodiment is also useful when an average face image of a small number of people, such as two or several people, is used instead of an individual face image. When the number of people to be averaged is small, removing high spatial frequency components by preprocessing has the same effect as in the case of an individual face image.

[0077] Fig. 9 shows a modification of the generation process 21 shown in Fig. 1. The procedure of Fig. 9 is also executed by the generation device 10.

[0078] In step S901 of FIG. 9, the generating device 10 acquires and stores a face image (initial face image) of a specific individual.

[0079] In step S902, the generating device 10 performs preprocessing on the acquired initial face image. The preprocessed image is used as a base image for the CI method. The preprocessing is performed as shown in FIG.

[0080] Step S904 in Fig. 9 is repeatedly executed multiple times (steps S903A and S903B). Step S904 is a process equivalent to steps S502 to S504 shown in Fig. 5, and step S904, which corresponds to steps S502 to S504, is repeatedly executed in the same manner as in Fig. 5. In step S905 after the repetition, a CI is generated.

[0081] If necessary, in step S906, the generating device 10 removes unnecessary noise contained in the CI. Note that noise removal does not necessarily have to be performed.

[0082] In step S907, the generating device 10 executes step S903A and subsequent steps using the generated CI as a new base image. By repeating steps S903A, S904, S903B, and S905 (and step S906, if necessary) multiple times, the initial base image can be changed more significantly to obtain a CI that more effectively captures the mental image. The number of times steps S903A, S904, S903B, and S905 (and step S906, if necessary) are repeated can be set appropriately. In step S908, the obtained CI is output to a display device or the like.

[0083] The present invention is not limited to the above-described embodiment, and various modifications are possible. [Explanation of symbols]

[0084] 10:Generation device 20: Processor 21: Generation process 30: Storage device 31: Computer Program 32: Initial face image 33: First face image 34: Second facial image (base image) 40: Interface A1: Personal face image (initial face image) A2: Personal face image (initial face image) A3: Personal face image (initial face image) A4: Personal face image (initial face image)

Claims

1. A method for generating a classification image (hereinafter referred to as "CI") from a base image by a CI method, comprising: performing preprocessing on the first face image by using a spatial filter to remove high spatial frequency components of the image, thereby obtaining a second face image from which the high spatial frequency components of the first face image have been removed; obtaining the CI by the CI method using the second face image as the base image; A method for generating a CI, comprising:

2. The first facial image is a facial image of a specific individual. The method for generating a CI according to claim 1 .

3. The first facial image is generated based on a plurality of facial images of a specific individual captured under different photographing conditions. The method for generating a CI according to claim 1 .

4. The CI method is further performed repeatedly using the CI obtained by the CI method as a base image. The method for generating a CI according to claim 1 .

5. further comprising performing post-processing to remove noise contained in the CI obtained by the CI method. The method for generating a CI according to claim 1 .

6. A generating device for generating a classification image (hereinafter referred to as "CI") from a base image by a CI method, performing preprocessing on the first face image by using a spatial filter to remove high spatial frequency components of the image, thereby obtaining a second face image from which the high spatial frequency components of the first face image have been removed; obtaining the CI by the CI method using the second face image as the base image; 10. A generating device configured to perform a process comprising:

7. A computer program for causing a computer to execute a process for generating a classification image (hereinafter referred to as "CI") from a base image by a CI method, The process comprises: performing preprocessing on the first face image by using a spatial filter to remove high spatial frequency components of the image, thereby obtaining a second face image from which the high spatial frequency components of the first face image have been removed; obtaining the CI by the CI method using the second face image as the base image; A computer program comprising:

Images