Image processing device, and program

Abstract

An image processing device according to one aspect of the present invention is equipped with an image acquisition unit, a feature point recognition unit, an emotion estimation unit, a movement amount storage unit, a cognitive tendency storage unit, a conversion intensity acquisition unit, a movement amount adjustment unit, and a facial expression conversion unit. The image acquisition unit acquires image data of an image including a person's face region. The feature point recognition unit recognizes feature points in the face region from the image data. The emotion estimation unit estimates the type and intensity of the person's emotion from the image data. The movement amount storage unit stores a movement amount set for each feature point according to each emotion type. The cognitive tendency storage unit stores each individual's cognitive tendency of facial expressions. The conversion intensity acquisition unit obtains a conversion intensity corresponding to the emotion intensity on the basis of a target individual's cognitive tendency. The movement amount adjustment unit adjusts the movement amounts corresponding to the estimated emotion type using the obtained conversion intensity. The facial expression conversion unit converts the image data by moving the feature points by the adjusted movement amounts to generate a face image for the target individual.

Description

Image processing device and program

[0001] One aspect of the present invention relates to an image processing device and a program.

[0002] For example, facial images of people are transformed to generate facial images with various emotions. There are known techniques that can obtain natural facial expressions through relatively simple image processing, even if they are not as large-scale as so-called generative AI (Artificial Intelligence). Rigid MLS, described in Non-Patent Document 1, is one such technique and can generate facial images with naturally transformed facial expressions in real time.

[0003] Shigeaki Yoshida et al., "Manipulating Emotional Experience with Real-Time Facial Deformation Feedback," Transactions of the Human Interface Society, Vol. 17, No. 1, 2015

[0004] The term autism spectrum disorder (ASD) is becoming more common. It is said that people with this tendency have difficulty reading emotions conveyed in other people's facial expressions, meaning they have difficulty recognizing facial expressions. Therefore, enhancing the facial expressions of captured facial images using techniques such as those described in Non-Patent Document 1 and displaying them in real time is thought to help resolve this difficulty. However, facial expression recognition tendencies vary greatly from person to person, which is why the condition is called a spectrum. For this reason, if the degree of change in facial expressions is insufficient or fixed, some people may still have difficulty recognizing facial expressions and emotions.

[0005] Therefore, the present invention aims to provide a technique that enables emotions to be reliably conveyed regardless of cognitive tendencies.

[0006] An image processing device according to one aspect of the present invention includes an image acquisition unit, a feature point recognition unit, an emotion estimation unit, a movement amount storage unit, a cognitive tendency storage unit, a conversion strength acquisition unit, a movement amount adjustment unit, and a facial expression conversion unit. The image acquisition unit acquires image data of an image including a facial region of a person. The feature point recognition unit recognizes feature points in the facial region from the image data. The emotion estimation unit estimates the person's emotion type and emotion intensity from the image data. The movement amount storage unit stores a movement amount set for each feature point according to the emotion type. The cognitive tendency storage unit stores the cognitive tendency of facial expressions for each individual. The conversion strength acquisition unit acquires a conversion strength according to the emotion intensity from the cognitive tendency of the target individual. The movement amount adjustment unit adjusts the movement amount corresponding to the estimated emotion type using the acquired conversion strength. The facial expression conversion unit converts the image data by moving the feature points by the adjusted movement amount, thereby generating a facial image for the target individual.

[0007] According to one aspect of the present invention, emotions can be reliably conveyed regardless of cognitive tendencies.

[0008] FIG. 1 is a functional block diagram showing an example of an image processing device according to an embodiment. FIG. 2 is a diagram showing an example of coordinates of recognized feature points. FIG. 3 is a diagram showing an example of movement amounts of feature points set for each facial expression. FIG. 4 is a diagram showing an example of cognitive tendencies for each individual. FIG. 5 is a diagram for explaining the operation of the image processing device 10. FIG. 6 is a block diagram showing an example of the hardware configuration of the image processing device 10.

[0009] 1 is a functional block diagram showing an example of an image processing device according to an embodiment. The image processing device 10 as a computer includes an image acquisition unit 21, a feature point recognition unit 22, a facial expression conversion unit 44, a feeling estimation unit 41, a conversion strength acquisition unit 43, a movement amount adjustment unit 45, a cognitive tendency storage unit 32, and a movement amount storage unit 31.

[0010] The image acquisition unit 21 acquires image data of an image to be processed from an image captured by a web camera, an existing database, etc. The acquired image data is stored in a storage device such as a memory or a hard disk drive (HDD).

[0011] The image includes a facial region, such as a person's face, that is the target of image processing. The image and its part, the facial region, are composed of a plurality of pixels. In the embodiment, it is considered that the facial region is deformed to, for example, enhance a smile to generate an image of a smile.

[0012] 1, the feature point recognition unit 42 processes the acquired image data using, for example, the technology described in Reference [1] to recognize feature points in the facial region. The position data (coordinates) of the recognized feature points are passed to the facial expression conversion unit 44.

[0013] 2 is a diagram showing an example of the coordinates of recognized feature points. Each feature point is identified by an identifier (ID) and is associated with x-coordinate and y-coordinate values. Image transformation corresponds to converting the positions of feature points and their surrounding pixels.

[0014] The emotion estimation unit 41 processes the acquired image data using, for example, the technology described in Reference [2], and estimates the emotion type and emotion intensity of the person in the image. The emotion type is a typical human emotion such as happiness or sadness, and reflects facial expressions. The emotion intensity is an index that represents the strength of facial expressions corresponding to the emotion.

[0015] The movement amount storage unit 31 stores the movement amount set for each feature point. The movement amount for each feature point is further subdivided and stored according to the type of emotion. FIG. 3 is a diagram showing an example of the movement amount of feature points set for each facial expression. Even for the same feature point, the movement amount changes depending on the type of facial expression to be converted. This corresponds to the fact that when a person is happy, the eyebrows are raised and the corners of the mouth are also raised, and when a person is sad, the opposite occurs. In this embodiment, the movement amount set in advance in this way is adjusted according to each individual's tendency to recognize facial expressions, so that anyone can understand the emotion conveyed in the facial expression in an image.

[0016] The cognitive tendency storage unit 32 stores the cognitive tendency of facial expressions for each user (individual) by associating emotion intensity with conversion intensity. For example, the constant method described in reference [3] can be used here. In other words, the cognitive tendency storage unit 32 stores the cognitive tendency of facial expressions for each individual.

[0017] 4 is a diagram showing an example of the cognitive tendency for each individual. The cognitive tendency can be expressed as shown in equation (1) using a logistic curve, for example.

[0018] In formula (1), y is emotion intensity, x is transformation intensity, and a and b are parameters. That is, cognitive tendency can be expressed by two parameters, a and b. In other words, the cognitive tendency of each individual can be associated with parameters representing a logistic curve, and the cognitive tendency storage unit 32 stores the parameters a and b in association with a user ID (identifier).

[0019] The conversion strength acquisition unit 43 receives the emotion type and emotion intensity of the image data estimated by the emotion estimation unit 41 and the cognitive tendency of the target user as input, and outputs a conversion strength. That is, the conversion strength acquisition unit 43 acquires a conversion strength according to the emotional strength from the cognitive tendency of the target individual.

[0020] The movement amount adjustment unit 45 receives as input the emotion type of the image data estimated by the emotion estimation unit 41 and the conversion strength acquired by the conversion strength acquisition unit 43, and outputs an adjusted movement amount obtained by multiplying the movement amount corresponding to the emotion type by the conversion strength. That is, the movement amount adjustment unit 45 adjusts the movement amount corresponding to the estimated emotion type using the acquired conversion strength, and outputs the adjusted movement amount.

[0021] The facial expression conversion unit 44 converts the facial expression of the facial region of the image data based on the image data, feature points, emotion type, and adjustment movement amount, and outputs the converted expression. Here, for example, the technology of Non-Patent Document 1 can be applied. That is, the facial expression conversion unit 44 converts the image data by moving the feature points of the facial region by the adjustment movement amount, and generates a facial image for the target individual.

[0022] FIG. 5 is a diagram illustrating the operation of the image processing device 10. In FIG. 5, feature points are extracted from image data of the original image before conversion. Furthermore, the type and intensity of emotion in the image data are estimated, for example, using known AI (Artificial Intelligence) technology. In the example of FIG. 5, a smile with an intensity of 0.8 is estimated.

[0023] Meanwhile, the cognitive tendency of a specific individual to whom a converted image is to be presented is obtained from the cognitive tendency storage unit 32. This cognitive tendency is expressed as a logistic curve showing the relationship between emotion intensity and conversion intensity, as shown in the graph of FIG. 5, for example. Here, emotion intensity refers to the intensity of emotion obtained by emotion estimation (vertical axis of the graph). Conversion intensity refers to the degree to which facial expression is converted (horizontal axis of the graph). In FIG. 5, a value of 0.7 is obtained as the value on the horizontal axis (conversion intensity) corresponding to the emotion intensity (0.8) on the vertical axis.

[0024] The amount of movement of the feature point corresponding to the emotion type (smile) is then read from the movement amount storage unit 31, and is multiplied by the transformation strength (0.7) to obtain an adjusted movement amount. By moving the feature point by this adjusted movement amount, a transformed image to be presented to the specific individual can be generated.

[0025] As described above, according to the embodiment, the type of facial expression and the facial expression intensity (emotion intensity) are obtained from the image before conversion. Meanwhile, the facial expression recognition ability (recognition tendency) for each individual is stored in advance, and the conversion intensity of the image for each individual is calculated based on the type of facial expression, the facial expression intensity, and the facial expression recognition ability. Then, the movement amount of the reference feature point is adjusted according to the conversion intensity, and the feature point is moved by the adjusted movement amount.

[0026] This allows the image to be transformed to a sufficient degree according to an individual's ability to recognize facial expressions, and therefore, according to the embodiment, emotions can be reliably conveyed regardless of cognitive tendencies.

[0027] The functions of the image processing device 10 can be realized by installing a program on a computer. For example, the functions of the image processing device 10 can be implemented by causing a computer to execute a program provided as package software or online software.

[0028] Fig. 6 is a block diagram showing an example of the hardware configuration of the image processing device 10. As shown in Fig. 15, the image processing device 10 includes a CPU (Central Processing Unit) 20A, a bridge circuit 102, a memory 30, a GPU (Graphics Processing Unit) 20B connected to a display 104, a BIOS-ROM 107, a storage 109, a USB connector 110, and an input unit 112.

[0029] The storage 109 is a non-volatile storage medium (block device), such as a hard disk drive (HDD) or a solid state drive (SSD). The storage 109 stores basic programs such as an operating system (OS) 35 and device drivers, as well as a program 34 for implementing the functions of the image processing device 10.

[0030] The memory 30 includes a ROM (Read Only Memory) and a RAM (Random Access Memory). The CPU 20A and GPU 20B are arithmetic elements related to the processor 20. The CPU 20A mainly controls the image processing device 10. The GPU 20B mainly performs calculations related to image processing (such as multiply-accumulate operations) at high speed. The CPU 20A executes a BIOS (basic input / output system) stored in the BIOS-ROM 107. The CPU 20A also loads a program 34 from the storage 109 into the memory 30 and executes it. The same is true for the GPU 20B.

[0031] The bridge circuit 102 relays data transmission between the CPU 20A and the GPU 20B and each component. For example, the bridge circuit 102 is interposed between the CPU 20A and the GPU 20B and hardware devices connected to a PCI (Peripheral Component Interconnect) bus or a PCIe bus (not shown), relaying communication between them.

[0032] The USB connector 110 connects a USB device or the like. For example, the program 34 may be installed in the image processing device 10 via the USB device. The input unit 112 connects the image processing device 10 to a communication network such as a LAN (Local Area Network) or a WAN (Wide Area Network).

[0033] The program 34 and various data may be stored in a removable storage medium other than the storage 109, and may be read by the CPU 20A from a disk drive or the like. Alternatively, the program 34 and various data may be stored in another computer connected via a communication network, and may be read by the CPU 20A via the input unit 112.

[0034] It should be noted that the present invention is not limited to the above-described embodiment. For example, the image processing device 10 may be a notebook, desktop, or server computer. Alternatively, it may be a portable terminal such as a smartphone or tablet. Furthermore, it is also possible to make computer resources virtualized on the cloud function as the image processing device 10.

[0035] That is, this invention is not limited to the above-described embodiments, and the components can be modified and embodied in practice without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

[0036] <References> [1] "dlib C++ Library", [online], [searched on March 26, 2024], Internet,<URL: http: / / dlib.net / > [2] Serengil+, “LightFace: A Hybrid Deep Face Recognition Framework”, IEEE ASYU, 2020. [3] Yasuhisa Nakano: “Psychophysical Measurement Method”, Vision 7, pp. 17-27, 1995.

[0037] 10...Image processing device 20...Processor 20A...CPU 21...Image acquisition unit 22...Feature point recognition unit 30...Memory 31...Movement amount storage unit 32...Cognitive tendency storage unit 34...Program 41...Emotion estimation unit 42...Feature point recognition unit 43...Conversion strength acquisition unit 44...Facial expression conversion unit 45...Movement amount adjustment unit 102...Bridge circuit 104...Display 109...Storage 110...USB connector 112...Input unit.

Claims

1. An image processing device comprising: an image acquisition unit that acquires image data of an image including a facial region of a person; a feature point recognition unit that recognizes feature points in the facial region from the image data; an emotion estimation unit that estimates the emotional type and emotional intensity of the person from the image data; a movement amount storage unit that stores a movement amount set for each of the feature points according to the emotional type; a cognitive tendency storage unit that stores the cognitive tendency of facial expressions for each individual; a conversion strength acquisition unit that acquires a conversion strength according to the emotional intensity from the cognitive tendency of a target individual; a movement amount adjustment unit that adjusts the movement amount corresponding to the estimated emotional type by the acquired conversion strength; and an expression conversion unit that converts the image data by moving the feature points by the adjusted movement amount, and generates a facial image for the target individual.

2. The image processing device according to claim 1, wherein the cognitive tendency storage unit stores cognitive tendencies expressed by a logistic curve.

3. A program containing instructions for causing a computer to function as the image processing device according to claim 1 or 2.

Images