Information processing device and information processing method

Abstract

To allow a line of sight of an attendant to be matched with a line of sight of another attendant.SOLUTION: An information processing device 1 includes: an estimation unit 133 that estimates a ratio between an iris width and a white eye width when an object person faces the front, by analyzing a first image obtained by imaging the face of the object person; an angle identification unit 132 that analyzes a second image acquired after the first image has been acquired, to identify a lateral deviation angle at which the face captured in the second image is laterally deviated using a front-face direction of the face of the object person as a reference; a width identification unit 137 that identifies a ratio between the iris width and the left white eye width / the right white eye width when the object person visually recognizes the front face direction while the face of the object person is deviated by the lateral deviation angle in the second image on the basis of the estimated ratio and a size of the eyeball, a size of the iris and the lateral deviation angle of the object person; a correction unit 138 that corrects the second image so that a shape of the iris and the white eye of the object person in the second image becomes a shape corresponding to the identified ratio between the iris width and the left white eye width / the right white eye width; and an output unit 139 that outputs the corrected second image.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an information processing device and an information processing method. [Background technology]

[0002] Conventionally, a conference system has been known that realizes a Web conference among multiple participants by transmitting images captured by each of the terminals of multiple participants participating in the conference remotely and collected audio (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2011-205243 A Summary of the Invention [Problem to be solved by the invention]

[0004] In a Web conference, if the monitor of a participant's terminal is located far away from the imaging device, the participant often looks at the monitor without looking at the imaging device. In this case, the participant's eyes do not meet those of the other participants on the other participants' terminals, which causes a problem of impediment to non-verbal communication based on eye contact.

[0005] The present invention has been made in consideration of these points, and has an object to enable a participant's gaze to be aligned with the gaze of other participants. [Means for solving the problem]

[0006] An information processing device according to a first aspect of the present invention includes an acquisition unit that acquires a plurality of images of a subject's face captured by an imaging device; a size identification unit that acquires size information indicating the size of the subject's eyeballs and the size of the iris to identify the size of the eyeballs and the size of the iris; an estimation unit that estimates a ratio of the width of the iris to the width of the white of the eye when the subject faces forward by analyzing a first image acquired by the acquisition unit; and an angle identification unit that analyzes a second image acquired by the acquisition unit after acquiring the first image to identify a lateral deviation angle, which is an angle at which the face shown in the second image is shifted laterally when the frontal direction of the subject's face is used as a reference; The device has a width specifying unit that specifies a ratio between the width of the subject's pupil, the left white of the eye width indicating the width of the left white of the eye that is the white of the eye reflected to the left of the pupil, and the right white of the eye width indicating the width of the right white of the eye that is the white of the eye reflected to the right of the pupil, when the subject looks at the front direction in the second image with the face shifted by the lateral shift angle based on the size, the lateral shift angle, and the estimated ratio; a correction unit that corrects the second image so that the shapes of the pupils and whites of the subject's eyes included in the second image correspond to the specified ratio of the pupil width, the left white of the eye width, and the right white of the eye width; and an output unit that outputs the second image in which the subject's eyes have been corrected by the correction unit.

[0007] The estimation unit may estimate the ratios corresponding to each of the left and right eyes of the subject by analyzing the first image acquired by the acquisition unit. The estimation unit may determine a ratio between the width of the pupil and the width of the white of the eye of the subject shown in each of the multiple first images by analyzing each of the multiple first images acquired by the acquisition unit, and estimate the ratio between the width of the pupil and the width of the white of the eye when the subject is facing forward by averaging the ratios determined for each of the multiple first images.

[0008] The angle identification unit may identify the lateral shift angle of the face shown in the first image when the forward direction of the face of the subject is used as a reference by analyzing the first image, and the estimation unit may estimate a ratio between the width of the pupil and the width of the white of the eye of the subject shown in each of the multiple first images by analyzing each of the multiple first images when the angle identification unit has identified a different lateral shift angle.

[0009] The estimation unit may estimate the ratio between the width of the subject's pupil and the width of the white of the eye in the first image acquired by the acquisition unit, under the condition that the white of the eye is shown on both sides of the pupil of the subject in the first image acquired by the acquisition unit.

[0010] The information processing device may further have a color identification unit that identifies the color of the pupil, the color of the left white of the eye, and the color of the right white of the eye by analyzing the first image, and the correction unit may correct the color of the subject's eyes appearing in the second image based on the color of the pupil, the color of the left white of the eye, and the color of the right white of the eye identified by the color identification unit.

[0011] The information processing device may further have a iris analysis unit that identifies a size of light reflected in the iris and a position of the light in the iris by analyzing the first image, and the correction unit may depict the light relative to the iris based on the identified size of the light and position of the light.

[0012] The information processing device may further have a iris analysis unit that determines the ratio between the width of the iris and the width of the pupil in the iris by analyzing the first image, and the correction unit may depict the pupil relative to the iris based on the determined ratio between the width of the iris and the width of the pupil in the iris.

[0013] The angle identification unit may identify a vertical deviation angle, which is an angle by which the face of the subject in the second image is shifted in the up-down direction when the front direction of the face of the subject is used as a reference, by analyzing the second image, and the correction unit may correct the eyes of the subject in the second image when the magnitude of the vertical deviation angle is equal to or smaller than a predetermined angle.

[0014] An information processing method according to a second aspect of the present invention is executed by a computer and includes the steps of: acquiring a plurality of images of a subject's face captured by an imaging device; identifying the size of the subject's eyeballs and the size of the iris by acquiring size information indicating the size of the eyeballs and the size of the iris; estimating the ratio of the width of the iris to the width of the white of the eye when the subject faces forward by analyzing the acquired first image; identifying a lateral deviation angle, which is an angle at which the face shown in the second image is shifted laterally when the front direction of the subject's face is taken as a reference, by analyzing a second image acquired after the first image is acquired; The method includes a step of determining a ratio between the width of the subject's pupil when the subject looks toward the front in the second image with the face shifted by the lateral shift angle relative to the front direction in a state in which the face is shifted by the lateral shift angle based on the size of the pupil, the left white of the eye width indicating the width of the left white of the eye that is the white of the eye reflected to the left of the pupil, and the right white of the eye width indicating the width of the right white of the eye that is the white of the eye reflected to the right of the pupil based on the estimated ratio; a step of correcting the second image so that the shapes of the pupil and white of the subject's eyes included in the second image correspond to the determined ratio of the pupil width, the left white of the eye width, and the right white of the eye width; and a step of outputting the second image in which the subject's eyes have been corrected. Effect of the Invention

[0015] According to the present invention, it is possible to allow a participant's line of sight to match the line of sight of other participants. [Brief description of the drawings]

[0016] [Figure 1] FIG. 1 is a diagram illustrating an overview of an information processing device. [Diagram 2] FIG. 2 is a diagram illustrating a functional configuration of an information processing device. [Diagram 3] 13A and 13B are diagrams illustrating the state of the eyeballs when the subject's face is shifted to the side; [Figure 4]FIG. 4 is a close-up view of the eyeball of the subject shown in FIG. [Diagram 5] 5 is an enlarged view of a sector portion of the eyeball shown in FIG. 4, the sector portion connecting the center of the eyeball, the left end of the left white of the eye width, and the right end of the right white of the eye width. [Figure 6] 13A and 13B are diagrams showing the relationship between the left white of the eye width in real space and the left white of the eye width in the second image, and the relationship between the right white of the eye width in real space and the right white of the eye width in the second image. [Figure 7] FIG. 13 is a diagram illustrating an example of correction of a subject's eyes. [Figure 8] 4 is a flowchart showing a process flow in an information processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] [Overview of information processing device 1] 1 is a diagram showing an overview of an information processing device 1. The information processing device 1 is a computer that corrects the eyes of a subject appearing in an image captured by an imaging device. The information processing device 1 is communicably connected to a first terminal 2 and a second terminal 3 via a communication network such as the Internet or a wireless LAN (Local Area Network). The information processing device 1 has a function for realizing a Web conference, for example, and outputs an image of a subject captured by the first terminal 2 and a sound collected by the first terminal 2 to the second terminal 3, and also outputs an image of a subject captured by the second terminal 3 and a sound collected by the second terminal 3 to the first terminal 2.

[0018] The first terminal 2 and the second terminal 3 are, for example, terminals such as a personal computer or a smartphone having an imaging device and a microphone, and transmit images captured by the imaging device and sounds collected by the microphone to the information processing device 1. The first terminal 2 and the second terminal 3 may acquire images captured by an imaging device connected thereto and transmit the images to the information processing device 1. In the following description, an example of correcting the eyes of a subject appearing in an image captured by the first terminal 2 will be described. The information processing device 1 may correct the eyes of a subject appearing in an image captured by the second terminal 3. Although only one second terminal 3 is shown in FIG. 1, a plurality of second terminals 3 may be present.

[0019] The information processing device 1 acquires multiple first images capturing the subject's face from the first terminal 2 ((1) in FIG. 1). The information processing device 1 analyzes the multiple acquired first images to estimate the ratio between the pupil width, which is the width of the pupil when the subject faces forward, and the white of the eye width, which is the width of the white of the eye when the subject faces forward ((2) in FIG. 1). The pupil width indicates the width (diameter) of the circle indicating the pupil, and the white of the eye width indicates the sum of the maximum widths of the two white of the eye areas on the left and right of the pupil.

[0020] After estimating the ratio between the width of the black eye and the width of the white eye, the information processing device 1 acquires a second image from the first terminal 2 ((3) in FIG. 1). The information processing device 1 analyzes the acquired second image to identify a lateral deviation angle, which is an angle at which the face reflected in the second image is shifted laterally with respect to the front direction of the subject's face ((4) in FIG. 1).

[0021] The information processing device 1 determines the ratio of the subject's pupil width, left white eye width, and right white eye width when the subject looks straight ahead with the subject's face shifted by the lateral shift angle with respect to the front direction in the second image, based on the previously acquired size of the subject's eyeball and pupil, the determined lateral shift angle, and the estimated ratio of pupil width to white eye width ((5) in FIG. 1). The left white eye width refers to the maximum width of the left white eye area, which is the white of the eye reflected to the left of the pupil, and the right white eye width refers to the maximum width of the right white eye area, which is the white of the eye reflected to the right of the pupil.

[0022] The information processing device 1 corrects the eyes of the subject appearing in the second image so that the shapes of the pupils and whites of the subject's eyes in the second image correspond to the specified ratio of pupil width, left white width, and right white width ((6) in FIG. 1). The information processing device 1 outputs the corrected second image, which is the corrected second image, to the second terminal 3 ((7) in FIG. 1). In this way, the information processing device 1 can align the gaze of the subject using the first terminal 2 appearing in the second image with the gaze of a subject using the second terminal 3 that is different from the subject.

[0023] [Functional configuration of information processing device 1] Next, a detailed description will be given of the configuration of the information processing device 1. Fig. 2 is a diagram showing the functional configuration of the information processing device 1. The information processing device 1 has a communication unit 11, a storage unit 12, and a control unit 13. The communication unit 11 is a communication interface for transmitting and receiving data to and from the first terminal 2 and the second terminal 3 via a communication network such as the Internet or a mobile phone communication network.

[0024] The storage unit 12 is a storage medium that stores various data, and includes a ROM (Read Only Memory), a RAM (Random Access Memory), a hard disk, and an SSD (Solid State Drive). The storage unit 12 stores a program executed by the control unit 13. The storage unit 12 stores a program that causes the control unit 13 to function as an acquisition unit 131, an angle identification unit 132, an estimation unit 133, a color identification unit 134, a iris analysis unit 135, a size identification unit 136, a width identification unit 137, a correction unit 138, and an output unit 139.

[0025] The control unit 13 is, for example, a CPU (Central Processing Unit). The control unit 13 executes a program stored in the storage unit 12 to function as an acquisition unit 131, an angle identification unit 132, an estimation unit 133, a color identification unit 134, a iris analysis unit 135, a size identification unit 136, a width identification unit 137, a correction unit 138, and an output unit 139. The acquisition unit 131, the angle identification unit 132, the estimation unit 133, the color identification unit 134, the iris analysis unit 135, the size identification unit 136, the width identification unit 137, the correction unit 138, and the output unit 139 will be described in detail below.

[0026] [Acquisition of an image of the subject] The acquisition unit 131 acquires a plurality of images of the face of the subject captured by an imaging device. Specifically, the acquisition unit 131 acquires in real time a plurality of images of the face of the subject using the first terminal 2 captured at different times by an imaging device provided in the first terminal 2.

[0027] [Subject's eye analysis] Next, a process of analyzing the subject's eyes will be described. In correcting the subject's eyes, the angle identification unit 132, the estimation unit 133, the color identification unit 134, and the iris analysis unit 135 of the control unit 13 work together to analyze the subject's eyes shown in the first image acquired by the acquisition unit 131. Hereinafter, a specific process of analyzing the subject's eyes will be described.

[0028] The angle identification unit 132 analyzes each of the multiple first images acquired by the acquisition unit 131 to identify a lateral deviation angle, which is an angle at which the subject's face reflected in the first image is shifted laterally when the front direction of the subject's face is used as a reference. The angle identification unit 132 analyzes each of the multiple first images using, for example, an existing face direction estimation technology to identify the lateral deviation angle of the subject's face reflected in each of the multiple first images. The angle identification unit 132 identifies a lateral deviation angle when the subject's face is facing more to the right than the front as a positive lateral deviation angle, and identifies a lateral deviation angle when the subject's face is facing more to the left than the front as a negative lateral deviation angle.

[0029] The estimation unit 133 estimates the ratio between the width of the iris and the width of the white of the eye when the subject faces forward by analyzing the first image acquired by the acquisition unit 131. The estimation unit 133 estimates the ratios corresponding to each of the left and right eyes of the subject by analyzing the first image acquired by the acquisition unit 131.

[0030] Specifically, first, estimation unit 133 identifies an image area of ​​the subject's eyes shown in each of the multiple first images by analyzing each of the multiple first images acquired by acquisition unit 131 when angle identification unit 132 identifies different lateral deviation angles. Estimation unit 133 identifies a ratio between the width of the subject's iris and the width of the subject's white eye shown in each of the multiple first images by analyzing the image area of ​​the subject's eyes shown in each of the multiple first images.

[0031] For example, the estimation unit 133 analyzes each of the images when the lateral deviation angle identified by the angle identification unit 132 is greater than or equal to -30° and less than -20°, greater than or equal to -20° and less than -10°, greater than or equal to -10° and less than 0°, greater than or equal to 0° and less than 10°, greater than or equal to 10° and less than 120°, and greater than or equal to 20° and less than 30°, thereby identifying the ratio between the width of the subject's pupil and the width of the white of the eye in each of these multiple images.

[0032] Then, the estimation unit 133 estimates the ratio of the width of the iris to the width of the white of the eye when the subject faces forward by averaging the ratio of the width of the iris to the width of the white of the eye specified in each of the multiple first images. The estimation unit 133 may specify the ratio of the width of the iris to the width of the white of the eye of the subject shown in the first image on the condition that the white of the eye is shown on both sides of the iris of the subject in the image area of ​​the subject's eyes shown in the first image. In this way, the information processing device 1 can accurately specify the width of the iris in the image area of ​​the subject's eyes. As a result, the information processing device 1 can accurately specify the width of the white of the eye based on the image area of ​​the eye and the specified width of the iris, and can improve the accuracy of specifying the ratio of the width of the iris to the width of the white of the eye.

[0033] The color identifying unit 134 identifies the color of the iris, the color of the left white of the eye, and the color of the right white of the eye of the subject by analyzing the first image acquired by the acquisition unit 131. For example, the color identifying unit 134 analyzes each of the multiple first images acquired by the acquisition unit 131 within a predetermined time, and identifies the color of the iris, the color of the left white of the eye, and the color of the right white of the eye of the subject that appear in each of the multiple first images. Then, the color identifying unit 134 identifies the average value of the colors of the iris identified from each of the multiple first images as the color of the iris of the left eye of the subject.

[0034] Similarly, the color specifying unit 134 specifies the average value of the color of the left white of the left eye specified from each of the multiple first images as the color of the left white of the left eye of the subject, and specifies the average value of the color of the right white of the left eye specified from each of the multiple first images as the color of the right white of the left eye of the subject. The color specifying unit 134 performs a similar process for the right eye of the subject, and specifies the color of the iris, the color of the left white of the eye, and the color of the right white of the right eye of the subject.

[0035] The iris analysis unit 135 analyzes the first image acquired by the acquisition unit 131 to determine the ratio between the width of the iris of the subject and the width of the pupil in the iris. For example, the iris analysis unit 135 analyzes the first image acquired by the acquisition unit 131 within a predetermined time, and determines the maximum width of the image area corresponding to the iris of the subject's left eye shown in the first image, and the maximum width of the image area of ​​the pupil included in the iris. The iris analysis unit 135 determines the ratio between the width of the iris of the subject's left eye and the width of the pupil based on the width of the image area corresponding to the identified iris and the width of the image area of ​​the pupil. The iris analysis unit 135 performs the same process on the subject's right eye, and determines the ratio between the width of the iris of the subject's right eye and the width of the pupil.

[0036] The iris analysis unit 135 analyzes the first image acquired by the acquisition unit 131 to identify the size of the light reflected in the iris and the position of the light in the iris. For example, the iris analysis unit 135 analyzes each of the multiple first images acquired by the acquisition unit 131 within a predetermined time, and identifies the position of the area in which the pixel value exceeds a predetermined threshold and the size of the area among the image areas corresponding to the iris of the subject's left eye shown in each of the multiple first images. The iris analysis unit 135 identifies the position of the light in the iris by, for example, identifying the relative position of the area corresponding to the light reflected in the iris with respect to the center position of the iris. The iris analysis unit 135 may identify an area whose size exceeds a predetermined size and whose pixel value exceeds a predetermined threshold as the area of ​​the light reflected in the iris. The iris analysis unit 135 performs the same process on the right eye of the subject, and identifies the size of the light reflected in the iris of the subject's right eye and the position of the light in the iris.

[0037] [Subject's Eye Correction] Next, a process of correcting the eyes of the subject appearing in the image acquired by the acquisition unit 131 will be described. The angle specifying unit 132, the size specifying unit 136, the width specifying unit 137, and the correction unit 138 of the control unit 13 cooperate to correct the eyes of the subject appearing in the second image acquired by the acquisition unit 131 after the analysis of the iris is completed. The specific process of correcting the eyes of the subject will be described below.

[0038] The angle identifying unit 132 identifies a horizontal deviation angle, which is an angle at which the face reflected in the second image is shifted in the horizontal direction when the front direction of the subject's face is used as a reference, by analyzing the second image acquired by the acquiring unit 131 after the analysis of the iris is completed. Also, the angle identifying unit 132 identifies a vertical deviation angle, which is an angle at which the face reflected in the second image is shifted in the vertical direction when the front direction of the subject's face is used as a reference, by analyzing the second image.

[0039] The size determination unit 136 determines the size of the eyeball and the size of the iris of the subject by acquiring size information indicating the size of the eyeball and the size of the iris of the subject. It is known that there is little individual difference in the size of a person's eyeball and the size of the iris. For this reason, size information indicating the size of an average person's eyeball and the size of the iris is stored in the storage unit 12, and the size determination unit 136 determines the size of the eyeball and the size of the iris of the average person indicated by the size information stored in the storage unit 12 as the size of the eyeball and the size of the iris of the subject. Here, the size of the eyeball indicates the radius of the eyeball, and the size of the iris indicates the maximum width of the iris.

[0040] Based on the identified eyeball size, pupil size, and lateral deviation angle and the estimated ratio, the width determination unit 137 determines the ratio between the pupil width, which is the width of the pupil of the subject when the subject looks straight ahead in the second image with the subject's face deviated by a lateral deviation angle with respect to the forward direction, the left white of the eye width, which indicates the width of the left white of the eye that is reflected to the left of the pupil, and the right white of the eye width, which indicates the width of the right white of the eye that is reflected to the right of the pupil.

[0041] The process of width determination unit 137 for determining the width of the iris, the width of the left white of the eye, and the width of the right white of the eye will be described below with reference to the drawings. FIG. 3 is a diagram for explaining the state of the eyeballs when the subject's face is shifted to the side. FIG. 3 shows the state of the eyeballs when the subject's head H is viewed from above. It can be seen that FIG. 3 shows the left eye LE and the right eye RE. In FIG. 3, the lateral shift angle at which the subject's face is shifted from the front direction is z.

[0042] FIG. 4 is an enlarged view of the subject's eyeball shown in FIG. 3. In FIG. 4, O indicates the center of the eyeball, and r indicates the radius of the eyeball. FIG. 4 also shows the iris width, left white of the eye width, and right white of the eye width that can be seen from the front direction, that is, the iris width, left white of the eye width, and right white of the eye width when the subject's eye is reflected in the second image. In the following description, an example of determining the ratio of the iris width, left white of the eye width, and right white of the eye width in the eyeball corresponding to the left eye will be described, but the same process will be performed on the right eye to determine the ratio of the iris width, left white of the eye width, and right white of the eye width in the eyeball corresponding to the right eye.

[0043] FIG. 5 is an enlarged view of a sector portion of the eyeball shown in FIG. 4, connecting the center of the eyeball, the left end of the left white of the eye width, and the right end of the right white of the eye width. WL in FIG. 5 indicates the left white of the eye width in real space, and WR indicates the right white of the eye width in real space. x indicates half the angle between the line connecting the left end of the iris width and the center O of the eyeball, and the line connecting the right end of the iris width and the center O of the eyeball. y indicates half the angle between the line connecting the left end of the left white of the eye width and the center O of the eyeball, and the line connecting the right end of the right white of the eye width and the center O of the eyeball. In the following description, x will also be referred to as the first angle, and y will also be referred to as the second angle.

[0044] α indicates half the angle made by a line connecting the left end of the left white of the eye width to the center O of the eyeball and a line connecting the right end of the left white of the eye width to the center O of the eyeball. β indicates half the angle made by a line connecting the left end of the right white of the eye width to the center of the eyeball and a line connecting the right end of the right white of the eye width to the center O of the eyeball.

[0045] The iris width is calculated by the following formula (1).

number

[0046] As shown in FIG. 5, 2α+x=yz, and α=(yzx) / 2, so the left white of the eye width WL in real space is calculated by the following formula (2).

number

[0047] As shown in FIG. 5, 2β+x=y+z, and β=(y+zx) / 2. Therefore, the right white width WR in real space is calculated by the following formula (3).

number

[0048] FIG. 6 is a diagram showing the relationship between the left white of the eye width in real space and the left white of the eye width in the second image, and the relationship between the right white of the eye width in real space and the right white of the eye width in the second image. FIG. 6(a) is a diagram showing the relationship between the left white of the eye width in real space and the left white of the eye width in the second image, and FIG. 6(b) is a diagram showing the relationship between the right white of the eye width in real space and the right white of the eye width in the second image. As shown in FIG. 6(a), the angle between the line connecting the center of the left white of the eye width WL and the center of the eyeball in real space and the line connecting the center of the iris width and the center of the eyeball is x+α. Therefore, the angle between the line extending the left white of the eye width WL in real space and the line extending the left white of the eye width is also x+α, and the left white of the eye width is expressed by the following formula (4).

number

[0049] As described above, since α=(yzx) / 2, the left white width is expressed by equation (5) based on equations (2) and (4).

number

[0050] As shown in Figure 6(b), the angle between the line connecting the center of WR to the center of the eyeball and the line connecting the center of the iris width to the center of the eyeball is x + β. Therefore, the angle between the line extending the right white of the eye width WR in real space and the line extending the right white of the eye width is also x + β, and the right white of the eye width is expressed by the following formula (6).

number

[0051] As described above, since β=(y+zx) / 2, the right white width is expressed by equation (7) based on equations (3) and (6).

number

[0052] First, the width specifying unit 137 substitutes the size of the eyeball and the size of the iris specified by the size specifying unit 136 into the formula (1) to calculate the first angle x. Note that when the size of the eyeball and the size of the iris specified by the size specifying unit 136 are fixed values, the first angle x is also a fixed value. Therefore, the first angle x may be stored in the storage unit 12, and the width specifying unit 137 may acquire the first angle x stored in the storage unit 12.

[0053] The width determination unit 137 determines the second angle y based on the first angle x, the lateral deviation angle z, the ratio of the width of the iris to the width of the white of the eye estimated by the estimation unit 133, and equations (1), (5), and (7). If the ratio of the width of the white of the eye to the width of the iris is R, the following equation (8) holds based on the ratio of the width of the iris to the width of the white of the eye estimated by the estimation unit 133 and equations (1), (5), and (7).

number

[0054] The width determination unit 137 substitutes the determined first angle x, lateral deviation angle z, eyeball radius r, and ratio R of the white of the eye width to the iris width into formula (8) to calculate the second angle y. The width determination unit 137 substitutes the determined first angle x, second angle y, lateral deviation angle z, and eyeball radius r into formula (5) to calculate the left white of the eye width. Similarly, the width determination unit 137 substitutes the determined first angle x, second angle y, lateral deviation angle z, and eyeball radius r into formula (7) to calculate the right white of the eye width. The width determination unit 137 calculates the ratio of the iris width to the left white of the eye width to the right white of the eye width based on the calculated iris width, left white of the eye width, and right white of the eye width.

[0055] Correction unit 138 corrects the eyes of the subject included in the second image acquired by acquisition unit 131 so that the shapes of the iris and white of the subject's eyes included in the second image acquired by acquisition unit 131 correspond to the ratio of the iris width, left white of the eye width, and right white of the eye width specified by width specification unit 137. For example, correction unit 138 determines the display area of ​​the left white of the left eye, the display area of ​​the iris, and the display area of ​​the right white of the left eye based on the ratio of the iris width, left white of the eye width, and right white of the eye width specified by width specification unit 137 in an image area showing the left eye of the subject included in the second image.

[0056] The correction unit 138 corrects the left eye of the subject by drawing the left white, the black eye, and the right white in the left white, the black eye, and the right white display areas determined for the left eye, respectively, based on the color of the black eye, the color of the left white, and the color of the right white of the left eye specified by the color specifying unit 134. The correction unit 138 also determines the left white, the black eye, and the right white display area for the right eye of the subject, and draws the left white, the black eye, and the right white. In this way, the information processing device 1 can correct the eyes of the subject so that the subject views the front.

[0057] Furthermore, when drawing the iris, the correction unit 138 draws the pupil at the center of the iris of the left eye based on the ratio between the width of the iris specified by the iris analysis unit 135 and the width of the pupil in the iris. Furthermore, the correction unit 138 draws light for the iris drawn in the left eye based on the size and position of the light specified by the iris analysis unit 135. In this way, the information processing device 1 can correct the subject's eyes in accordance with how the subject's eyes are reflected in the first image.

[0058] In addition, when the vertical deviation angle, which is the angle at which the face of the subject reflected in the second image is shifted in the up-down direction, is equal to or greater than a predetermined angle, correcting the subject's eyes may cause the subject's eyes to look unnatural. For this reason, the correction unit 138 may correct the subject's eyes reflected in the second image when the magnitude of the vertical deviation angle identified by the angle identification unit 132 is equal to or less than a predetermined angle. In this way, the information processing device 1 can prevent the subject's eyes from being corrected to look unnatural.

[0059] [Image Output] The output unit 139 outputs the image acquired by the acquisition unit 131 to the second terminal 3. Specifically, the output unit 139 outputs the first image acquired by the acquisition unit 131 as is to the second terminal 3. In addition, the output unit 139 outputs to the second terminal 3 a second image in which the eyes of the subject have been corrected by the correction unit 138 after the eyes of the subject have been analyzed based on the first image.

[0060] Fig. 7 is a diagram showing an example of correction of the subject's eyes. Fig. 7(a) is a diagram showing the second image before correction, and Fig. 7(b) is a diagram showing the second image after correction. It can be seen that the subject's line of sight shown in Fig. 7(a) is not looking straight ahead, whereas the subject's line of sight has been corrected to look straight ahead as shown in Fig. 7(b).

[0061] [Operation flow] Next, a description will be given of the flow of processing in the information processing device 1. FIG 7 is a flowchart showing the flow of processing in the information processing device 1.

[0062] First, the acquisition unit 131 acquires a first image (S1) of the subject's face captured by an imaging device provided in the first terminal 2. The first image is used to analyze the subject's eyes.

[0063] Next, the estimation unit 133 estimates the ratio between the width of the iris and the width of the white of the eye when the subject faces forward by analyzing the first image acquired by the acquisition unit 131 (S2). Next, the color identification unit 134 analyzes the first image acquired by the acquisition unit 131 to identify the color of the iris, the color of the left white of the eye, and the color of the right white of the eye of the subject (S3). Next, the iris analysis unit 135 analyzes the first image acquired by the acquisition unit 131 to identify the ratio between the width of the iris of the subject and the width of the pupil in the iris, and also identify the size and position of the light reflected in the iris (S4).

[0064] Next, the size determination unit 136 determines the size of the eyeball and the size of the iris of the subject by acquiring size information indicating the size of the eyeball and the size of the iris of the subject (S5). Note that the processes from S2 to S5 may be executed in parallel, or may be executed in an order different from that shown in FIG.

[0065] Next, the acquisition unit 131 acquires a second image of the subject's face captured by an imaging device provided in the first terminal 2 (S6). Next, the angle identifying unit 132 identifies the lateral deviation angle and the vertical deviation angle of the subject's face by analyzing the second image (S7).

[0066] The correction unit 138 determines whether the vertical deviation angle determined in S7 is less than a predetermined threshold value (S8). If the vertical deviation angle is less than the predetermined threshold value (YES in S8), the correction unit 138 proceeds to S9, and if the vertical deviation angle is equal to or greater than the predetermined threshold value (NO in S8), the correction unit 138 proceeds to S11.

[0067] In S9, the width determination unit 137 determines the ratio of the pupil width, the left white of the eye width, and the right white of the eye width when the subject looks straight ahead in the second image with the subject's face shifted by a lateral deviation angle based on the straight ahead direction, based on the determined eyeball size, pupil size, and lateral deviation angle, and the ratio of the estimated pupil width and white of the eye width.

[0068] Next, the correction unit 138 corrects the subject's eyes included in the second image acquired by the acquisition unit 131 (S10). Specifically, the correction unit 138 corrects the subject's eyes included in the second image so that the shapes of the iris and white of the eye in the second image correspond to the ratio of the iris width, left white of the eye width, and right white of the eye width specified in S9. The correction unit 138 also draws the iris, left white of the eye, and right white of the eye based on the color of the iris, the color of the left white of the eye, and the color of the right white of the eye specified in S3. When drawing the iris, the correction unit 138 draws the pupil at the center of the iris of the left eye based on the ratio of the iris width and the pupil width of the iris specified in S4. The correction unit 138 also draws light for the iris drawn in the left eye based on the size and position of the light specified in S4.

[0069] Next, the output unit 139 outputs the second image to the second terminal 3 (S11). If the eyes of the subject have been corrected in S10, the output unit 139 outputs the second image in which the eyes of the subject have been corrected to the second terminal 3, and if the eyes of the subject have not been corrected, the output unit 139 outputs the second image acquired in S6 to the second terminal 3 as is.

[0070] Next, the acquisition unit 131 determines whether or not a stop of the process shown in Fig. 7 has been received (S12). If the acquisition unit 131 determines that a stop of the process shown in Fig. 7 has been received (YES in S12), the acquisition unit 131 ends the process according to this flowchart, and if the acquisition unit 131 determines that a stop of the process has not been received (NO in S12), the acquisition unit 131 proceeds to S6.

[0071] [Effects of information processing device 1] As described above, the information processing device 1 according to the present embodiment estimates the ratio of the width of the iris to the width of the white of the eye when the subject faces forward by analyzing the first image, and specifies the lateral shift angle, which is the angle at which the face reflected in the second image is shifted in the lateral direction when the front direction of the face of the subject is used as a reference, by analyzing the second image acquired after acquiring the first image. The information processing device 1 specifies the ratio of the width of the iris, the width of the left white of the eye, and the width of the right white of the eye when the subject looks at the front direction in the second image in a state in which the face is shifted by the lateral shift angle with the front direction as a reference based on the specified eyeball size, the size of the iris, and the lateral shift angle, and the estimated ratio, and corrects the second image so that the shape of the iris and the white of the eye of the subject included in the second image corresponds to the specified ratio of the width of the iris, the width of the left white of the eye, and the width of the right white of the eye. In this way, the information processing device 1 can align the gaze of the participant with the gaze of another participant.

[0072] Furthermore, this invention will make it possible to contribute to Goal 9 of the United Nations' Sustainable Development Goals (SDGs), which is to "build resilient infrastructure, promote industry, innovation and infrastructure."

[0073] Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. For example, all or part of the device can be configured by distributing or integrating functionally or physically in any unit. In addition, new embodiments resulting from any combination of multiple embodiments are also included in the embodiments of the present invention. The effect of the new embodiment resulting from the combination combines the effect of the original embodiment. [Explanation of symbols]

[0074] 1. Information processing device 2. Terminal 1 3. Terminal 2 11 Communications Department 12 Storage section 13 Control section 131 Acquisition Department 132 Angle identification part 133 Estimation Department 134 Color identification section 135 Black Eye Analysis Department 136 Size Specific Part 137 Width specific part 138 Correction section 139 Output section

Claims

1. an acquisition unit that acquires a plurality of images of a face of a subject captured by an imaging device; a size determination unit that determines the size of the eyeball and the size of the iris by acquiring size information indicating the size of the eyeball and the size of the iris of the subject; an estimation unit that estimates a ratio of a width of the black eye to a width of the white eye when the subject faces forward by analyzing the first image acquired by the acquisition unit; an angle identification unit that identifies a lateral deviation angle, which is an angle at which the face shown in the second image is shifted in a lateral direction relative to a front direction of the face of the subject, by analyzing a second image acquired by the acquisition unit after acquiring the first image; a width specifying unit that specifies a ratio between a width of the pupil of the subject when the subject views the front direction in the second image with the face deviated by the lateral deviation angle with respect to the front direction in the second image, a left white eye width indicating the width of the left white of the eye that is the white of the eye reflected to the left of the pupil, and a right white eye width indicating the width of the right white of the eye that is the white of the eye reflected to the right of the pupil, based on the specified eyeball size, the size of the pupil, and the lateral deviation angle and the estimated ratio; a correction unit that corrects the second image so that the shapes of the iris and white of the subject's eyes included in the second image correspond to the specified ratio of the iris width, the left white of the eye width, and the right white of the eye width; an output unit that outputs the second image in which the eye of the subject has been corrected by the correction unit; An information processing device having the above configuration.

2. The estimation unit estimates the ratios corresponding to the left and right eyes of the subject by analyzing the first image acquired by the acquisition unit. The information processing device according to claim 1 .

3. The estimation unit analyzes each of the multiple first images acquired by the acquisition unit to identify a ratio between the width of the iris and the width of the white of the eye of the subject shown in each of the multiple first images, and estimates the ratio between the width of the iris and the width of the white of the eye when the subject faces forward by averaging the ratios identified for each of the multiple first images. The information processing device according to claim 1 .

4. the angle identification unit identifies the lateral deviation angle of the face shown in the first image with respect to a front direction of the face of the subject by analyzing the first image; The estimation unit estimates a ratio of a width of the iris to a width of the white of the eye of the subject captured in each of the first images by analyzing each of the first images when the angle identification unit identifies a different lateral deviation angle. The information processing device according to claim 3 .

5. The estimation unit estimates a ratio of a width of the iris and a width of the white of the eye of the subject shown in the first image, on the condition that the white of the eye is shown on both sides of the iris of the subject in the first image acquired by the acquisition unit.

5. The information processing device according to claim 3.

6. a color specifying unit that specifies a color of the black eye, a color of the left white eye, and a color of the right white eye by analyzing the first image, The correction unit corrects the color of the subject's eyes shown in the second image based on the color of the iris, the color of the left white of the eye, and the color of the right white of the eye identified by the color identification unit.

3. The information processing device according to claim 1 or 2.

7. The method further includes a iris analysis unit that analyzes the first image to identify a size of light reflected in the iris and a position of the light in the iris, The correction unit depicts the light with respect to the black eye based on the identified size and position of the light. The information processing device according to claim 1 .

8. Further comprising a iris analysis unit that identifies a ratio between a width of the iris and a width of a pupil in the iris by analyzing the first image, The correction unit depicts the pupil relative to the iris based on a ratio between the identified width of the iris and a width of the pupil at the iris. The information processing device according to claim 1 .

9. the angle identification unit identifies a vertical deviation angle, which is an angle at which the face shown in the second image is deviated in an up-down direction when a front direction of the face of the subject is used as a reference, by analyzing the second image; The correction unit corrects the eyes of the subject appearing in the second image when the magnitude of the vertical deviation angle is equal to or smaller than a predetermined angle. The information processing device according to claim 1 .

10. The computer executes acquiring a plurality of images of a face of a subject captured by an imaging device; determining a size of the eyeball and a size of the iris of the subject by obtaining size information indicative of the size of the eyeball and the size of the iris of the subject; estimating a ratio of a width of the iris to a width of the white of the eye when the subject is facing forward by analyzing the acquired first image; A step of identifying a lateral deviation angle, which is an angle at which the face shown in the second image is shifted laterally with respect to a front direction of the face of the subject by analyzing a second image acquired after the first image is acquired; a step of determining a ratio between a width of the subject's pupil, a left white eye width indicating the width of the left white eye, which is the white of the eye reflected to the left of the pupil, and a right white eye width indicating the width of the right white eye, which is the white of the eye reflected to the right of the pupil, when the subject views the front direction in the second image with the face shifted by the lateral shift angle with respect to the front direction; and correcting the second image so that the shape of the iris and white of the subject's eye included in the second image corresponds to the specified ratio of the iris width, the left white of the eye width, and the right white of the eye width; outputting the second image with the subject's eyes corrected; An information processing method comprising the steps of:

Images