Avatar Generator

Abstract

A server as this avatar generation system includes a first acquisition unit that acquires a first image of the front side of a user's face, a second acquisition unit that acquires a second image of the front side and lateral sides of the user's face in response to the motion of a user's head, a recognition unit that recognizes the user according to the first and second images, and an image generation unit that generates a head image of a three-dimensional avatar corresponding to the user using the second image.

Description

[Technical Field]

[0001] The present invention relates to an avatar generation device. [Background technology]

[0002] On the Internet, characters called avatars are sometimes used to represent users. In recent years, with the use of technologies such as 3D scanning, avatars that represent users in three-dimensional virtual spaces have become widely available.

[0003] For example, Patent Document 1 discloses a virtual dressing device that uses a three-dimensional image avatar. The virtual dressing device accepts a head image of a model, body shape information of the model, any number of pieces of clothing for the same body part, and the order in which the any number of pieces of clothing are worn. The virtual dressing device also extracts two-dimensional images corresponding to any number of pieces of clothing from an image information database that converts the state of wearing the multiple pieces of clothing into two-dimensional images. The virtual dressing device also generates a composite image by combining the two-dimensional images based on the order in which the multiple pieces of clothing are worn. The virtual dressing device also converts the composite image into a three-dimensional image based on the body shape information. The virtual dressing device also transmits the three-dimensional image and the head image to a terminal device. An example of the "head image" is a three-dimensional image 3D scanned by the user's terminal device. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2020 / 009066 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the technology disclosed in Patent Document 1, a user of the terminal device can impersonate another person by inputting a different head image into the virtual fitting device. If such impersonation occurs, for example, a malicious person may unfairly damage the reputation of the individual.

[0006] Therefore, the present invention aims to solve the problem of preventing impersonation by generating a three-dimensional avatar that can be verified as the user's identity. [Means for solving the problem]

[0007] An avatar generation device according to a preferred embodiment of the present invention includes a first acquisition unit that acquires a first image showing the front of a user's face, a second acquisition unit that acquires a second image showing the front and side of the user's face in accordance with the user's head movement, an authentication unit that authenticates the user based on the first image and the second image, and an image generation unit that uses the second image to generate a head image of a three-dimensional avatar corresponding to the user. [Effects of the Invention]

[0008] According to the present invention, it is possible to prevent impersonation by generating a three-dimensional avatar that has been verified as the user's identity. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a diagram showing the overall configuration of an information processing system 1 according to a first embodiment. [Figure 2] 1 is a perspective view showing the appearance of MR glasses 30 according to a first embodiment. [Figure 3] 1 is a schematic diagram of a virtual space VS provided to a user U1 by using MR glasses 30 according to the first embodiment. [Figure 4] 1 is a schematic diagram of a virtual space VS provided to a user U1 by using MR glasses 30 according to the first embodiment. [Figure 5]FIG. 2 is a block diagram showing an example of the configuration of MR glasses 30 according to the first embodiment. [Figure 6] FIG. 2 is a block diagram showing an example of the configuration of a terminal device 20 according to the first embodiment. [Figure 7] 10 is a flow chart showing a generation flow of a three-dimensional image WP of the entire avatar A1 according to the first embodiment. [Figure 8] FIG. 2 is a block diagram showing an example of the configuration of a server 10 according to the first embodiment. [Figure 9] 4 is a flowchart showing the operation of the server 10 according to the first embodiment. [Figure 10] FIG. 10 is a block diagram showing an example of the configuration of a server 10A according to a second embodiment. [Figure 11] FIG. 11 is a block diagram showing an example of the configuration of a server 10B according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] 1: First embodiment Hereinafter, with reference to FIGS. 1 to 9, a configuration of an information processing system 1 including a server 10 as an avatar generation device according to a first embodiment of the present invention will be described.

[0011] 1-1: Configuration of the first embodiment 1-1-1: Overall structure FIG. 1 is a diagram showing the overall configuration of an information processing system 1 according to a first embodiment of the present invention. The information processing system 1 uses MR technology to provide a virtual space to users U1 and U2 wearing MR glasses 30. In particular, in this embodiment, the information processing system 1 displays an avatar A1 corresponding to user U1 and an avatar A2 corresponding to user U2 on the MR glasses 30. Note that MR technology is a technology that precisely overlays a digital virtual space on real space using XR glasses such as the MR glasses 30 or a device such as an HMD (Head Mounted Display). The MR technology allows users U1 and U2 to simultaneously experience a real space in which real objects are placed and a virtual space in which virtual objects are placed.

[0012] The information processing system 1 includes a server 10, a terminal device 20, and MR glasses 30. The server 10 is an example of an avatar generation device. In the information processing system 1, the server 10 and the terminal device 20 are connected to each other so as to be able to communicate with each other via a communication network NET. Furthermore, the terminal device 20 and the MR glasses 30 are connected to each other so as to be able to communicate with each other. Note that in FIG. 1, two pairs of terminal devices 20 and MR glasses 30 are shown: a pair of terminal device 20-1 and MR glasses 30-1, and a pair of terminal device 20-2 and MR glasses 30-2. However, this number of pairs is merely an example, and the information processing system 1 can include any number of pairs of terminal devices 20 and MR glasses 30. Note that in FIG. 1, it is assumed that a user U1 uses a pair of terminal device 20-1 and MR glasses 30-1, and a user U2 uses a pair of terminal device 20-2 and MR glasses 30-2.

[0013] The server 10 provides various data and cloud services to the terminal device 20 via the communication network NET. In particular, the server 10 provides the terminal device 20 with various data for displaying an avatar A1 corresponding to user U1 and an avatar A2 corresponding to user U2 on the MR glasses 30 connected to the terminal device 20. More specifically, the server 10 provides the terminal device 20-1 with various data for displaying the avatar A2 on a display 38-1 provided in the MR glasses 30-1 used by the user U1. The server 10 also provides the terminal device 20-2 with various data for displaying the avatar A1 on a display 38-2 provided in the MR glasses 30-2 used by the user U2. In this embodiment, the avatar A1 is a real avatar generated using a real-life image of the user U1. Similarly, the avatar A2 is a real avatar generated using a real-life image of the user U2.

[0014] The terminal device 20-1 displays virtual objects arranged in a virtual space on the MR glasses 30-1 worn by the user U1. The terminal device 20-2 displays virtual objects arranged in the virtual space on the MR glasses 30-2 worn by the user U2. The virtual space is, for example, a celestial sphere. Examples of the virtual objects include virtual objects representing data such as still images, videos, 3DCG models, HTML files, and text files, as well as virtual objects representing applications. Examples of text files include memos, source code, diaries, and recipes. Examples of applications include browsers, applications for using SNS, and applications for generating document files. The terminal device 20-1 is preferably, for example, a mobile terminal device such as a smartphone or a tablet.

[0015] In particular, in this embodiment, the terminal device 20-1 causes the MR glasses 30-1 to display a virtual object corresponding to the avatar A2, and the terminal device 20-2 causes the MR glasses 30-2 to display a virtual object corresponding to the avatar A1.

[0016] The MR glasses 30 are display devices worn on the heads of the user U1 and the user U2. More specifically, the MR glasses 30-1 are display devices worn on the head of the user U1. The MR glasses 30-2 are display devices worn on the head of the user U2. The MR glasses 30 are see-through wearable displays. The MR glasses 30 are controlled by the terminal device 20 to display virtual objects on display panels provided corresponding to the lenses for both eyes. The MR glasses 30 are an example of a display device.

[0017] 1-1-2: Composition of MR glasses Fig. 2 is a perspective view showing the appearance of the MR glasses 30. As shown in Fig. 2, the appearance of the MR glasses 30 is similar to that of ordinary eyeglasses, and includes temples 91 and 92, a bridge 93, frames 94 and 95, and lenses 41L and 41R. An imaging device 36 is provided on the bridge 93. The imaging device 36 captures an image of the outside world. The imaging device 36 also outputs imaging information indicating the captured image.

[0018] Each of the lenses 41L and 41R is provided with a half mirror. A liquid crystal panel or organic EL panel for the left eye is provided in the frame 94. The liquid crystal panel or organic EL panel will hereinafter be collectively referred to as a display panel. The frame 94 is also provided with an optical member that guides light emitted from the display panel for the left eye to the lens 41L. The half mirror provided in the lens 41L transmits external light to guide it to the left eye and reflects the light guided by the optical member to make it incident on the left eye. The frame 95 is provided with a display panel for the right eye and an optical member that guides light emitted from the display panel for the right eye to the lens 41R. The half mirror provided in the lens 41R transmits external light to guide it to the right eye and reflects the light guided by the optical member to make it incident on the right eye.

[0019] The display 38, which will be described later, includes a lens 41L, a display panel for the left eye, and an optical member for the left eye, as well as a lens 41R, a display panel for the right eye, and an optical member for the right eye.

[0020] In the above configuration, users U1 and U2 can observe the images displayed on the display panels in a see-through state where the images are superimposed on the outside world. Furthermore, the MR glasses 30 display the left-eye image on the left-eye display panel and the right-eye image on the right-eye display panel, among the binocular images with parallax, so that users U1 and U2 perceive the displayed images as if they have depth and a three-dimensional effect.

[0021] 3 and 4 are schematic diagrams of a virtual space VS provided to users U1 and U2 by using MR glasses 30. As shown in FIG. 3, virtual objects VO1 to VO5 representing various contents such as a browser, cloud services, images, and videos are arranged in the virtual space VS. By moving around a public space while wearing the MR glasses 30 displaying the virtual objects VO1 to VO5 arranged in the virtual space VS, user U1 can experience the virtual space VS as a private space in the public space. Consequently, user U1 can act in the public space while receiving the benefits provided by the virtual objects VO1 to VO5 arranged in the virtual space VS. The same applies to user U2.

[0022] 4, it is also possible for users U1 and U2 to share a virtual space VS. By users U1 and U2 sharing the virtual space VS, users U1 and U2 can share one or more virtual objects VO, and can communicate with each other via the shared virtual objects VO.

[0023] FIG. 5 is a block diagram showing an example of the configuration of the MR glasses 30. The MR glasses 30 include a processing device 31, a storage device 32, a gaze detection device 33, a GPS device 34, a motion detection device 35, an imaging device 36, a communication device 37, and a display 38. The elements of the MR glasses 30 are connected to each other by one or more buses for communicating information. Note that the term "device" in this specification may be replaced with other terms such as circuit, device, or unit. In the following description, a case where the MR glasses 30 are used by a user U1, i.e., a case where the MR glasses 30 are MR glasses 30-1, will be described.

[0024] The processing device 31 is a processor that controls the entire MR glasses 30. The processing device 31 is configured using, for example, one or more chips. The processing device 31 is also configured using, for example, a central processing unit (CPU) including an interface with peripheral devices, an arithmetic unit, a register, etc. Some or all of the functions of the processing device 31 may be realized by hardware such as a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), and a field programmable gate array (FPGA). The processing device 31 executes various processes in parallel or sequentially.

[0025] The storage device 32 is a recording medium that can be read from and written to by the processing device 31. The storage device 32 also stores a plurality of programs including a control program PR1 that the processing device 31 executes.

[0026] The gaze detection device 33 detects the gaze of the user U1. Any method may be used for detecting the gaze by the gaze detection device 33. For example, the gaze detection device 33 may detect gaze information based on the position of the inner corner of the eye and the position of the iris. Furthermore, the gaze detection device 33 supplies gaze information indicating the direction of the gaze of the user U1 based on the detection result to the processing device 31 described below. The gaze information supplied to the processing device 31 is transmitted to the terminal device 20 via the communication device 37.

[0027] The GPS device 34 receives radio waves from multiple satellites. The GPS device 34 also generates location information from the received radio waves. The location information indicates the location of the MR glasses 30. The location information may be in any format as long as it can identify the location. The location information indicates, for example, the latitude and longitude of the MR glasses 30. As an example, the location information is obtained from the GPS device 34. However, the MR glasses 30 may obtain the location information by any method. The obtained location information is supplied to the processing device 31. The location information supplied to the processing device 31 is transmitted to the terminal device 20 via the communication device 37.

[0028] The motion detection device 35 detects the motion of the MR glasses 30. Examples of the motion detection device 35 include inertial sensors such as an acceleration sensor that detects acceleration and a gyro sensor that detects angular acceleration. The acceleration sensor detects acceleration on orthogonal X-, Y-, and Z-axes. The gyro sensor detects angular acceleration around the X-, Y-, and Z-axes as central axes of rotation. The motion detection device 35 can generate motion information indicating the motion of the MR glasses 30 based on the output information of the gyro sensor. The motion information includes acceleration data indicating the acceleration on each of the three axes and angular acceleration data indicating the angular acceleration on each of the three axes. The motion detection device 35 also supplies the motion information related to the motion of the MR glasses 30 to the processing device 31. The motion information supplied to the processing device 31 is transmitted to the terminal device 20 via the communication device 37.

[0029] The imaging device 36 outputs imaging information obtained by capturing an image of the outside world. The imaging device 36 also includes, for example, a lens, an imaging element, an amplifier, and an AD converter. Light collected through the lens is converted into an imaging signal, which is an analog signal, by the imaging element. The amplifier amplifies the imaging signal and supplies it to the AD converter. The AD converter converts the amplified imaging signal, which is an analog signal, into imaging information, which is a digital signal. The converted imaging information is supplied to the processing device 21. The imaging information supplied to the processing device 31 is transmitted to the terminal device 20 via the communication device 37.

[0030] The communication device 37 is hardware serving as a transmitting / receiving device for communicating with other devices. The communication device 37 is also called, for example, a network device, a network controller, a network card, a communication module, etc. The communication device 37 may include a connector for wired connection and an interface circuit corresponding to the connector. The communication device 37 may also include a wireless communication interface. Examples of the connector and interface circuit for wired connection include products that comply with wired LAN, IEEE1394, and USB. Examples of the wireless communication interface include products that comply with wireless LAN, Bluetooth (registered trademark), etc.

[0031] The display 38 is a device that displays images. The display 38 displays various images under the control of the processing device 31. As described above, the display 38 includes the lens 41L, a display panel for the left eye, and an optical member for the left eye, as well as the lens 41R, a display panel for the right eye, and an optical member for the right eye. As the display panel, various display panels such as a liquid crystal display panel and an organic EL display panel are suitably used.

[0032] The processing device 31 functions as an acquisition unit 311 and a display control unit 312, for example, by reading out a control program PR1 from the storage device 32 and executing it.

[0033] The acquisition unit 311 acquires image information indicating an image to be displayed on the MR glasses 30 from the terminal device 20.

[0034] Furthermore, the acquisition unit 311 acquires gaze information supplied from the gaze detection device 33, position information supplied from the GPS device 34, movement information supplied from the movement detection device 35, and imaging information supplied from the imaging device 36. Then, the acquisition unit 311 supplies the acquired gaze information, position information, movement information, and imaging information to the communication device 37. The gaze information, position information, movement information, and imaging information supplied to the communication device 37 are transmitted to the terminal device 20.

[0035] Based on the image information acquired by the acquisition unit 311 from the terminal device 20, the display control unit 312 causes the display 38 to display an image indicated by the image information.

[0036] 1-1-3: Terminal device configuration 6 is a block diagram showing an example configuration of the terminal device 20. The terminal device 20 includes a processing device 21, a storage device 22, a communication device 23, a display 24, an input device 25, and an imaging device 26. The elements of the terminal device 20 are connected to each other by one or more buses for communicating information.

[0037] The processing device 21 is a processor that controls the entire terminal device 20. The processing device 21 is configured, for example, using one or more chips. The processing device 21 is configured, for example, using a central processing unit (CPU) including an interface with peripheral devices, an arithmetic unit, a register, etc. Some or all of the functions of the processing device 21 may be realized by hardware such as a DSP, an ASIC, a PLD, and an FPGA. The processing device 21 executes various processes in parallel or sequentially.

[0038] The storage device 22 is a recording medium that can be read from and written to by the processing device 21. The storage device 22 also stores a plurality of programs including a control program PR2 that the processing device 21 executes.

[0039] The communication device 23 is hardware serving as a transmitting / receiving device for communicating with other devices. The communication device 23 is also called, for example, a network device, a network controller, a network card, or a communication module. The communication device 23 may include a connector for wired connection and an interface circuit corresponding to the connector. The communication device 23 may also include a wireless communication interface. Examples of the connector and interface circuit for wired connection include products that comply with wired LAN, IEEE1394, and USB. Examples of the wireless communication interface include products that comply with wireless LAN, Bluetooth (registered trademark), etc.

[0040] The display 24 is a device that displays images and text information. The display 24 displays various images under the control of the processing device 21. For example, various display panels such as a liquid crystal display panel and an organic EL (Electro Luminescence) display panel are suitably used as the display 24.

[0041] In particular, in this embodiment, when the server 10 authenticates the user U1, the display 24 displays images and text information instructing the user U1 to move his or her head.

[0042] The input device 25 receives operations from a user U1 wearing the MR glasses 30 on their head. For example, the input device 25 includes a keyboard, a touchpad, a touch panel, or a pointing device such as a mouse. Here, if the input device 25 includes a touch panel, it may also serve as the display 24.

[0043] In this embodiment, user U1 uploads a first image TP1 showing the front of the user U1's face from the terminal device 20 to the server 10 for the purpose of generating a three-dimensional realistic avatar. The first image TP1 is typically a two-dimensional image generated based on a facial photograph of user U1. However, the first image TP1 is not limited to a two-dimensional image generated based on a facial photograph of user U1. FIG. 7 shows a flow for generating a three-dimensional image WP of the entire avatar A1, which is a realistic avatar corresponding to user U1. As shown in FIG. 7, the first image TP1 is used for the head image HP of the avatar A1. During uploading, the input device 25 is used by user U1 to input the first image TP1 to the terminal device 20. The first image TP1 may be obtained by capturing an image of user U1 using the imaging device 26 (described later) or may be acquired from an external device using the communication device 23.

[0044] The imaging device 26 outputs imaging information obtained by capturing an image of the outside world. The imaging device 26 also includes, for example, a lens, an imaging element, an amplifier, and an AD converter. Light collected through the lens is converted into an imaging signal, which is an analog signal, by the imaging element. The amplifier amplifies the imaging signal and outputs it to the AD converter. The AD converter converts the amplified imaging signal, which is an analog signal, into imaging information, which is a digital signal. The converted imaging information is output to the processing device 21. The imaging information output to the processing device 21 is output to the server 10 via the communication device 23.

[0045] In this embodiment, user U1 needs to be authenticated by the server 10 when generating the real avatar. During this authentication, user U1 moves his / her head based on images and text information displayed on the display 24 that instructs user U1 to move his / her head. The imaging device 26 captures an image of user U1's head while moving his / her head. It is preferable that imaging device 26 generate a video capturing the head movement of user U1.

[0046] The processing device 21 functions as an acquisition unit 211, an image generation unit 212, and an output unit 213 by reading and executing the control program PR2 from the storage device 22.

[0047] The acquisition unit 211 acquires a first image TP1 showing the front part of the face of the user U1. The acquisition unit 211 also acquires image information showing an image to be displayed on the MR glasses 30 from the server 10 by using the communication device 23. Furthermore, the acquisition unit 211 acquires line of sight information, position information, movement information, and imaging information from the MR glasses 30 using the communication device 23.

[0048] The image generation unit 212 generates a second image FP showing the front and side of the face of the user U1 based on an image of the user U1's head captured by the imaging device 26 while the head is moving. The second image FP is typically a three-dimensional image. However, the second image FP is not limited to such a three-dimensional image. For example, the second image FP may be a plurality of two-dimensional images, each showing the front or side of the face of the user U1. As shown in FIG. 7, the second image FP is used for a head image HP of avatar A1, which is a real avatar corresponding to the user U1.

[0049] The output unit 213 transmits the image information, gaze information, position information, movement information, and imaging information indicating the first image TP1 showing the front part of the face of the user U1, acquired by the acquisition unit 211, to the server 10. The output unit 213 also transmits the second image FP showing the front part and side part of the face of the user U1, generated by the image generation unit 212, to the server 10.

[0050] Furthermore, the output unit 213 transmits the image information acquired by the acquisition unit 211 to the MR glasses 30 for the purpose of displaying the virtual object VO in the virtual space VS based on the image information. The image information is image information that indicates an image of the virtual object VO. In detail, the output unit 213 transmits the image information to the MR glasses 30 so that the virtual object VO is displayed in the virtual space VS viewed by the user U1 through the MR glasses 30.

[0051] 1-1-4: Server configuration 8 is a block diagram showing an example of the configuration of the server 10. The server 10 includes a processing device 11, a storage device 12, a communication device 13, a display 14, and an input device 15. The elements of the server 10 are connected to each other by one or more buses for communicating information.

[0052] The processing device 11 is a processor that controls the entire server 10. The processing device 11 is configured, for example, using one or more chips. The processing device 11 is configured, for example, using a central processing unit (CPU) that includes an interface with peripheral devices, an arithmetic unit, a register, etc. Some or all of the functions of the processing device 11 may be realized by hardware such as a DSP, an ASIC, a PLD, and an FPGA. The processing device 11 executes various processes in parallel or sequentially.

[0053] The storage device 12 is a recording medium that can be read and written by the processing device 11. The storage device 12 also stores a plurality of programs including a control program PR3 executed by the processing device 11. The storage device 12 also stores avatar information AI and instruction information DI. The avatar information AI is information used by the image generation unit 114 (described later) when generating image information showing a body image BP of the avatar A1. The instruction information DI is information instructing the head movement of the user U1, and is displayed on the MR glasses 30.

[0054] The communication device 13 is hardware serving as a transmitting / receiving device for communicating with other devices. The communication device 13 is also called, for example, a network device, a network controller, a network card, or a communication module. The communication device 13 may include a connector for wired connection and an interface circuit corresponding to the connector. The communication device 13 may also include a wireless communication interface. Examples of the connector and interface circuit for wired connection include products that comply with wired LAN, IEEE1394, and USB. Examples of the wireless communication interface include products that comply with wireless LAN, Bluetooth (registered trademark), etc.

[0055] The display 14 is a device that displays images and text information. The display 14 displays various images under the control of the processing device 11. For example, various display panels such as a liquid crystal display panel and an organic EL display panel are suitably used as the display 14.

[0056] The input device 15 is a device that receives operations from an administrator of the information processing system 1. For example, the input device 15 includes a keyboard, a touchpad, a touch panel, or a pointing device such as a mouse. Here, if the input device 15 includes a touch panel, it may also serve as the display 14.

[0057] The processing device 11 functions as a first acquisition unit 111, a second acquisition unit 112, an authentication unit 113, an image generation unit 114, and an output unit 115, for example, by reading and executing a control program PR3 from the storage device 12.

[0058] The first acquisition unit 111 acquires a first image TP1 showing the front part of the face of the user U1 from the terminal device 20. More specifically, the first acquisition unit 111 receives the first image TP1 showing the front part of the face of the user U1 from the terminal device 20 using the communication device 23. The first image TP1 is the first image TP1 showing the front part of the face of the user U1 that was input to the terminal device 20 by the user U1 using the input device 25, as described above.

[0059] The second acquisition unit 112 acquires a second image FP showing the front and side of the face of the user U1 from the terminal device 20. More specifically, the second acquisition unit 112 receives the second image FP showing the front and side of the face of the user U1 from the terminal device 20 using the communication device 23. As described above, the second image FP is the second image FP showing the front and side of the face of the user U1 that is generated by the image generation unit 212 based on an image captured by the imaging device 26 included in the terminal device 20 when the user U1 makes a head movement.

[0060] The authentication unit 113 authenticates the user U1 based on the first image TP1 acquired by the first acquisition unit 111 and the second image FP acquired by the second acquisition unit 112. For example, the authentication unit 113 generates a third image TP2 showing the front of the user U1's face based on the second image FP and authenticates the user U1 by comparing the first image TP1 with the third image TP2. For this comparison, the authentication unit 113 preferably uses pattern matching technology, for example. Specifically, the authentication unit 113 compares feature data indicating the facial features of the user U1 extracted from the image information showing the first image TP1 with feature data indicating the facial features of the user U1 extracted from the image information showing the third image TP2. If the degree of match between the two sets of feature data is equal to or greater than a predetermined threshold, the authentication unit 113 authenticates the user U1 as authentic.

[0061] The image generation unit 114 generates image information indicating an image to be displayed on the MR glasses 30. The image information is transmitted to the terminal device 20 by the communication device 13. The output unit 213 included in the terminal device 20 outputs the image information to the MR glasses 30 based on the image information, with the aim of displaying the virtual object VO in the virtual space VS, as described above.

[0062] In particular, in this embodiment, when the authentication unit 113 authenticates the user U1 as authentic, the image generation unit 114 generates image information showing a three-dimensional image WP of the entire avatar A1 corresponding to the user U1.

[0063] More specifically, the image generation unit 114 generates image information representing the head image HP of the avatar A1 using the second image FP acquired by the second acquisition unit 112. The image generation unit 114 may also generate image information representing the head image HP of the avatar A1 using the first image TP1 acquired by the first acquisition unit 111. The image generation unit 114 generates image information representing the head image HP of the avatar A1 using not only an image representing the front of the user U1's face but also an image representing the side of the user U1's face included in the second image FP. With this configuration, the server 10 can make the head image HP of the avatar A1 more similar to the face of the user U1 than when generating a 3D avatar A1 using only an image representing the front of the user U1's face. This improves the quality of the head image HP of the avatar A1.

[0064] Furthermore, the image generation unit 114 generates image information representing a body image BP of the avatar A1 using the avatar information AI stored in the storage device 12. Finally, the image generation unit 114 generates image information representing a 3D image WP of the entire avatar A1 using the image information representing the head image HP and the image information representing the body image BP, as shown in Fig. 7 .

[0065] The output unit 115 transmits the instruction information DI stored in the storage device 12 to the terminal device 20 using the communication device 13. The instruction information DI is information instructing the user U1 to move his / her head, and is displayed on the MR glasses 30. The output unit 115 also transmits image information indicating the entire three-dimensional image WP of the avatar A1 generated by the image generation unit 114 to the terminal device 20 using the communication device 13.

[0066] 1-2: Operation of the first embodiment 9 is a flowchart showing the operation of the server 10 according to the first embodiment. The operation of the server 10 will be described below with reference to FIG.

[0067] In step S1, the processing device 11 functions as a first acquisition unit 111. The processing device 11 acquires, from the terminal device 20, a first image TP1 showing the front part of the face of the user U1.

[0068] In step S2, the processing device 11 functions as the output unit 115. The processing device 11 transmits instruction information DI to the terminal device 20 using the communication device 13. The instruction information DI is information instructing the user U1 to move the head, and is displayed on the MR glasses 30.

[0069] In step S3, the processing device 11 functions as the second acquisition unit 112. The processing device 11 acquires, from the terminal device 20, a second image FP showing the front and side of the face of the user U1.

[0070] In step S4, the processing device 11 functions as the authentication unit 113. The processing device 11 authenticates the user U1 based on the first image TP1 acquired by the first acquisition unit 111 and the second image FP acquired by the second acquisition unit 112. For example, the authentication unit 113 generates a third image TP2 showing the front of the face of the user U1 based on the second image FP, and authenticates the user U1 by comparing the first image TP1 with the third image TP2. If the user U1 is authenticated as authentic, i.e., if the result of the authentication in step S4 is positive, the processing device 11 executes the process of step S5. If the user U1 is not authenticated as authentic, i.e., if the result of the authentication in step S4 is negative, the processing device 11 executes the process of step S1.

[0071] In step S5, the processing device 11 functions as the image generating unit 114. The processing device 11 generates image information indicating a head image HP of the avatar A1 using the second image FP acquired in step S3.

[0072] In step S6, the processing device 11 functions as the image generating unit 114. The processing device 11 uses the avatar information AI stored in the storage device 12 to generate image information indicating a body part image BP of the avatar A1.

[0073] In step S7, the processing device 11 functions as the image generation unit 114. The processing device 11 generates image information representing a three-dimensional image WP of the entire avatar A1, using image information representing the head image HP and image information representing the body image BP.

[0074] In step S8, the processing device 11 functions as the output unit 115. The processing device 11 outputs image information indicating the entire three-dimensional image WP of the avatar A1 generated in step S7 to the terminal device 20 using the communication device 13. Thereafter, the processing device 11 ends all of the processes described in FIG.

[0075] 1-3: Effects of the First Embodiment According to the above description, the server 10 serving as an avatar generation device includes a first acquisition unit 111, a second acquisition unit 112, an authentication unit 113, and an image generation unit 114. The first acquisition unit 111 acquires a first image TP1 showing the front of the face of the user U1. The second acquisition unit 112 acquires a second image FP showing the front and side of the face of the user U1 in accordance with the movement of the user U1's head. The authentication unit 113 authenticates the user U1 based on the first image TP1 and the second image FP. The image generation unit 114 uses the second image FP to generate a head image HP of a three-dimensional avatar corresponding to the user U1.

[0076] By using the above configuration, the server 10 generates a three-dimensional avatar that is verified as the user U1 has identified him / her. This process allows the server 10 to prevent impersonation. The server 10 can also improve the quality of the three-dimensional avatar by generating the three-dimensional avatar using a second image FP showing the face of the user U1, which is acquired in conjunction with the movement of the user U1's head. Specifically, the server 10 can acquire images of not only the front face of the user U1 but also the side face of the user U1 as a result of the movement of the user U1's head. The server 10 generates the head image HP of the three-dimensional avatar using not only the image of the front face but also the image of the side face. This process allows the server 10 to improve the quality of the three-dimensional avatar.

[0077] Furthermore, according to the above description, the authentication unit 113 authenticates the user U1 by comparing the first image TP1 with a third image TP2 showing the front part of the face of the user U1, which is generated from the second image FP.

[0078] By using the above configuration, the server 10 can authenticate the user U1 using, for example, pattern matching technology. Through this process, the server 10 can generate a three-dimensional image WP of the avatar A1 corresponding to the user U1 who has been authenticated as legitimate. As a result, the server 10 can prevent other users U from impersonating the user U1.

[0079] As described above, the server 10 generates the head image HP of the 3D avatar using not only the image of the front face but also the image of the side face. When the server 10 performs a process to verify the identity of the user U1 during the generation of the head image HP, one possible method is to verify the identity of the user U1 using both the image of the front face and the image of the side face. However, in this embodiment, the server 10 only targets the image of the front face for identity verification. When acquiring the image of the side face, the server 10 does not perform a process to verify the identity of the user U1 using the image of the side face itself, but instead performs identity verification based on the degree to which the head movement of the user U1 matches the movement indicated by the instruction information DI. Therefore, the process for identity verification in this embodiment imposes a smaller load on the server 10 than a process that performs identity verification using both the image of the front face and the image of the side face.

[0080] Furthermore, according to the above explanation, the second acquisition unit 112 acquires the second image FP based on a moving image capturing the movement of the head of the user U1.

[0081] By using the above configuration, the server 10 can use a higher-quality image as the side image of the face of the user U1 compared to when using multiple still images capturing the head movements of the user U1. Through this process, the server 10 can generate a higher-quality head image HP of the avatar A1, which is a three-dimensional avatar.

[0082] 2: Second embodiment The configuration of an information processing system 1A including a server 10A as an avatar generation device according to a second embodiment of the present invention will be described below with reference to Fig. 10. In the following description, for the purpose of simplifying the description, the same components of the information processing system 1A according to the second embodiment as those of the information processing system 1 according to the first embodiment will be designated by the same reference numerals, and their description may be omitted.

[0083] 2-1: Configuration of the second embodiment 2-1-1: Overall structure An information processing system 1A according to the second embodiment of the present invention differs from the information processing system 1 according to the first embodiment in that it includes a server 10A instead of the server 10. In other respects, the overall configuration of the information processing system 1A is the same as the overall configuration of the information processing system 1 according to the first embodiment shown in Fig. 1, and therefore illustration and description thereof will be omitted.

[0084] 2-1-2: Server configuration 10 is a block diagram showing an example of the configuration of the server 10A. The server 10A differs from the server 10 in that it includes a processing device 11A instead of the processing device 11 and a storage device 12A instead of the storage device 12.

[0085] The storage device 12A stores a control program PR3, avatar information AI, instruction information DI, and a learning model LM.

[0086] The learning model LM is a learning model used by the authentication unit 113A described below to authenticate user U1 based on the second image FP showing the front and side of user U1's face acquired by the second acquisition unit 112.

[0087] The learning model LM is generated by learning training data in a learning phase. The training data used to generate the learning model LM includes a plurality of pairs of first feature information extracted from a first image TP1 showing the faces of a plurality of persons acquired by a first acquisition unit 111, and an authentication result for each of the plurality of persons.

[0088] The learning model LM is also generated outside the server 10. In particular, it is preferable that the learning model LM is generated in a second server (not shown). In this case, the server 10 acquires the learning model LM from the second server (not shown) via the communication network NET.

[0089] The processing device 11A includes an authentication unit 113A instead of the authentication unit 113 included in the processing device 11.

[0090] The authentication unit 113A authenticates the user U1 by inputting second feature information indicating features extracted from the second image FP showing the front and side of the face of the user U1, acquired from the second acquisition unit 112, into the learning model LM.

[0091] 2-2: Operation of the second embodiment Unlike the server 10 according to the first embodiment, the processing device 11A included in the server 10A according to the second embodiment functions as an authentication unit 113A in step S4. The processing device 11A authenticates the user U1 by inputting, into the learning model LM, second feature information indicating features extracted from the second image FP showing the front and side of the face of the user U1, acquired from the second acquisition unit 112. In other steps, the processing executed by the server 10A is the same as the processing executed by the server 10, and therefore a flowchart showing the operation of the server 10A is not shown.

[0092] 2-3: Effects of the second embodiment According to the above description, in the server 10A as an avatar generation device, the authentication unit 113A authenticates the user U1 by inputting the second feature information indicating the features extracted from the second image FP into a learning model LM that has learned the relationship between the first feature information indicating the features extracted from the first image TP1 showing the faces of each of the multiple persons and the authentication results of the multiple persons.

[0093] By using the above configuration, the server 10A can authenticate the user U1 using machine learning. Through this process, the server 10A can generate a three-dimensional image WP of the avatar A1 corresponding to the user U1 who has been authenticated as legitimate. As a result, the server 10A can prevent other users U from impersonating the user U1.

[0094] 3: Third embodiment The configuration of an information processing system 1B including a server 10B as an avatar generation device according to a third embodiment of the present invention will be described below with reference to Fig. 11. In the following description, for the purpose of simplifying the description, the same components of the information processing system 1B according to the third embodiment as those of the information processing system 1 according to the first embodiment will be designated by the same reference numerals, and their description may be omitted.

[0095] 3-1: Configuration of the third embodiment 3-1-1: Overall structure An information processing system 1B according to the third embodiment of the present invention differs from the information processing system 1 according to the first embodiment in that it includes a server 10B instead of the server 10. In other respects, the overall configuration of the information processing system 1B is the same as the overall configuration of the information processing system 1 according to the first embodiment shown in Fig. 1, and therefore illustration and description thereof will be omitted.

[0096] 3-1-2: Server configuration 11 is a block diagram showing an example of the configuration of the server 10B. The server 10B differs from the server 10 in that it includes a processing device 11B instead of the processing device 11.

[0097] The processing device 11B includes an authentication unit 113B instead of the authentication unit 113 included in the processing device 11. The processing device 11B further includes a third acquisition unit 116 and a determination unit 117.

[0098] The third acquisition unit 116 acquires the above-mentioned movement information related to the movement of the MR glasses 30 from the terminal device 20 using the communication device 13. Furthermore, the third acquisition unit 116 calculates motion information related to the movement of the head of the user U1 based on the acquired movement information, and outputs the motion information to the determination unit 117.

[0099] The judgment unit 117 judges whether the value indicating the degree of agreement between the movement indicated by the instruction information DI instructing the head movement of the user U1, which is output to the terminal device 20 by the output unit 115, and the head movement of the user U1 indicated by the movement information is greater than or equal to a predetermined value.

[0100] Similar to the authentication method according to the first embodiment, the authentication unit 113B compares, for example, feature data indicating the facial features of the user U1 extracted from the image information showing the first image TP1 with feature data indicating the facial features of the user U1 extracted from the image information showing the third image TP2, and if the degree of match between the two feature data is equal to or greater than a predetermined threshold and the result of the judgment by the judgment unit 117 is positive, it authenticates that the user U1 is legitimate.

[0101] 3-2: Operation of the third embodiment Unlike the server 10 according to the first embodiment, the processing device 11B included in the server 10B according to the third embodiment functions as a determination unit 117 in step S4. The processing device 11B determines whether a value indicating the degree of agreement between the movement instructed by the instruction information DI for displaying instructions for the head movement of the user U1 on the MR glasses 30 and the head movement of the user U1 indicated by the movement information is equal to or greater than a predetermined value. The processing device 11B also functions as an authentication unit 113B. When the result of the determination made by the processing device 11B functioning as the determination unit 117 is positive, the processing device 11B authenticates that the user U1 is legitimate. On the other hand, when the result of the determination made by the processing device 11B functioning as the determination unit 117 is negative, the processing device 11B authenticates that the user U1 is not legitimate. In other steps, the processing performed by the server 10B is the same as the processing performed by the server 10, so a flowchart showing the operation of the server 10B is omitted.

[0102] 3-3: Effects of the third embodiment According to the above description, the server 10B serving as an avatar generation device includes an output unit 115 and a determination unit 117. The output unit 115 outputs instruction information DI for displaying instructions for the head movement of the user U1 on the MR glasses 30 serving as a display device. The determination unit 117 determines whether a value indicating the degree of agreement between the head movement of the user U1 and the movement instructed by the instruction information DI is equal to or greater than a predetermined value. In addition to performing authentication similar to that performed by the authentication unit 113 according to the first embodiment, the authentication unit 113B further certifies that the user U1 is legitimate when the determination result by the determination unit 117 is positive.

[0103] By using the above configuration, the server 10B can authenticate the user U1 based on the user U1's head movement itself. Through this process, the server 10B can generate a three-dimensional image WP of the avatar A1 corresponding to the user U1 who has been authenticated as legitimate. As a result, the server 10B can prevent other users U from impersonating the user U1.

[0104] 4: Variation The present disclosure is not limited to the above-described exemplary embodiments. Specific modified embodiments are exemplified below. Two or more embodiments selected arbitrarily from the following examples may be combined.

[0105] 4-1: Variation 1 The terminal device 20 according to the above embodiment includes an image generation unit 212. The image generation unit 212 generates a second image FP showing the front and side of the face of the user U1 based on an image of the user U1's head captured by the imaging device 26 during a head movement. However, the movement may be performed by a device other than the terminal device 20. For example, the server 10 or 10B may include an image generation unit similar to the image generation unit 212, thereby generating the second image FP showing the front and side of the face of the user U1. For example, the image generation unit 114 included in the server 10 or 10B may also have the function of the image generation unit 212, thereby generating the second image FP showing the front and side of the face of the user U1.

[0106] 4-2: Variation 2 In the information processing systems 1 to 1B according to the above-described embodiments, the terminal device 20 and the MR glasses 30 are realized as separate entities. However, the method for realizing the terminal device 20 and the MR glasses 30 in the embodiments of the present invention is not limited to this. For example, the terminal device 20 and the MR glasses 30 may be realized in a single housing by the MR glasses 30 having the same functions as the terminal device 20.

[0107] 4-3: Variation 3 The information processing systems 1 to 1B according to the above-described embodiments include MR glasses 30. However, the information processing systems 1 to 1B may include any one of an HMD employing VR (Virtual Reality) technology, an HMD employing AR (Augmented Reality) technology, and AR glasses employing AR technology, instead of the MR glasses 30. Alternatively, the information processing systems 1 to 1B may include any one of a regular smartphone and tablet equipped with an imaging device, instead of the MR glasses 30. These HMDs, AR glasses, smartphones, and tablets are examples of display devices.

[0108] 5:Other (1) In the above-described embodiment, storage device 12, storage device 22, and storage device 32 are exemplified by ROM and RAM, but may be a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disk), a smart card, a flash memory device (e.g., a card, a stick, a key drive), a CD-ROM (Compact Disc-ROM), a register, a removable disk, a hard disk, a floppy (registered trademark) disk, a magnetic strip, a database, a server, or any other suitable storage medium. The program may also be transmitted from a network via a telecommunications line. The program may also be transmitted from a communications network NET via a telecommunications line.

[0109] (2) In the above-described embodiments, the described information, signals, etc. may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

[0110] (3) In the above-described embodiment, input and output information may be stored in a specific location (for example, a memory) or may be managed using a management table. Input and output information may be overwritten, updated, or added to. Output information may be deleted. Input information may be transmitted to another device.

[0111] (4) In the above-described embodiment, the determination may be made by a value (0 or 1) represented using one bit, by a Boolean value (true or false), or by a comparison of numerical values ​​(e.g., comparison with a predetermined value).

[0112] (5) The order of the process procedures, sequences, flowcharts, etc. illustrated in the above-described embodiments may be rearranged unless inconsistent. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.

[0113] (6) Each function illustrated in Figures 1 to 11 is realized by any combination of at least one of hardware and software. Furthermore, the method for realizing each functional block is not particularly limited. That is, each functional block may be realized by using one device that is physically or logically coupled, or may be realized by using two or more devices that are physically or logically separated and connected directly or indirectly (for example, by wire, wirelessly, etc.). A functional block may also be realized by combining software with the one device or the multiple devices.

[0114] (7) The programs exemplified in the above-described embodiments should be broadly construed to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, functions, etc., regardless of whether they are called software, firmware, middleware, microcode, hardware description language, or by other names.

[0115] Software, instructions, information, etc. may also be transmitted or received over a transmission medium. For example, if software is transmitted from a website, server, or other remote source using wired technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and / or wireless technologies (such as infrared, microwave), then these wired and / or wireless technologies are included within the definition of transmission media.

[0116] (8) In each of the foregoing embodiments, the terms "system" and "network" are used interchangeably.

[0117] (9) The information, parameters, etc. described in this disclosure may be expressed using absolute values, relative values ​​from a predetermined value, or corresponding other information.

[0118] (10) In the above-described embodiments, the server 10 to the server 10B and the terminal device 20 may be mobile stations (MS). A mobile station may also be referred to by those skilled in the art as a subscriber station, a mobile unit, a subscriber unit, a wireless unit, a remote unit, a mobile device, a wireless device, a wireless communication device, a remote device, a mobile subscriber station, an access terminal, a mobile terminal, a wireless terminal, a remote terminal, a handset, a user agent, a mobile client, a client, or some other appropriate term. In the present disclosure, terms such as "mobile station," "user terminal," "user equipment (UE)," and "terminal" may be used interchangeably.

[0119] (11) In the above-described embodiments, the terms "connected," "coupled," or any variations thereof refer to any direct or indirect connection or coupling between two or more elements, including the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be a physical coupling or connection, a logical coupling or connection, or a combination thereof. For example, "connected" may be read as "access." As used in this disclosure, two elements are considered to be "connected" or "coupled" to each other using at least one of one or more wires, cables, and printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and optical (both visible and invisible) range, as some non-limiting and non-exhaustive examples.

[0120] (12) In the above embodiments, the phrase "based on" does not mean "based only on," unless otherwise specified. In other words, the phrase "based on" means both "based only on" and "based at least on."

[0121] (13) As used in this disclosure, the terms "determining" and "determining" may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiring (e.g., searching in a table, database, or other data structure), ascertaining, and the like. "Determining" and "determining" may also include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in memory), and the like. Furthermore, "judgment" and "decision" can include regarding resolving, selecting, choosing, establishing, comparing, etc. as having been "judgment" or "decision." In other words, "judgment" and "decision" can include regarding some action as having been "judgment" or "decision." Furthermore, "judgment (decision)" can be interpreted as "assuming," "expecting," "considering," etc.

[0122] (14) In the above embodiments, when "include," "including," and variations thereof are used, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, the term "or" as used in this disclosure is not intended to be an exclusive or.

[0123] (15) In this disclosure, where articles are added by translation, such as a, an, and the in English, this disclosure may include that the nouns following these articles are plural.

[0124] (16) In this disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "combined" may also be interpreted in the same way as "different."

[0125] (17) Each aspect / embodiment described in this disclosure may be used alone, in combination, or switched depending on the implementation. Notification of predetermined information (e.g., notification that "X is true") is not limited to explicit notification, but may be implicit (e.g., not notifying the predetermined information).

[0126] Although the present disclosure has been described in detail above, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described herein. The present disclosure can be implemented in modified and altered forms without departing from the spirit and scope of the present disclosure as defined by the claims. Therefore, the description of the present disclosure is intended to be illustrative and does not have any limiting meaning on the present disclosure. [Explanation of symbols]

[0127] 1, 1A, 1B... Information processing system, 10, 10A, 10B... Server, 11, 11A, 11B... Processing device, 12, 12A... Storage device, 13... Communication device, 14... Display, 15... Input device, 16... Imaging device, 20... Terminal device, 21... Processing device, 22... Storage device, 23... Communication device, 24... Display, 25... Input device, 26... Imaging device, 30... MR glasses, 31... Processing device, 32... Storage device, 33... Gaze detection device, 34... GPS device, 35... Motion detection device, 36... Imaging device, 37... Communication device, 38... Display, 41L, 41R... Lens, 9 1, 92...Temple, 93...Bridge, 94, 95...Frame, 111...First acquisition unit, 112...Second acquisition unit, 113, 113A, 113B...Authentication unit, 114...Image generation unit, 115...Output unit, 116...Third acquisition unit, 117...Determination unit, 211...Acquisition unit, 212...Image generation unit, 213...Output unit, 311...Acquisition unit, 312...Display control unit, A1, A2...Avatar, PR1, PR2, PR3...Control program, TI1, TI2...Image information, TP1...First two-dimensional image, TP2...Second two-dimensional image, U1, U2...User, VO, VO1 to VO5...Virtual object

Claims

1. a first acquisition unit that acquires a first image showing a frontal portion of a face of a user; a second acquisition unit that acquires second images showing the front and side portions of the user's face in accordance with a movement of the user's head; an authentication unit that authenticates the user based on the first image and the second image; an image generation unit that generates a head image of a three-dimensional avatar corresponding to the user using the second image; An avatar generation device comprising:

2. The avatar generation device according to claim 1 , wherein the authentication unit authenticates the user by comparing the first image with a third image showing the frontal portion of the user's face generated from the second image.

3. 2. The avatar generation device according to claim 1, wherein the authentication unit authenticates the user by inputting second feature information indicating the facial features of the user extracted from the second image to a learning model that has learned the relationship between first feature information indicating the facial features of each of the plurality of persons extracted from a first image showing a frontal portion of the face of each of the plurality of persons and an authentication result for each of the plurality of persons.

4. an output unit that outputs instruction information for displaying an instruction for the user's head movement on a display device; a determination unit that determines whether a value indicating a degree of agreement between the user's head movement and the movement indicated by the instruction information is equal to or greater than a predetermined value; The avatar generation device according to claim 1 , wherein the authentication unit further authenticates that the user is legitimate when the determination result of the determination unit is positive.

5. The avatar generation device according to claim 1 , wherein the second acquisition unit acquires the second image based on a video capturing a head movement of the user.

Images