Online dialogue support system
The online dialogue support system generates and authenticates complementary face images to prevent impersonation, ensuring the displayed face accurately represents the genuine user, thereby enhancing communication security.
Patent Information
- Application Number
- JP2023546844
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2021-09-10
- Filing Date
- 2022-08-08
- Publication Date
- 2024-11-21
- Estimated Expiration
- 2042-08-08
AI Technical Summary
Existing online dialogue systems fail to prevent impersonation when a user's face is partially hidden by a head-mounted display, facilitating unauthorized participation in online interactions.
An online dialogue support system that generates a complementary face image by completing hidden regions using machine learning, performs authentication based on reference and complementary face images, and displays the complementary image if authentication is successful, thereby confirming the user's identity.
Prevents impersonation by ensuring the displayed face image accurately represents the genuine user, enhancing communication integrity and security.
Smart Images

Figure 0007588243000001 
Figure 0007588243000002 
Figure 0007588243000003
Abstract
Description
[Technical field]
[0001] One aspect of the present invention relates to an online dialogue support system. [Background technology]
[0002] Patent document 1 discloses an apparatus that creates a mask pattern of the HMD area to be replaced from a facial moving image with a head mounted display (HMD), and replaces the area corresponding to the mask pattern in a facial still image without an HMD, thereby synthesizing a facial moving image without an HMD. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] Japanese Patent Application Publication No. 11-096366 Summary of the Invention [Problem to be solved by the invention]
[0004] For example, in a system in which multiple users have online conversations, when the mechanism described in the above Patent Document 1 is applied, a facial image of a first user without an HMD can be displayed on the terminals of other users, which is expected to promote communication between users. However, if the user wearing the HMD is not actually the first user (for example, if a user other than the first user is trying to participate in an online conversation by impersonating the first user), the facial image of the first user will be displayed on the terminals of the other users, which will encourage impersonation.
[0005] Therefore, an object of one aspect of the present invention is to provide an online dialogue support system capable of preventing impersonation of a user with a part of his or her face hidden in an online dialogue. [Means for solving the problem]
[0006] An online dialogue support system according to one aspect of the present invention is an online dialogue support system that supports online dialogue between a sending user's terminal and a receiving user's terminal, and includes a memory unit that stores a reference face image showing the sending user's face, an acquisition unit that acquires a hidden face image showing the sending user's face with some areas of the sending user's face hidden, a generation unit that complements some areas of the hidden face image and generates a complemented face image, an authentication unit that performs authentication based on the reference face image and the complemented face image, and a display control unit that displays the complemented face image on the receiving user's terminal if authentication is successful.
[0007] In an online dialogue support system according to one aspect of the present invention, a complemented face image is generated by complementing a part of the face of a sending user from a hidden face image in which the part is hidden. Then, authentication is performed based on a reference face image, which is the sending user's original (complete) face image, and the complemented face image. If authentication is successful, the complemented face image is displayed on the receiving user's terminal. According to the above online dialogue support system, the receiving user can confirm that the sending user is the person in question by confirming that the complemented face image is displayed on the receiving user's terminal. Therefore, impersonation of the sending user can be prevented. Effect of the Invention
[0008] According to one aspect of the present invention, it is possible to provide an online dialogue support system capable of preventing impersonation of a user with a partially hidden face area in an online dialogue. [Brief description of the drawings]
[0009] [Figure 1] 1 is a diagram showing an overview of an online dialogue support system according to an embodiment. [Diagram 2] 1 is a block diagram showing an example of a functional configuration of an online dialogue support system. [Diagram 3] 10A to 10C are diagrams illustrating a facial image complementation process. [Figure 4]FIG. 2 is a sequence diagram showing an example of the operation of the online dialogue support system according to the first embodiment. [Diagram 5] FIG. 11 is a block diagram showing an example of a functional configuration of an online dialogue support system according to a second embodiment. [Figure 6] FIG. 11 is a sequence diagram showing an example of an operation of the online dialogue support system according to the second embodiment. [Figure 7] FIG. 13 is a block diagram showing an example of a functional configuration of an online dialogue support system according to a third embodiment. [Figure 8] FIG. 11 is a sequence diagram showing an example of the operation of the online dialogue support system according to the third embodiment. [Figure 9] FIG. 1 is a diagram illustrating an example of a hardware configuration related to an online dialogue support system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0010] Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings. In the description of the drawings, the same or corresponding elements are designated by the same reference numerals, and duplicated explanations will be omitted.
[0011] FIG. 1 is a diagram showing an overview of an online dialogue support system 1 according to an embodiment. The online dialogue support system 1 is a computer system that supports online dialogue between terminals of a plurality of users. In the online dialogue support system 1, a facial image showing the face of a user is captured, and the facial image is transmitted and received. The facial image may be real 3D data, or may show the entire body of the user. In this embodiment, a user who transmits his / her own facial image is called a transmitting user, and a user who receives the facial image of the transmitting user is called a receiving user. However, the transmitting user and the receiving user are not fixedly set for each user. The transmitting user becomes the receiving user when he / she receives the facial image of another user. Also, the receiving user becomes the transmitting user when he / she transmits his / her own facial image to another user.
[0012] The online dialogue support system 1 includes a sender's user terminal 10 and a receiver's user terminal 20. The sender's user terminal 10 and the receiver's user terminal 20 are communicatively connected via a communication network N. The configuration of the communication network N is not limited. For example, the communication network N may include the Internet or an intranet. In the example of FIG. 1, one sender's user terminal 10 and one receiver's user terminal 20 are shown, but the number is not limited to this. For example, the online dialogue support system 1 may include a plurality of sender's user terminals 10 and a plurality of receiver's user terminals 20. In other words, the online dialogue support system 1 can be applied as a system for conducting online dialogue between multiple people.
[0013] The sender's user terminal 10 is a terminal used by a sender user. The type and configuration of the sender's user terminal 10 are not limited. The sender's user terminal 10 may be a portable terminal such as a highly functional mobile phone (smartphone), a tablet terminal, a wearable terminal, a laptop personal computer, or a mobile phone. Alternatively, the sender's user terminal 10 may be a stationary terminal such as a desktop personal computer. The sender's user terminal 10 may be a user terminal owned by each sender user as described above, or a server device configured to be able to communicate with the user terminal of each sender user. Alternatively, the sender's user terminal 10 may be configured by a combination of a user terminal and a server device. That is, the sender's user terminal 10 may be configured by a single computer device, or may be configured by multiple computer devices that can communicate with each other.
[0014] The receiving user terminal 20 is a terminal used by a receiving user. The type and configuration of the receiving user terminal 20 are the same as those of the sending user terminal 10. As described above, a receiving user can be a sending user, and a sending user can be a receiving user. Therefore, when a receiving user is a sending user, the receiving user terminal 20 functions as the sending user terminal 10. Also, when a sending user is a receiving user, the sending user terminal 10 functions as the receiving user terminal 20.
[0015] The transmitting user wears a head mounted display D on his / her head. The form of the head mounted display D is not limited to a specific form. The head mounted display D may take various forms, such as goggles, glasses (eyeglasses), and hats. The head mounted display D is, for example, smart glasses such as XR (eXtended Reality) glasses. In one example, the head mounted display D is AR glasses having a function of providing an augmented reality (AR) to the user. That is, the head mounted display D is a see-through type of glasses configured to allow the user to view the real space (outside world) together with the virtual space. However, the head mounted display D is not limited to the above, and may be an MR device such as MR glasses having a function of providing a mixed reality (MR) to the user, or a VR device such as VR glasses having a function of providing a virtual reality (VR) to the user.
[0016] As an example, the online dialogue support system 1 may be applied with Volumetric Video (or Volumetric Capture) technology. This technology is a technology for capturing images of a subject from all directions using multiple cameras or the like, and creating 3D content that reproduces the subject's natural appearance, shape, and movements (hereinafter referred to as "movements, etc.") with high accuracy. The online dialogue support system 1 to which this technology is applied reproduces the 3D movements, etc. of multiple users in the same virtual space in real time, providing each user with a user experience that makes them feel as if they are having a conversation in the same space. To enjoy such a user experience, the sending user and the receiving user participate in the online dialogue while wearing the head-mounted display D.
[0017] [First embodiment] FIG. 2 is a block diagram showing an example of the functional configuration of the online dialogue support system 1A(1) according to the first embodiment. The online dialogue support system 1A includes a sender's user terminal 10A(10) and a receiver's user terminal 20A(20). In the first embodiment, the main functions of the online dialogue support system are executed by the receiver's user terminal 20A. That is, in the first embodiment, it can be considered that the receiver's user terminal 20A alone constitutes the online dialogue support system. The sender's user terminal 10A includes an image capture unit 11 and a transmission unit 12.
[0018] The photographing unit 11 photographs the face of the sending user to obtain a facial image. For example, the photographing unit 11 photographs a reference facial image showing the face of the sending user. The reference facial image is an original (complete) facial image of the sending user photographed in a state in which the sending user's face is not hidden. For example, the reference facial image is an image in which the sending user's entire face is photographed in a state in which the sending user's face is not hidden by the head mounted display D (i.e., a state before the sending user wears the head mounted display D). For example, the photographing unit 11 photographs the reference facial image as a still image.
[0019] Furthermore, the photographing unit 11 photographs a hidden face image showing the face of the transmitting user in a state where a part of the face of the transmitting user is hidden. The hidden face image is an incomplete face image of the transmitting user photographed in a state where a part of the face of the transmitting user is hidden by an object existing between the photographing unit 11 and the face of the transmitting user. For example, the hidden face image is an image in which the entire face of the transmitting user is photographed in a state where a part of the face of the transmitting user is hidden by the head mounted display D (i.e., a state after the transmitting user wears the head mounted display D). In other words, the hidden face image is an image in which the head mounted display D that hides a part of the face of the transmitting user is reflected together with the face of the transmitting user. For example, the photographing unit 11 photographs the hidden face image in a moving image format.
[0020] The transmission unit 12 transmits the reference face image and the hidden face image acquired by the photographing unit 11 to the receiver's user terminal 20A. For example, the transmission unit 12 transmits the reference face image to the receiver's user terminal 20A before the online dialogue starts, and transmits the hidden face image to the receiver's user terminal 20A after the online dialogue starts.
[0021] The receiver's user terminal 20A includes a receiver 21 (acquirer), a generator 22, an authentication unit 23, and a display controller .
[0022] The receiving unit 21 functions as an acquiring unit that acquires the reference face image and the hidden face image by receiving the reference face image and the hidden face image from the sending user terminal 10A. The receiving unit 21 stores the reference face image in the storage unit 30 described below. Note that the reference face image of the sending user may be stored (registered) directly from the sending user terminal 10 to the storage unit 30 without passing through the receiving user terminal 20A.
[0023] The generating unit 22 complements a part of the hidden face image to generate a complemented face image. The complemented face image is a face image showing the face of the transmitting user in a state where a part of the hidden face image is not concealed. The generating unit 22 generates the complemented face image in a moving image format. A part of the hidden face image is concealed, for example, by a head mounted display D worn by the transmitting user. The generating unit 22 complements the part of the hidden face image corresponding to the head mounted display D by replacing it with another image. For example, the generating unit 22 replaces a part of the hidden face image corresponding to the head mounted display D with an image representing a part of the face of the transmitting user, thereby reproducing the face of the transmitting user in a state where the head mounted display D is not worn. The complementing process of the face image will be described later.
[0024] The authentication unit 23 performs authentication based on the reference face image and the complemented face image. The authentication unit 23 performs face authentication using a known method. In one example, the authentication unit 23 extracts feature points and face areas in the reference face image and the complemented face image. The authentication unit 23 calculates the similarity between the two by comparing the extracted values. If the similarity between the two is equal to or greater than a predetermined threshold, the authentication unit 23 determines that the authentication is successful, and if the similarity between the two is less than the predetermined threshold, the authentication is unsuccessful.
[0025] The display control unit 24 controls the display of the face image of the sending user on the receiver's user terminal 20A. In response to the authentication result by the authentication unit 23, the display control unit 24 displays a complemented face image or a hidden face image on an output device (display device) such as a display provided in the receiver's user terminal 20A. For example, if the authentication is successful, the display control unit 24 displays the complemented face image on the receiver's user terminal 20A. On the other hand, if the authentication is unsuccessful, the display control unit 24 displays the hidden face image on the receiver's user terminal 20A.
[0026] The storage unit 30 stores various data used or generated in the receiver's user terminal 20A. For example, the storage unit 30 stores a reference face image showing the face of the sender's user. The storage unit 30 may store data such as feature points of the reference face image used for authentication by the authentication unit 23. The storage unit 30 may store at least one of the hidden face image acquired by the receiver 21 and the complemented face image generated by the generator 22. The storage unit 30 may store the shape of the head mounted display D. The storage unit 30 may be a device separate from the receiver's user terminal 20A, or may be a component of the receiver's user terminal 20A.
[0027] The facial image complementation process will be described with reference to Fig. 3. Fig. 3 is a diagram showing the facial image complementation process. Note that although an example of a reference facial image is shown on the receiver's user terminal 20A side, the reference facial image is not used for complementing the facial image.
[0028] The sender's user terminal 10A captures a hidden face image and transmits it to the receiver's user terminal 20A. In this hidden face image, a part of the sender's face is hidden by the head mounted display D worn by the sender. Specifically, in the hidden face image, the area around the sender's eyes is hidden by the lens, frame, bridge, etc. of the head mounted display D.
[0029] The generating unit 22 specifies a partial region R of the target to be complemented from the hidden face image. For example, the generating unit 22 reads out the shape of the head mounted display D stored in advance in the storage unit 30, and specifies the partial region R by specifying the region of the hidden face image corresponding to the head mounted display D based on the shape.
[0030] The generating unit 22 generates a complemented face image by complementing a part of the region R of the hidden face image. Completion of the part of the region R may be performed by machine learning. For example, the generating unit 22 prepares in advance a model configured to input an image showing a part of the face of the transmitting user and output an estimation result of the other part of the face of the transmitting user by machine learning using face images of a plurality of transmitting users (so-called positive examples) as well as face images of a plurality of users different from the transmitting user as teacher data. The face images of the plurality of transmitting users are, for example, face images showing the entire face of the transmitting user taken from various angles. The face images of the plurality of users different from the transmitting user are, for example, face images showing the entire face of each of the plurality of users taken from various angles. By performing machine learning using teacher data including positive examples and negative examples as described above, a model configured as follows can be obtained. That is, a model can be obtained that is configured to output an estimation result of other parts reflecting the characteristics of the genuine sending user (e.g., an image including a part hidden by the head mounted display D) when an image showing a part of the face of the genuine sending user (e.g., a part including the mouth that is not hidden by the head mounted display D) is input, and to output an estimation result not reflecting the characteristics of the genuine sending user when an image showing a part of the face of a user different from the genuine sending user (e.g., a user trying to impersonate the genuine sending user) is input. The generation unit 22 inputs the area of the hidden face image excluding the part R to the model, and complements the part R based on the output result from the model.
[0031] In another example of complementing the partial region R, the generating unit 22 prepares in advance a model configured to input an image of a first range showing a part of the partial region R and output an estimation result of a second range showing a part of the partial region R different from the first range by machine learning using face images of a plurality of transmitting users as well as face images of a plurality of users different from the transmitting users as teacher data. The image of the first range showing a part of the partial region R may be acquired by a head mounted display D. For example, the head mounted display D may be provided with a camera on the inside (user side) of the bridge part of the glasses. The head mounted display D captures an image of the eyes of the transmitting user (for example, a video including both eyes of the user as a subject) while the transmitting user is wearing the head mounted display D. The head mounted display D may capture an image of the eyes of the transmitting user by two cameras arranged for each eye of the transmitting user (for example, two cameras arranged inside each lens). Then, the generation unit 22 may input an image showing a part of the sending user's face from the part of the region R (for example, an image of the sending user's eyes) into the model, and complement the part of the region based on the input image and the output result from the model.
[0032] Through the above-described processing, the portion of the hidden face image corresponding to the head mounted display D is replaced with an image representing the face of the transmitting user. As a result, in a part of the region R in the complemented face image, the face of the transmitting user without wearing the head mounted display D is reproduced.
[0033] The operation of the online dialogue support system 1A will be described with reference to Fig. 4. Fig. 4 is a sequence diagram showing the operation of the online dialogue support system 1A as a processing flow S1. In the following, it is assumed that the storage unit 30 stores in advance a reference face image showing the face of the sending user.
[0034] In step S11, the photographing unit 11 photographs a hidden face image showing the face of the transmitting user in a state where a part of the face of the transmitting user is hidden. In one example, the photographing unit 11 photographs the face of the transmitting user in a state where a part of the face of the transmitting user is hidden by a head mounted display D worn by the transmitting user. The photographing unit 11 photographs the hidden face image in a moving image format.
[0035] In step S12, the transmission unit 12 transmits the hidden face image acquired by the photographing unit 11 to the receiver's user terminal 20A. For example, the transmission unit 12 transmits the hidden face image in a moving image format to the receiver's user terminal 20A in real time.
[0036] In step S13, the receiving unit 21 receives the hidden face image from the sender's user terminal 10 A, thereby acquiring the hidden face image. The receiving unit 21 may store the hidden face image in the storage unit 30.
[0037] In step S14, the generating unit 22 complements a partial area of the hidden face image to generate a complemented face image. In one example, the generating unit 22 identifies a partial area R (see FIG. 3) by identifying a region of the hidden face image corresponding to the head mounted display D based on the shape of the head mounted display D pre-stored in the storage unit 30.
[0038] As an example of the complementation process, the generation unit 22 prepares in advance a model configured to input an image showing a part of the face of the transmitting user and output an estimation result of the other part of the face of the transmitting user by machine learning using face images of the transmitting users together with face images of the transmitting users as teacher data, inputs a region of the hidden face image excluding a part of the region R to the model, and complements the part of the region R based on the output result from the model. In another example, the generation unit 22 prepares in advance a model configured to input an image of a first range showing a part of the part of the region R and output an estimation result of a second range showing a part of the part of the region R different from the first range by machine learning using face images of the transmitting users together with face images of the transmitting users as teacher data, inputs an image showing a part of the part of the region R to the model, and complements the part of the region based on the input image and the output result from the model.
[0039] In step S15, the authentication unit 23 performs authentication based on the reference face image and the complemented face image. The authentication unit 23 performs face authentication using a known method and determines whether the authentication is successful. The authentication unit 23 does not need to perform authentication on frames after the frame in which authentication is successful, among a series of frames of the hidden face image in the moving image format.
[0040] In step S16, the display control unit 24 controls the display of the face image of the sending user on the receiving user terminal 20A. For example, if the authentication is successful in the process of step S15, the display control unit 24 causes the receiving user terminal 20A to display a complemented face image. On the other hand, if the authentication is unsuccessful in the process of step S15, the display control unit 24 causes the receiving user terminal 20A to display a hidden face image. The display control unit 24 may display the complemented face image or the hidden face image of the sending user on the head mounted display D worn by the receiving user.
[0041] According to the online dialogue supporting system 1A described above, a complemented face image is generated by complementing a part of the region R of the sending user's face from a hidden face image in which the part of the region R is hidden. Then, authentication is performed based on the reference face image, which is the sending user's original (complete) face image, and the complemented face image. If the authentication is successful, the complemented face image is displayed on the receiving user terminal 20A. According to the online dialogue supporting system 1A described above, the receiving user can confirm that the sending user is the person in question by confirming that the complemented face image is displayed on the receiving user terminal 20A. Therefore, it is possible to prevent the sending user from impersonating the other person.
[0042] If the authentication fails, the display control unit 24 causes the receiver's user terminal 20A to display a hidden face image. In this case, the receiver's user can detect the possibility that the sender is an impersonator by checking that the hidden face image is displayed on the receiver's user terminal 20A.
[0043] A part of the region R is hidden by the head mounted display D worn by the sending user. The generating unit 22 complements the part of the region R by replacing the part of the hidden face image corresponding to the head mounted display D with another image. According to such a configuration, the sending user is displayed on the receiving user terminal 20A as if he is not wearing the head mounted display D. This allows the sending user to visually recognize the display by the head mounted display D and realize more natural communication between the sending user and the receiving user.
[0044] The generating unit 22 specifies a region of the hidden face image corresponding to the head mounted display D based on the pre-stored shape of the head mounted display D, thereby specifying a portion of the region R. In this case, the head mounted display D is precisely specified, and thus the accuracy of complementing the portion of the region R is improved.
[0045] The generating unit 22 prepares a model configured to input an image showing a part of the face of the transmitting user and output an estimation result of the other part of the face of the transmitting user by machine learning using the face images of the transmitting user and the face images of the multiple users different from the transmitting user as teacher data, inputs the area of the hidden face image excluding the part of the area R to the model, and complements the part of the area R based on the output result from the model. With such a configuration, the part of the face of the transmitting user is complemented more naturally. Furthermore, even if the user moves, the part of the user's face is complemented according to the user's movement, so the complement accuracy of the part of the area R is improved.
[0046] The generation unit 22 prepares a model configured to input an image of a first range showing a part of the part of the region R and output an estimation result of a second range showing a part of the part of the region R different from the first range by machine learning using face images of a plurality of sending side users and face images of a plurality of users different from the sending side users as teacher data, inputs an image showing a part of the sending side user's face in the part of the region R to the model, and complements the part of the region based on the input image and the output result from the model. According to such a configuration, the part of the region R is complemented by a combination of an image showing a part of the sending side user's face in the part of the region R (an image of the sending side user's eyes in this embodiment) and the output result from the model. In this case, since a part of the sending side user's face in the part of the region R is complemented by the image, a complemented face image closer to the user's original face can be obtained. In addition, since the sending side user's gaze, expression, etc. are reflected in the complemented face image, a part of the sending side user's face is complemented more naturally.
[0047] [Second embodiment] 5 is a block diagram showing an example of a functional configuration of an online dialogue supporting system 1B(1) according to the second embodiment. The online dialogue supporting system 1B differs from the online dialogue supporting system 1A in that it includes a sender's user terminal 10B(10) and a receiver's user terminal 20B(20) instead of the sender's user terminal 10A and the receiver's user terminal 20A. In the online dialogue supporting system 1B, a complemented face image is generated on the sender's user terminal 10B, and authentication is performed based on the reference face image and the complemented face image.
[0048] The sender's user terminal 10B differs from the sender's user terminal 10A in that it has a generating unit 22 and an authenticating unit 23, and has a transmitting unit 12B instead of the transmitting unit 12. The receiver's user terminal 20B differs from the receiver's user terminal 20A in that it does not have the generating unit 22 and the authenticating unit 23, and has a receiving unit 21B instead of the receiving unit 21. The transmitting unit 12B transmits a hidden face image or a complemented face image to the receiver's user terminal 20B according to the authentication result of the authenticating unit 23. Specifically, when the authentication by the authenticating unit 23 is successful, the transmitting unit 12B transmits the complemented face image to the receiver's user terminal 20B. On the other hand, when the authentication by the authenticating unit 23 is unsuccessful, the transmitting unit 12B transmits the hidden face image to the receiver's user terminal 20B. The receiving unit 21B receives the hidden face image or the complemented face image from the sender's user terminal 10B.
[0049] In the online dialogue support system 1B, the photographing unit 11 functions as an acquisition unit that acquires a hidden face image. In addition, the transmission unit 14 that transmits a hidden face image or a complemented face image to the receiving user terminal 20B according to the authentication result functions as a display control unit that essentially displays the complemented face image on the receiving user terminal 20B when authentication is successful, and displays the hidden face image on the receiving user terminal 20B when authentication is unsuccessful. Therefore, in the second embodiment, the main functions of the online dialogue support system are executed by the sending user terminal 10B. That is, in the second embodiment, it can be considered that the sending user terminal 10B alone constitutes the online dialogue support system. Note that the storage unit 30 may be a device separate from the sending user terminal 10B, or may be a component of the sending user terminal 10B.
[0050] The operation of the online dialogue support system 1B will be described with reference to Fig. 6. Fig. 6 is a sequence diagram showing the operation of the online dialogue support system 1B as a process flow S2. In the following, it is assumed that the storage unit 30 stores in advance a reference face image showing the face of the sending user.
[0051] In step S21, the photographing unit 11 photographs a hidden face image showing the face of the transmitting user in a state where a part of the face of the transmitting user is hidden. In one example, the photographing unit 11 photographs the face of the transmitting user in a state where a part of the face is hidden by a head mounted display D worn by the transmitting user. The photographing unit 11 photographs the hidden face image in a moving image format. The photographing unit 11 may also store the hidden face image in the storage unit 30.
[0052] In step S22, the generation unit 22 complements a partial area of the hidden face image to generate a complemented face image. The process of step S22 differs from step S14 in FIG. 4 in that it is performed on the sender's user terminal 10B.
[0053] In step S23, the authentication unit 23 performs authentication based on the reference face image and the complemented face image. The process of step S23 differs from step S15 in FIG. 4 in that it is performed on the sender's user terminal 10B.
[0054] In step S24, the transmission unit 12B transmits the hidden face image or the complemented face image to the receiver's user terminal 20B. The transmission unit 12B transmits the hidden face image or the complemented face image in a moving image format to the receiver's user terminal 20B in real time. For example, if the authentication is successful in the process of step S23, the transmission unit 12B transmits the complemented face image to the receiver's user terminal 20B. On the other hand, if the authentication is unsuccessful in the process of step S23, the transmission unit 12B transmits the hidden face image to the receiver's user terminal 20B.
[0055] In step S25, the receiving unit 21B receives the hidden face image or the complemented face image from the sender's user terminal 10B, thereby acquiring the hidden face image or the complemented face image.
[0056] In step S26, the display control unit 24 controls the display of the face image of the sending user on the receiving user terminal 20B. For example, the display control unit 24 causes the receiving user terminal 20B to display the hidden face image or the complemented face image acquired in the process of step S25. The display control unit 24 may cause the sending user's complemented face image or the hidden face image to be displayed on the head mounted display D worn by the receiving user.
[0057] The online dialogue supporting system 1B described above provides the same operational effects as the online dialogue supporting system 1A. That is, the receiving user can confirm that the sending user is the person in question by checking that the complemented face image is displayed on the receiving user terminal 20B. This makes it possible to prevent the sending user from impersonating another person. Furthermore, the online dialogue supporting system 1B allows the authentication process to be performed on the sending user terminal 10B side, thereby reducing the processing load on the receiving user terminal 20B.
[0058] [Third embodiment] 7 is a block diagram showing an example of a functional configuration of an online dialogue supporting system 1C(1) according to the third embodiment. The online dialogue supporting system 1C differs from the online dialogue supporting system 1A in that it includes a sender's user terminal 10C(10) and a receiver's user terminal 20C(20) instead of the sender's user terminal 10A and the receiver's user terminal 20A, and further includes a server 40. In the online dialogue supporting system 1C, a complemented face image is generated on the server 40, and authentication is performed based on the reference face image and the complemented face image.
[0059] The sender's user terminal 10C differs from the sender's user terminal 10A in that it has a sending unit 12C instead of the sending unit 12. The sending unit 12C sends a hidden face image to the server 40. The receiver's user terminal 20C differs from the receiver's user terminal 20A in that it does not have the generating unit 22 and the authenticating unit 23, and has a receiving unit 21C instead of the receiving unit 21. The receiving unit 21C receives the hidden face image or the complemented face image from the server 40.
[0060] The server 40 includes an image receiving unit 41 (acquisition unit), a generating unit 22, an authentication unit 23, and an image transmitting unit 42. The image receiving unit 41 functions as an acquisition unit that acquires the reference face image and the hidden face image by receiving the reference face image and the hidden face image from the sending user terminal 10C. The image transmitting unit 42 transmits the hidden face image or the complemented face image to the receiving user terminal 20C according to the authentication result of the authentication unit 23. Specifically, when the authentication by the authentication unit 23 is successful, the image transmitting unit 42 transmits the complemented face image to the receiving user terminal 20C. On the other hand, when the authentication by the authentication unit 23 is unsuccessful, the image transmitting unit 42 transmits the concealed face image to the receiving user terminal 20C. That is, the image transmitting unit 42 essentially functions as a display control unit that displays the complemented face image on the receiving user terminal 20C when the authentication is successful, and displays the concealed face image on the receiving user terminal 20C when the authentication is unsuccessful. Therefore, in the third embodiment, the main functions of the online dialogue support system are executed by the server 40. That is, in the third embodiment, it can be considered that the server 40 alone constitutes the online dialogue support system. Note that the storage unit 30 may be a device separate from the server 40, or may be one of the components of the server 40.
[0061] The operation of the online dialogue support system 1C will be described with reference to Fig. 8. Fig. 8 is a sequence diagram showing the operation of the online dialogue support system 1C as a processing flow S3. In the following, it is assumed that the storage unit 30 stores in advance a reference face image showing the face of the sending user.
[0062] The process of step S31 is similar to step S11 in FIG.
[0063] In step S32, the transmission unit 12C transmits the hidden face image acquired by the photographing unit 11 to the server 40. For example, the transmission unit 12C transmits the hidden face image in a moving image format to the server 40 in real time.
[0064] In step S33, the image receiving unit 41 receives the hidden face image from the sender's user terminal 10C, thereby acquiring the hidden face image. The image receiving unit 41 may store the hidden face image in the storage unit 30.
[0065] In step S34, the generation unit 22 complements a partial area of the hidden face image to generate a complemented face image. The process of step S34 differs from step S14 in FIG.
[0066] In step S35, the authentication unit 23 performs authentication based on the reference face image and the complemented face image. The process of step S35 differs from step S15 in FIG.
[0067] In step S36, the image transmission unit 42 transmits the hidden face image or the complemented face image to the receiver's user terminal 20C. The image transmission unit 42 transmits the hidden face image or the complemented face image in a moving image format to the receiver's user terminal 20C in real time. For example, if the authentication is successful in the process of step S35, the image transmission unit 42 transmits the complemented face image to the receiver's user terminal 20C. On the other hand, if the authentication is unsuccessful in the process of step S35, the image transmission unit 42 transmits the hidden face image to the receiver's user terminal 20C.
[0068] In step S37, the receiving unit 21C receives the hidden face image or the complemented face image from the server 40, thereby acquiring the hidden face image or the complemented face image.
[0069] In step S38, the display control unit 24 controls the display of the face image of the sending user on the receiving user terminal 20C. For example, the display control unit 24 causes the receiving user terminal 20C to display the hidden face image or the complemented face image acquired in the process of step S37. The display control unit 24 may cause the sending user's complemented face image or the hidden face image to be displayed on the head mounted display D worn by the receiving user.
[0070] The online dialogue support system 1C described above provides the same operational effects as the online dialogue support system 1A. That is, the receiving user can confirm that the sending user is the person in question by checking that the complemented face image is displayed on the receiving user terminal 20C. This makes it possible to prevent the sending user from impersonating another person. Furthermore, the online dialogue support system 1C can reduce the processing load on the receiving user terminal 20C. Furthermore, when multiple people (i.e., terminals of multiple users) are engaged in an online dialogue, time synchronization can be easily achieved regardless of the equipment and performance of the sending user terminal 10C.
[0071] (Modification) In the above embodiment, an example was described in which a part of the face of the sending user is hidden by the head mounted display D, but the hidden face image may be hidden by a mask or the like. In the above embodiment, an example was described in which the hidden face image is in a moving image format, but in the case where the face image of the sending user is displayed as a still image on the receiving user terminal 20, the hidden face image may be a still image. The hidden face image may be acquired separately as a still image used for face authentication and as a moving image format after face authentication. The hidden face image may be an image in which a part of the area is hidden by, for example, a mosaic or the like. In the above embodiment, the part of the area R is specified based on the shape of the head mounted display D stored in advance, but the generating unit 22 may specify the part of the area R by accepting an input operation or the like for specifying the part of the area R by the sending user or the receiving user. In addition, in the above embodiment, an example was described in which the hidden face image of the sending user is displayed when authentication fails, but the display control unit 24 may display an error message indicating the failure of authentication, or may end the online dialogue.
[0072] The block diagrams used to explain the above embodiments show functional blocks. These functional blocks (components) are realized by any combination of at least one of hardware and software. The method of realizing each functional block is not particularly limited. That is, each functional block may be realized by using one device that is physically or logically combined, or may be realized by using two or more devices that are physically or logically separated and directly or indirectly connected (for example, by wire, wirelessly, etc.). The functional blocks may be realized by combining the one device or the multiple devices with software.
[0073] Functions include, but are not limited to, judging, determining, assessing, calculating, processing, deriving, investigating, searching, verifying, receiving, sending, outputting, accessing, resolving, selecting, choosing, establishing, comparing, anticipating, expecting, regarding, broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocating, mapping, and assigning.
[0074] For example, the sender's user terminal 10, the receiver's user terminal 20, and the server 40 in an embodiment of the present disclosure may function as a computer that performs the information processing method of the present disclosure. Fig. 9 is a diagram showing an example of a hardware configuration common to the sender's user terminal 10, the receiver's user terminal 20, and the server 40 according to an embodiment of the present disclosure. Each of the sender's user terminal 10, the receiver's user terminal 20, and the server 40 may be physically configured as a computer device including a processor 1001, a memory 1002, a storage 1003, a communication device 1004, an input device 1005, an output device 1006, a bus 1007, etc.
[0075] In the following description, the term "apparatus" may be interpreted as a circuit, a device, a unit, etc. The hardware configurations of the sender's user terminal 10, the receiver's user terminal 20, and the server 40 may be configured to include one or more of the apparatuses shown in Fig. 9, or may be configured to exclude some of the apparatuses.
[0076] Each function of the sending user terminal 10, the receiving user terminal 20, and the server 40 is realized by loading specific software (programs) onto hardware such as a processor 1001 and a memory 1002, causing the processor 1001 to perform calculations, control communications via a communication device 1004, and control at least one of the reading and writing of data in the memory 1002 and the storage 1003.
[0077] The processor 1001 controls the entire computer by running, for example, an operating system. The processor 1001 may be configured with a central processing unit (CPU) including an interface with peripheral devices, a control device, an arithmetic unit, a register, and the like.
[0078] Moreover, the processor 1001 reads out a program (program code), a software module, data, etc. from at least one of the storage 1003 and the communication device 1004 into the memory 1002, and executes various processes according to the read out programs. The programs used are those that cause a computer to execute at least a part of the operations described in the above-mentioned embodiments. For example, each functional unit (e.g., the generating unit 22, etc.) of the sender's user terminal 10, the receiver's user terminal 20, and the server 40 may be realized by a control program stored in the memory 1002 and operated in the processor 1001, and other functional blocks may be realized in a similar manner. Although the above-mentioned various processes have been described as being executed by one processor 1001, they may be executed by two or more processors 1001 simultaneously or sequentially. The processor 1001 may be implemented by one or more chips. The programs may be transmitted from a network via a telecommunication line.
[0079] The memory 1002 is a computer-readable recording medium, and may be configured by at least one of, for example, a Read Only Memory (ROM), an Erasable Programmable ROM (EPROM), an Electrically Erasable Programmable ROM (EEPROM), a Random Access Memory (RAM), etc. The memory 1002 may be called a register, a cache, a main memory (primary storage device), etc. The memory 1002 can store a program (program code), software modules, etc. that are executable to implement an information processing method according to an embodiment of the present disclosure.
[0080] Storage 1003 is a computer-readable recording medium, and may be composed of at least one of, for example, an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disk), a smart card, a flash memory (e.g., a card, a stick, a key drive), a floppy (registered trademark) disk, a magnetic strip, and the like. Storage 1003 may be referred to as an auxiliary storage device. The above-mentioned storage medium may be, for example, a database, a server, or other suitable medium including at least one of memory 1002 and storage 1003.
[0081] The communication device 1004 is hardware (transmission / reception device) for performing communication between computers via at least one of a wired network and a wireless network, and is also called, for example, a network device, a network controller, a network card, or a communication module.
[0082] The input device 1005 is an input device (e.g., a keyboard, a mouse, a microphone, a switch, a button, a sensor, etc.) that accepts input from the outside. The output device 1006 is an output device (e.g., a display, a speaker, an LED lamp, etc.) that performs output to the outside. The input device 1005 and the output device 1006 may be integrated into one device (e.g., a touch panel).
[0083] In addition, each device such as the processor 1001 and the memory 1002 is connected by a bus 1007 for communicating information. The bus 1007 may be configured using a single bus, or may be configured using different buses between each device.
[0084] The sender's user terminal 10, the receiver's user terminal 20, and the server 40 may be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), etc., and some or all of the functional blocks may be realized by the hardware. For example, the processor 1001 may be implemented using at least one of these pieces of hardware.
[0085] Although the present embodiment has been described in detail above, it is clear to those skilled in the art that the present embodiment is not limited to the embodiment described in this specification. The present embodiment can be implemented as a modified or altered form without departing from the spirit and scope of the present invention defined by the description of the claims. Therefore, the description in this specification is intended to be illustrative and does not have any limiting meaning on the present embodiment.
[0086] The order of the steps, sequences, flow charts, etc. of each aspect / embodiment described in this disclosure may be changed unless inconsistent. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.
[0087] The input and output information may be stored in a specific location (e.g., memory) or may be managed using a management table. The input and output information may be overwritten, updated, or added to. The output information may be deleted. The input information may be transmitted to another device.
[0088] The determination may be based on a value represented by a single bit (0 or 1), a Boolean value (true or false), or a numerical comparison (e.g., comparison with a predetermined value).
[0089] Each aspect / embodiment described in this disclosure may be used alone, in combination, or switched according to execution. In addition, notification of specific information (e.g., notification that "X is the case") is not limited to being done explicitly, but may be done implicitly (e.g., not notifying the specific information).
[0090] Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.
[0091] Additionally, software, instructions, information, etc. may be transmitted or received over a transmission medium. For example, if the software is transmitted from a website, server, or other remote source using wired and / or wireless technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and / or wireless technologies (such as infrared, microwave, etc.), then these wired and / or wireless technologies are included within the definition of transmission media.
[0092] The information, signals, etc. described in this disclosure may be represented using any of a variety of different technologies. For example, the data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.
[0093] Furthermore, the information, parameters, etc. described in this disclosure may be expressed using absolute values, may be expressed using relative values from a predetermined value, or may be expressed using other corresponding information.
[0094] The names used for the above-mentioned parameters are not limiting in any way. Furthermore, the formulas etc. using these parameters may differ from those explicitly disclosed in this disclosure. The various information elements may be identified by any suitable names, and the various names assigned to these various information elements are not limiting in any way.
[0095] As used in this disclosure, the phrase "based on" does not mean "based only on," unless expressly stated otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."
[0096] Any reference to an element using a designation such as "first," "second," etc., used in this disclosure does not generally limit the quantity or order of those elements. These designations may be used in this disclosure as a convenient method of distinguishing between two or more elements. Thus, a reference to a first and a second element does not imply that only two elements may be employed or that the first element must precede the second element in some way.
[0097] When the terms "include," "including," and variations thereof are used in this disclosure, these terms are intended to be inclusive, similar to the term "comprising." Further, when used in this disclosure, the term "or" is not intended to be an exclusive or.
[0098] In this disclosure, where articles have been added due to translation, such as a, an, and the in English, the disclosure may include that the nouns following these articles are in the plural form.
[0099] In the present disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "coupled" may also be interpreted in the same way as "different." [Explanation of symbols]
[0100] 1, 1A, 1B, 1C...online dialogue support system, 10, 10A, 10B, 10C...sending user terminal, 11...imaging unit, 12, 12B, 12C...transmission unit, 20, 20A, 20B, 20C...receiving user terminal, 21, 21B, 21C...receiving unit, 22...generation unit, 23...authentication unit, 24...display control unit, 30...memory unit, 40...server, 41...image receiving unit, 42...image transmitting unit, D...head-mounted display, R...part of the area.
Claims
1. An online dialogue support system for supporting an online dialogue between a sending user terminal and a receiving user terminal, comprising: a storage unit for storing a reference face image representing a face of the sending user; an acquisition unit that acquires a hidden face image showing the face of the sending user in a state where a part of the face of the sending user is hidden; a generation unit that complements the partial area of the hidden face image to generate a complemented face image; an authentication unit that performs authentication based on the reference face image and the complemented face image; a display control unit that displays the complemented face image on the terminal of the receiving user when the authentication is successful, and displays the hidden face image on the terminal of the receiving user when the authentication is unsuccessful; An online dialogue support system comprising:
2. the partial area is hidden by a head mounted display worn by the transmitting user, The generating unit replaces a portion of the hidden face image corresponding to the head mounted display with another image, thereby complementing the partial area. The online dialogue support system according to claim 1.
3. the generating unit specifies a region of the hidden face image corresponding to the head mounted display based on a pre-stored shape of the head mounted display, thereby specifying the partial region; The online dialogue support system according to claim 2.
4. the generation unit prepares in advance a model configured to input an image showing a part of the face of the sending user and output an estimation result of the other part of the face of the sending user by machine learning using a plurality of face images of the sending user and face images of a plurality of users different from the sending user as teacher data, inputs an area of the hidden face image excluding the portion into the model, and complements the portion based on the output result from the model; The online dialogue support system according to claim 1.
5. the generation unit prepares in advance a model configured to input an image of a first range indicating a part of the partial area and output an estimation result of a second range indicating a part of the partial area different from the first range by machine learning using a plurality of facial images of the sending side user as well as facial images of a plurality of users different from the sending side user as teacher data, inputs an image indicating a part of the face of the sending side user within the partial area to the model, and complements the partial area based on the input image and the output result from the model; The online dialogue support system according to claim 1.
Citation Information
Patent Citations
Mask-wearing face recognition method, system, computer equipment and storage medium
CN112597867A
Method and device for synthesizing facial image of person wearing head mount display
JP1999096366A
Image processor for video camera and head mounted display
JP2000004395A
Image authentication apparatus
JP2007148872A
Portable terminal
JP2009135705A
