Information processing system, information processing apparatus, information processing method, and program

Abstract

To determine the identity of a participant in an on-line conference, considering the nature of the conference.SOLUTION: An information processing system has: a motion information storage unit that stores motion information of a first user generated from image data obtained by photographing the first user in a first on-line conference in which the first user participates, and conference information indicating the nature of the first online conference, in association with each other; an acquisition unit that acquires image data obtained by photographing a participant participating as the first user in a second online conference being held; a determination unit that determines the identity of the participant with the first user, on the basis of the motion information stored in the motion information storage unit in association with conference information, in the conference information of the first on-line conference, having high similarity to conference information indicating the nature of the second on-line conference, and motion information of the participant generated from the image data acquired by the acquisition unit; and a control unit that performs control of the second on-line conference on the basis of a result of determination made by the determination unit.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an information processing system, an information processing device, an information processing method, and a program. [Background technology]

[0002] Deepfake technology is a new technology born from the evolution of AI, and it has raised concerns about the problem of impersonation. Impersonation mainly occurs in online spaces such as online conferences, where people, voices, and images are elaborately synthesized or manipulated to create highly sophisticated impersonations that make it difficult to distinguish between them.

[0003] Conventionally, a technique has been disclosed for collecting behavior information and determining the identity of participants in a remote dialogue system (Patent Document 1). Summary of the Invention [Problem to be solved by the invention]

[0004] The behavior of participants in online conferences is not necessarily constant in each online conference, but may differ depending on the nature of the online conference.

[0005] However, prior art techniques for determining participant identity do not take into account the nature of the conference.

[0006] The present invention has been made in view of the above points, and has as its object to determine the identities of participants in an online conference taking into account the characteristics of the conference. [Means for solving the problem]

[0007] In order to solve the above problem, the information processing system includes: a motion information storage unit that stores motion information of a first user generated from image data of the first user in a first online conference in which the first user participated, in association with conference information indicating the nature of the first online conference; an acquisition unit that acquires image data of a participant participating as the first user in a second online conference that is being held; a determination unit that determines the identity of the participant and the first user based on the motion information stored in the motion information storage unit and associated with conference information of the first online conference that has a high similarity to conference information indicating the nature of the second online conference, and the motion information of the participant generated from the image data acquired by the acquisition unit; and a control unit that controls the second online conference based on the determination result of the determination unit. [Effects of the Invention]

[0008] The identity of participants in an online meeting can be determined by taking into account the nature of the meeting. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a diagram illustrating an example of a configuration of an information processing system according to an embodiment of the present invention. [Figure 2] 1 is a diagram illustrating an example of a hardware configuration of an information processing device 10 according to an embodiment of the present invention. [Figure 3] 1 is a diagram illustrating an example of a functional configuration of an information processing system according to an embodiment of the present invention. [Figure 4] FIG. 10 is a sequence diagram illustrating an example of a processing procedure for extracting user motion information. [Figure 5] FIG. 2 is a diagram illustrating an example of the configuration of a conference data storage unit 151. [Figure 6] FIG. 2 is a diagram illustrating an example of the configuration of an operation information storage unit 152. [Figure 7] FIG. 10 is a sequence diagram illustrating an example of a processing procedure for determining whether user X is identical to user A. [Figure 8] FIG. 10 is a diagram showing an example of a screen displayed on a terminal device 20 of a user X during an online conference. [Figure 9] 10 is a flowchart illustrating an example of a procedure for determining identity. [Figure 10] 10 is a flowchart illustrating an example of a processing procedure of a process according to a control instruction. [Figure 11] FIG. 10 is a diagram showing a first example of an identity verification screen. [Figure 12] FIG. 10 is a diagram showing a second example of an identity verification screen. [Figure 13] FIG. 10 is a diagram showing a first display example of notification information. [Figure 14] FIG. 10 is a diagram showing a second display example of notification information. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Fig. 1 is a diagram showing an example of the configuration of an information processing system according to an embodiment of the present invention. In Fig. 1, a plurality of terminal devices 20 are connected to an information processing device 10 via a network such as a LAN or the Internet.

[0011] The terminal device 20 is a terminal used by a user to participate in an online conference such as a video conference or a web conference. For example, a PC (Personal Computer), a smartphone, or a tablet terminal may be used as the terminal device 20. The terminal device 20 may have a built-in camera, microphone, and speaker, or may be connectable to an external camera, microphone, and speaker. The camera is used to capture an image of the user. The microphone is used to collect the user's speech. The speaker is used to output the voices of other participants in the online conference.

[0012] The information processing device 10 is one or more computers that determine whether or not a user participating in an online conference is the user himself / herself. In this embodiment, determining whether or not a user participating in an online conference is the user himself / herself is referred to as determining the identity of the user. The information processing device 10 controls the terminal device 20 of a user whose identity is denied so as to prevent the user from participating in the online conference. The information processing device 10 may also function as a server for the online conference (hereinafter referred to as a "conference server"), or a computer separate from the information processing device 10 may function as the conference server. The conference server refers to a computer that mediates the exchange of image data (video data) and audio data between the terminal devices 20 of each user participating in the online conference.

[0013] Fig. 2 is a diagram showing an example of the hardware configuration of information processing device 10 according to an embodiment of the present invention. As shown in Fig. 2, information processing device 10 is constructed by a computer and includes CPU 101, ROM 102, RAM 103, HD 104, HDD (Hard Disk Drive) controller 105, display 106, external device connection I / F (Interface) 108, network I / F 109, data bus 110, keyboard 111, pointing device 112, DVD-RW (Digital Versatile Disk Rewritable) drive 114, and media I / F 116. However, when information processing device 10 is operated via a network, information processing device 10 does not need to include display 106, keyboard 111, pointing device 112, etc.

[0014] The CPU 101 controls the overall operation of the information processing device 10. The ROM 102 stores programs, such as an IPL, used to drive the CPU 101. The RAM 103 is used as a work area for the CPU 101. The HD 104 stores various data, such as programs. The HDD controller 105 controls the reading and writing of various data from and to the HD 104 under the control of the CPU 101. The display 106 displays various information, such as a cursor, menus, windows, characters, or images. The external device connection I / F 108 is an interface for connecting various external devices. In this case, the external devices are, for example, USB (Universal Serial Bus) memories, printers, etc. The network I / F 109 is an interface for data communication using the communication network 100. The data bus 110 is an address bus, a data bus, etc. for electrically connecting the components, such as the CPU 101, shown in FIG. 2.

[0015] The keyboard 111 is a type of input means having multiple keys for inputting characters, numbers, various instructions, etc. The pointing device 112 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The DVD-RW drive 114 controls reading and writing of various data from a DVD-RW 113, which is an example of a removable recording medium. Note that this is not limited to a DVD-RW, and may be a DVD-R, etc. The media I / F 116 controls reading and writing (storing) of data from a recording medium 115, such as a flash memory.

[0016] The terminal device 20 may also have the same hardware configuration as the information processing device 10.

[0017] Fig. 3 is a diagram showing an example of the functional configuration of an information processing system according to an embodiment of the present invention. In Fig. 3, a terminal device 20 includes a conference application 21, a generating unit 22, a transmitting unit 23, and a controlled unit 24. Each of these units is realized by a process executed by a CPU of the terminal device 20 by one or more programs installed in the terminal device 20.

[0018] The conference application 21 is an application used for online conferences. The conference application 21 transmits image data (video data) captured by a camera of the terminal device 20 and audio collected by a microphone of the terminal device 20 to the conference server. The conference application 21 also receives image data (video data) and audio data from the conference server, displays the image data on the terminal device 20, and outputs the audio data from a speaker of the terminal device 20. When an online conference is held via a browser, the processing of the conference application 21 is realized by the conference server.

[0019] The generation unit 22 generates data (hereinafter referred to as "conference data") including image data of a user captured by a camera of the terminal device 20 during an online conference and conference information of the online conference. The user image data and the conference information may be acquired from the conference app 21. The online conference in this embodiment may be held using avatars. In this case, the image data may be image data of avatars. The conference information is information indicating the nature of the conference (including the situation, etc.), such as conference holding information. For example, the conference information is information including one or more of the conference type, time period, number of participants, etc. The conference type is a category when conference types are classified based on the format of the conference. Examples of conference types include meeting, presentation, and meeting. The time period is the time period when the conference is held. The number of participants is the number of users participating in the conference. Other information that can define the nature of the online conference may be included in the conference information. The generation unit 22 may be included in the information processing device 10 or the conference server. If a conference server different from the information processing device 10 has the generating unit 22, the conference data generated by the generating unit 22 is transmitted from the conference server to the information processing device 10.

[0020] The transmission unit 23 transmits the conference data generated by the generation unit 22 to the information processing device 10.

[0021] The controlled unit 24 receives a control instruction from the information processing device 10 according to the result of the identity determination made by the information processing device 10 based on the conference data, and executes processing according to the control instruction.

[0022] The information processing device 10 has an acquisition unit 11, an extraction unit 12, a determination unit 13, and a control unit 14. Each of these units is realized by processing that is executed by a CPU 101 of one or more programs installed in the information processing device 10. The information processing device 10 also uses a conference data storage unit 151 and an action information storage unit 152. Each of these storage units can be realized using, for example, the HD 104 or a storage device connectable to the information processing device 10 via a network.

[0023] The acquisition unit 11 acquires (receives) conference data (image data of the user of the terminal device 20 and conference information) transmitted by the transmission unit 23 of the terminal device 20 every time an online conference is held. The acquisition unit 11 records the acquired conference data in the conference data storage unit 151. Therefore, the conference data storage unit 151 stores conference data for each online conference.

[0024] For each piece of conference data stored in the conference data storage unit 151, the extraction unit 12 extracts motion information of a user associated with the image data from image data of the user included in the conference data. The motion information is, for example, information (feature amount) indicating characteristics (patterns) of a motion or behavior. The motion information is used for behavioral biometric authentication. Therefore, the motion information may be information in a format appropriate for the behavioral biometric authentication to be used. The extraction unit 12 associates the motion information extracted from the image data with conference information included in the same conference data as the image data, and records the information in the motion information storage unit 152. Therefore, for each user, the motion information storage unit 152 stores, for each online conference that the user previously participated in, the motion information of the user based on the image data of the user in the online conference in association with the conference information of the online conference.

[0025] The determination unit 13 determines whether each user participating in the ongoing online conference is the person in question (identity determination) by taking into account the conference information. More specifically, for each user participating in the ongoing online conference, the determination unit 13 determines whether the user is the same as a participant participating in the ongoing online conference as the user in question, based on the operation information stored in the operation information storage unit 152 in association with some conference information of online conferences in which the user has previously participated that has a relatively high similarity to the conference information of the ongoing online conference, and the operation information based on image data included in the conference data acquired by the acquisition unit 11 for the ongoing online conference.

[0026] Here, the reason why the behavior information associated with a portion of conference information that has a relatively high similarity to the conference information of the online conference currently being held is used to determine identity is because the behavior information from past online conferences similar in nature to the online conference currently being held is used to determine identity. This embodiment is based on the idea that the behavior (including facial expressions, attitudes, behavior, etc.) of users participating in an online conference may differ depending on the nature of the online conference. For example, a user who actively speaks in an online conference with a small number of participants may hardly speak in an online conference with a large number of participants. User behavior may also change depending on the time of day. User behavior may also differ depending on the type of conference. This embodiment focuses on such tendencies in user behavior and determines identity using behavior information from past online conferences similar in nature to the online conference currently being held.

[0027] The control unit 14 performs control according to the determination result by the determination unit 13. Control according to the determination result means controlling the online conference currently being held based on the determination result. For example, if the identity of a participant participating in an online conference currently being held as a certain user is denied (if the certain user and the participant are determined to be different persons), the control unit 14 controls the participant not to participate in the online conference. Controlling the participant not to participate means controlling the participant to leave the online conference. Control to prevent the participant from participating in the online conference includes controlling the output of information indicating that the participant is not the certain user and controlling the authentication of the participant. As an example of controlling the authentication of the participant, the control unit 14 controls the output of a predetermined question to the participant. On the other hand, if the identity is confirmed (if the certain user and the participant are determined to be the same person), the control unit 14 controls the participant to continue participating in the online conference. The control by the control unit 14 is realized by transmitting an instruction (control instruction) for performing the control to the controlled unit 24.

[0028] The following describes the processing procedure executed in the information processing system. Fig. 4 is a sequence diagram for explaining an example of the processing procedure when extracting user motion information. The extraction of user motion information refers to a phase in which motion information of a certain user is extracted for an online conference (an example of a first online conference) being held in a situation where the user's identity is guaranteed. Fig. 4 focuses on one user (hereinafter referred to as "user A" and an example of a first user), but similar processing is also executed in parallel for other users participating in the same online conference as user A. That is, the terminal device 20 in Fig. 4 is the terminal device 20 used by user A.

[0029] In step S101, the generation unit 22 of the terminal device 20 generates conference data for an ongoing online conference, which is a set of image data (video data) of user A and conference information of the online conference. Step S101 may be executed at the end of the online conference or at some point during the online conference. If the time is at the end of the online conference, image data of user A taken throughout the entire online conference may be included in the conference data. If the time is during the online conference, image data of user A taken during the period up to that time may be included in the conference data.

[0030] The generation unit 22 outputs the generated conference data to the transmission unit 23 (S102). The transmission unit 23 adds identification information of user A (hereinafter referred to as "user ID") to the conference data output from the generation unit 22, and transmits the conference data to the information processing device 10 (S103). Note that the user ID may be identification information of the terminal device 20, or may be account information of user A for the conference app 21, etc.

[0031] Upon receiving the conference data, the acquisition unit 11 of the information processing device 10 records the conference data in the conference data storage unit 151 (S104).

[0032] FIG. 5 is a diagram illustrating an example of the configuration of the conference data storage unit 151. The conference data storage unit 151 stores a table T1, as shown in FIG. 5, for each user. The table T1 shown in FIG. 5 is a table corresponding to user A. As shown in FIG. 5, table T1 stores, for each online conference in which user A participates, a "data ID," "image data," "conference type," "time slot," and "number of participants." The "data ID" is an item for distinguishing each record in the conference data storage unit 151, and its value may be generated by the acquisition unit 11. The "image data" is a file name that stores image data included in the conference data. The "conference type," "time slot," and "number of participants" are conference information included in the conference data. In step S104, the acquisition unit 11 adds a new record based on the received conference data to table T1 corresponding to the user ID added to the conference data.

[0033] Next, the extraction unit 12 extracts motion information from the image data included in the record (hereinafter referred to as the "target record") added to the table T1 (S106). The extracted motion information is motion information generated from image data capturing an image of the user. The extraction of the motion information may be performed based on known techniques related to behavioral biometric authentication.

[0034] Next, the extraction unit 12 associates the action information with the conference information of the target record and records them in the action information storage unit 152 (S107).

[0035] FIG. 6 is a diagram illustrating an example of the configuration of the motion information storage unit 152. The conference data storage unit 151 stores a table T2, as shown in FIG. 6, for each user. The table T2 shown in FIG. 6 corresponds to user A. As shown in FIG. 6, table T2 stores, for each online conference, "motion information," "conference type," "time slot," and "number of participants." The "motion information" is the name of the data storing the motion information extracted from the image data. The reason why one record contains multiple names is to take into consideration the possibility that multiple pieces of data constituting the motion information exist. As long as the motion information can be expressed by one piece of data, the value (data name) of the "motion information" may be one. The "conference type," "time slot," and "number of participants" are conference information corresponding to the "motion information." In step S107, the acquisition unit 11 adds a new record to table T2 corresponding to the user ID related to table T1 to which the target record belongs (table T2 corresponding to user A in FIG. 4).

[0036] Each time user A participates in an online conference, the user A's behavior information is stored in table T2 in association with the conference information. Therefore, the behavior information of user A corresponding to online conferences of various types is stored in table T2.

[0037] Next, a processing procedure for determining the identity of a participant participating in an online conference as user A will be described. This online conference is a different online conference from the online conference that was being held when the processing procedure described in FIG. 4 was executed, and is an example of a second online conference. A participant participating in an online conference as user A is a participant who is logged in using user A's account, or a participant who is participating in the online conference using user A's terminal device 20, and who is recognized as user A by other users participating in the online conference. This participant may be user A himself or herself, or may be impersonating user A using Deepfake technology or the like. This participant will be referred to as "user X" hereinafter.

[0038] 7 is a sequence diagram for explaining an example of a processing procedure for determining whether user X is identical to user A. In FIG. 7, steps having the same processing content as those in FIG. 4 are assigned the same step numbers, and their explanations will be omitted as appropriate. The terminal device 20 in FIG. 7 is the terminal device 20 used by user X.

[0039] 7, the steps enclosed by dashed lines (S101 to S106, S201 to S203) are loop processes for each time interval during which sufficient image data (e.g., video data for a predetermined period of time) is captured to determine identity. Hereinafter, the time interval to be processed is referred to as the target time interval.

[0040] Steps S101 to S106 are as described in FIG. 4. As a result, conference data including image data of user X in the target time period and conference information of the online conference in which user X is participating is stored in the conference data storage unit 151. At this point, user X is also recognized as user A by the information processing system, so the user ID of user A is added to the conference data. Therefore, the conference data is added to table T1 corresponding to user A (S101 to S105). Furthermore, the extraction unit 12 extracts motion information of user X from the image data included in the conference data (S106). The motion information is information indicating the characteristics of the motion of user X in the target time period. The motion information X is information generated from image data of user X.

[0041] At this point, for example, a screen of the conference application 21 as shown in Fig. 8 is displayed on the terminal device 20 of user X. Fig. 8 shows an example in which user X is in an online conference with user B and user C. Therefore, images of user B and user C are displayed. In addition, an image of user X himself is displayed in the upper left.

[0042] Next, the extraction unit 12 inputs the action information of user X (hereinafter referred to as "action information X"), the conference information included in the conference data (hereinafter referred to as "conference information X"), and the user ID of user A to the determination unit 13 (S201).

[0043] When the motion information X, the conference information X, and the user ID are input, the determination unit 13 executes identity determination processing based on the input information (S202). The identity determination processing will be described using a flowchart.

[0044] FIG. 9 is a flowchart illustrating an example of a procedure for determining identity.

[0045] The determination unit 13 waits for input of the action information X, the conference information X, and the user ID (S301). In step S202 of Fig. 7, when the action information X, the conference information X, and the user ID are input (Yes in S301), the determination unit 13 acquires the action information (hereinafter referred to as "action information A") associated with a part of the conference information (hereinafter referred to as "conference information A") that has a relatively high similarity to the conference information X among the conference information (hereinafter referred to as "conference information A") stored in the table T2 (Fig. 6) that corresponds to the user ID (i.e., corresponding to user A) (S302).

[0046] The similarity between two pieces of conference information, conference information A and conference information X, may be evaluated based on, for example, the number of matching items among the items constituting the conference information. In this embodiment, the items constituting the conference information are three: conference type, time period, and number of participants. For example, for each item, 1 point is assigned if there is a match, and 0 point is assigned if there is no match, and the total score of the three items is calculated. In this case, 3 points is the maximum value. Therefore, the action information A corresponding to the conference information A that is closest to 3 points is acquired. Note that weighting and priority items may be predetermined for each item constituting the conference information. For example, the determination unit 13 may evaluate the similarity between the two pieces of conference information regardless of the time period and the number of participants, and acquire the action information A corresponding to the conference information A whose conference type matches the conference information X.

[0047] However, because the time period and the number of participants are time ranges or numerical values, there is a possibility that similarity may not be appropriately evaluated by determining whether or not they match. For example, it is inappropriate to evaluate a case where the number of participants is 100 and 101 as well as a case where the number of participants is 100 and 10 as 0 points. Therefore, the determination unit 13 may assign scores according to the difference between the time period and the number of participants. For example, for the time period, the percentage of the time period of the conference information A that overlaps with the time period of the conference information X may be assigned as a score. For the number of participants, the score may be calculated by dividing the difference between the number of participants in the conference information X and the number of participants in the conference information A by the number of participants in the conference information A and then subtracting the result from 1.

[0048] Similarity may also be assessed by other methods.

[0049] Furthermore, the acquired motion information A may be one motion information associated with the conference information A having the highest similarity evaluation, or may be a plurality of motion information associated with each of the plurality of conference information A having the highest N similarity evaluations. Whether or not to acquire a plurality of motion information (or whether or not it is valid) may be determined depending on the behavioral biometric authentication to be used.

[0050] Next, the determination unit 13 determines the identity of user X and user A using behavioral biometric authentication technology based on the motion information X and the motion information A (S304). That is, it is determined whether user X is user A. If the identity is confirmed (No in S304), the determination unit 13 outputs a determination result that confirms the identity to the control unit 14 (S305), and the process returns to step S301. If the identity is denied (Yes in S304), the determination unit 13 determines whether a control execution condition is satisfied (S306). The control execution condition refers to a condition that must be satisfied in order for the control unit 14 to execute control in response to a denial of identity.

[0051] For example, the control execution condition may be that identity is denied once. However, in this case, depending on the accuracy of behavioral biometric authentication, control based on erroneous determinations may be frequently performed. Therefore, a condition that takes into account erroneous determinations in behavioral biometric authentication, such as that identity is denied α consecutive times or that the rate of erroneous determinations in the past β identity determinations is γ% or more, may be set as the control execution condition.

[0052] If the control execution condition is not satisfied (No in S306), the determination unit 13 outputs a determination result that affirms identity to the control unit 14 (S305), and the process returns to step S301. Note that if the process returns to step S301, the processing procedure in FIG. 10 is executed again when step S202 in FIG. 7 is executed in the time interval next to the target time interval.

[0053] On the other hand, if the control execution condition is satisfied (Yes in S306), the determination unit 13 outputs a determination result denying the identity to the control unit 14 (S307).

[0054] The step of outputting the determination result in step S304 or S307 corresponds to step S203 in FIG.

[0055] In the above example, the motion information extracted based on the image data is associated with the conference information and stored in the motion information storage unit 152, but the present embodiment is not limited to this. When the determination unit 13 determines the similarity of the conference information, the determination unit 13 may refer to the conference data storage unit 151 and acquire image data associated with a portion of the conference information that has a relatively high similarity to the conference information X. In this case, the extraction unit 12 may extract the motion information A based on the acquired image data, and the determination unit 13 may determine the identity based on the extracted motion information A and the motion information X.

[0056] 7, the control unit 14 performs control according to the determination result. If the determination result is positive for identity, the control unit 14 performs control so that user X can continue to participate in the online conference. In this case, the control unit 14 may achieve such control by not sending a special control instruction to the terminal device 20 of user X, or may send a control instruction to the terminal device 20 of user X to permit continued participation in the online conference.

[0057] On the other hand, if the determination result denies identity, step S204 and subsequent steps are executed. In step S204, the control unit 14 transmits a control instruction to the terminal device 20 of the user X to prohibit the user X from participating in the online conference.

[0058] Upon receiving the control instruction, the controlled unit 24 of the terminal device 20 of user X executes processing in accordance with the control instruction (S205). For example, the controlled unit 24 may cause user X to leave the online conference by immediately executing processing to stop user X from using the conference app 21. Alternatively, the controlled unit 24 may execute the processing procedure shown in FIG. 10.

[0059] FIG. 10 is a flowchart illustrating an example of a processing procedure for processing in response to a control instruction.

[0060] In step S401, the controlled unit 24 displays a screen for confirming that the user X is the user A (hereinafter referred to as an "identity confirmation screen").

[0061] Fig. 11 is a diagram showing a first example of an identity verification screen. The identity verification screen 510 shown in Fig. 11 is a screen that requests entry of a password. Note that the identity verification screen 510 includes a message indicating that it is unlikely that user X is user A, as the reason for requesting entry of a password.

[0062] Fig. 12 is a diagram showing a second example of an identity verification screen. The identity verification screen 520 shown in Fig. 12 is a screen that requests input of an answer to a secret question set by user A. Note that the identity verification screen 520 includes a message indicating that user X is unlikely to be user A as the reason for requesting input of the answer.

[0063] When user X inputs a password or an answer to a secret question (Yes in S402), the controlled unit 24 performs authentication by comparing the input information with correct information (S403). If authentication is successful (Yes in S404), the controlled unit 24 hides the identity verification screen and ends the processing procedure in FIG. 10. In this case, since user X is likely to be user A, user X is permitted to continue participating in the online conference. However, step S101 and subsequent steps in FIG. 7 may be repeated.

[0064] On the other hand, if the authentication fails (No in S404), the process returns to step S402, and the controlled unit 24 requests re-input on the identity verification screen. If the correct input is not made within a certain time period after the start of the processing procedure in FIG. 10 (Yes in S406), the controlled unit 24 executes processing to make the user X leave the conference (S407). For example, the controlled unit 24 may terminate the conference application 21.

[0065] 7, the control unit 14 of the information processing device 10 may perform control to prevent user X from participating in the online conference by transmitting notification information to the terminal devices 20 of users other than user X (user B, user C) indicating that user X is unlikely to be user A. Upon receiving the notification information, the controlled unit 24 of the terminal device 20 displays the notification information (S207).

[0066] Fig. 13 is a diagram showing a first display example of notification information. Fig. 13 shows a display example on the terminal device 20 of user B. The notification screen 530 shown in Fig. 13 includes a message notifying that user A is unlikely to be the person in question. User B or user C who has viewed the notification screen 530 can avoid leaks of confidential information, etc. by asking user X questions to verify his or her identity or interrupting the online conference.

[0067] FIG. 14 is a diagram showing a second display example of notification information. FIG. 14 shows a display example on the terminal device 20 of user B. The notification screen 540 shown in FIG. 14 includes a message notifying that user A is unlikely to be the person in question, and a button for selecting whether to forcibly remove the participant claiming to be user A from the online conference. User B or user C who refers to the notification screen 540 can avoid the leakage of confidential information, etc., by asking user X questions for identity verification, suspending the online conference, or forcibly removing user X. When forced removal is selected by one or more users, or by all users other than user X, the control unit 14 may transmit a control instruction indicating forced removal to the terminal device 20 of user X. In this case, the controlled unit 24 of the terminal device 20 of user X executes a process for removing user X from the conference. When the control unit 14 controls to display the notification screen 530 or the notification screen 540 (when sending notification information), the decision to have the user X leave or interrupt the online conference can be left to a user other than the user X. Therefore, in this case, steps S204 and S205 in FIG. 7 may not be executed.

[0068] The present embodiment described above can be applied to the following specific scenarios, for example.

[0069] [Scene 1] Suppose that another person X, pretending to be a certain person A, joins a web conference by disguising his or her own face as A's face and turning on the camera. X, who appears to other participants through the video of the web conference, is recognized as A. The information processing device 10 repeatedly determines whether X is identical to A (identity determination based on A's behavior pattern). If it is determined that X is not A, an identity verification screen is displayed on the terminal device 20 used by X. If X cannot enter the correct information on the identity verification screen within a certain time, X will be forced to leave the web conference.

[0070] [Scene 2] Suppose that when two avatars, A and B, communicate in a virtual space, X, disguised as B's avatar, is impersonating B. To A, X appears to be B, but it is actually X who is playing B's avatar. The information processing device 10 repeatedly determines whether X is identical to B (identity determination based on B's movement pattern). If it is determined that X is not B, an identity verification screen is displayed on the terminal device 20 used by X. If X fails to enter the correct information on the identity verification screen within a certain period of time, X is also logged out from the avatar.

[0071] As described above, according to this embodiment, the identity of a participant who is participating in an online conference as a certain user in an ongoing online conference is determined based on the behavioral information of the certain user in a past online conference whose characteristics are the same as or similar to the characteristics of the ongoing online conference. Therefore, the identity of the participant in the online conference can be determined taking into account the characteristics of the conference. As a result, for example, effective detection of impersonation can be expected.

[0072] The terminal device 20 may be, for example, an output device such as a PJ (Projector), an IWB (Interactive White Board: an electronic white board with a blackboard function that allows mutual communication), digital signage, a HUD (Head Up Display) device, industrial machinery, an imaging device, a sound collection device, medical equipment, network home appliances, a notebook PC (Personal Computer), a mobile phone, a smartphone, a tablet terminal, a game console, a PDA (Personal Digital Assistant), a digital camera, a wearable PC or a desktop PC.

[0073] In addition, tables in the statement (e.g., Table T2) may be generated by the learning effect of machine learning. Also, by classifying keywords and account items that may be included in the description of transaction details using machine learning, it is possible to eliminate the need for a correspondence table.

[0074] Here, machine learning is a technology that allows a computer to acquire human-like learning capabilities, where the computer autonomously generates the algorithms necessary for making judgments such as data classification from training data that is input in advance, and applies these to new data to make predictions. The learning method for machine learning may be any of supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, or deep learning, or may be a combination of these learning methods; the learning method for machine learning is not important.

[0075] Furthermore, each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), and conventional circuit modules designed to perform each of the above-described functions.

[0076] Additionally, the devices described in the embodiments are merely representative of one of several computing environments for implementing the embodiments disclosed herein.

[0077] In one embodiment, information processing apparatus 10 includes multiple computing devices, such as a server cluster, configured to communicate with each other over any type of communications link, including a network, shared memory, etc., to perform the processes disclosed herein. Similarly, terminal apparatus 20 may include multiple computing devices configured to communicate with each other.

[0078] In the above-described embodiment, an example has been described in which the motion information storage unit 152 stores motion information for each previously held conference. However, this is not limiting. When the conference information of previously held conferences is compared with each other and there is a past conference with matching conference information, motion information may be extracted from the image data of the conference, and the motion information and the conference information may be stored as a single record. In other words, instead of storing motion information for each previously held conference, motion information may be stored for each piece of uniquely identified conference information. As a result, for example, if multiple conferences with the same conference information have been held in the past, motion information may be extracted based on those multiple conferences, and motion information indicating user A's behavior for each type of conference indicated in the conference information may be accumulated.

[0079] Although the embodiments of the present invention have been described in detail above, the present invention is not limited to such specific embodiments, and various modifications and variations are possible within the scope of the gist of the present invention as set forth in the claims.

[0080] For example, aspects of the present invention are as follows. <1> an action information storage unit that stores action information of a first user generated from image data of the first user in a first online conference in which the first user participated, in association with conference information indicating the nature of the first online conference; an acquisition unit that acquires image data of a participant who is participating as the first user in a second online conference that is being held; a determination unit that determines whether the participant is the first user based on the motion information stored in the motion information storage unit in association with conference information of the first online conference that has a high similarity to conference information indicating the nature of the second online conference, and on the motion information of the participant generated from the image data acquired by the acquisition unit; and a control unit that controls the second online conference based on the determination result of the determination unit; An information processing system comprising: <2> the control unit controls the participant to leave the second online conference when the identity is denied. characterized in that it has <1> An information processing system according to claim 1. <3> When the identity is confirmed, the control unit controls the participant to continue participating in the second online conference. Characterized by <2> An information processing system according to claim 1. <4> the control unit controls output of information indicating that the participant is not the first user when the identity is denied. Characterized by <2> or <3> An information processing system according to claim 1. <5> the control unit controls to authenticate the participant when the identity is denied. Characterized by <2> ~ <4> 10. The information processing system according to claim 9, wherein: <6> the control unit controls output of a predetermined question to the participant when the identity is denied. Characterized by <5> An information processing system according to claim 1. <7> The conference information includes at least one of a conference type, a time period, and a number of participants. Characterized by <1> ~ <6> 10. An information processing system according to claim 9, wherein: <8> the determination unit evaluates the similarity based on a difference in a time period or a number of participants of the conference included in the conference information. Characterized by <1> ~ <7> 10. An information processing system according to claim 9, wherein: <9> The first online conference and the second online conference are online conferences using avatars. Characterized by <1> ~ <8> 10. The information processing system according to claim 9, wherein: <10> A terminal device used by a user participating in the first online conference or the second online conference, Characterized by <1> ~ <9> 10. The information processing system according to claim 9, wherein: <11> The determination unit determines whether the participant is the first user based on the action information stored in the action information storage unit in association with the first online conference, the conference information of which includes a conference type that matches a conference type included in the conference information of the second online conference, among conference types included in the conference information of the first online conference, and the action information of the participant based on the image data acquired by the acquisition unit. Characterized by <1> ~ <10> 10. The information processing system according to claim 9, wherein: <12> The determination unit determines whether the participant is the first user based on the motion information stored in the motion information storage unit in association with conference information of the first online conference that has a relatively large number of items matching the items constituting the conference information of the second online conference, and the motion information of the participant based on the image data acquired by the acquisition unit. Characterized by <1> ~ <11> 10. The information processing system according to claim 9, wherein: <13> an action information storage unit that stores action information of a first user generated from image data of the first user in a first online conference in which the first user participated, in association with conference information indicating the nature of the first online conference; an acquisition unit that acquires image data of a participant who is participating as the first user in a second online conference that is being held; a determination unit that determines whether the participant is the first user based on the motion information stored in the motion information storage unit in association with conference information of the first online conference that has a high similarity to conference information indicating the nature of the second online conference, and on the motion information of the participant generated from the image data acquired by the acquisition unit; and a control unit that controls the second online conference based on the determination result of the determination unit; An information processing device comprising: <14> a storage step of storing, in a motion information storage unit, motion information of the first user generated from image data of the first user in a first online conference in which the first user participated, in association with conference information indicating the nature of the first online conference; an acquisition step of acquiring image data of a participant who is participating as the first user in a second online conference that is being held; a determination step of determining whether the participant is the first user based on the motion information stored in the motion information storage unit in association with conference information of the first online conference that has a high similarity to conference information indicating the nature of the second online conference, and on the motion information of the participant generated from the image data acquired in the acquisition step; and a control procedure for controlling the second online conference based on a determination result of the determination procedure; An information processing method characterized by being executed by a computer. <15> a storage step of storing, in a motion information storage unit, motion information of the first user generated from image data of the first user in a first online conference in which the first user participated, in association with conference information indicating the nature of the first online conference; an acquisition step of acquiring image data of a participant who is participating as the first user in a second online conference that is being held; a determination step of determining whether the participant is the first user based on the motion information stored in the motion information storage unit in association with conference information of the first online conference that has a high similarity to conference information indicating the nature of the second online conference, and on the motion information of the participant generated from the image data acquired in the acquisition step; and a control procedure for controlling the second online conference based on a determination result of the determination procedure; A program characterized by causing a computer to execute the above. [Explanation of symbols]

[0081] 10. Information processing equipment 11 Acquisition Department 12 Extraction part 13 Judgment section 14 Control Unit 20 Terminal equipment 21. Meeting Apps 22 Generation part 23 Transmitter 24 Controlled part 101 CPU 102 ROM 103 RAM 104 HD 105 HDD (Hard Disk Drive) Controller 106 Display 108 External device connection I / F (Interface) 109 Network I / F 110 Data Bus 111 keyboard 112 Pointing Device 114 DVD-RW (Digital Versatile Disk Rewritable) drive 115 Recording Media 116 Media I / F 151 Conference Data Storage Unit 152 Operation information storage unit [Prior art documents]

Charter Documents

[0082] [Patent Document 1] Patent Gazette No. 6901190

Claims

1. an action information storage unit that stores action information of a first user generated from image data of the first user in a first online conference in which the first user participated, in association with conference information indicating the nature of the first online conference; an acquisition unit that acquires image data of a participant who is participating as the first user in a second online conference that is being held; a determination unit that determines whether the participant is the first user based on the motion information stored in the motion information storage unit in association with conference information of the first online conference that has a high similarity to conference information indicating the nature of the second online conference, and on the motion information of the participant generated from the image data acquired by the acquisition unit; and a control unit that controls the second online conference based on a determination result of the determination unit; An information processing system comprising:

2. the control unit controls the participant to leave the second online conference when the identity is denied.

2. The information processing system according to claim 1, further comprising:

3. When the identity is confirmed, the control unit controls the participant to continue participating in the second online conference.

3. The information processing system according to claim 2.

4. the control unit controls output of information indicating that the participant is not the first user when the identity is denied.

3. The information processing system according to claim 2.

5. the control unit controls to authenticate the participant when the identity is denied.

3. The information processing system according to claim 2.

6. the control unit controls output of a predetermined question to the participant when the identity is denied.

6. The information processing system according to claim 5.

7. The conference information includes at least one of a conference type, a time period, and a number of participants.

2. The information processing system according to claim 1, wherein:

8. the determination unit evaluates the similarity based on a difference in a time period or a number of participants of the conference included in the conference information.

2. The information processing system according to claim 1, wherein:

9. the first online conference and the second online conference are online conferences using avatars; 2. The information processing system according to claim 1, wherein:

10. a terminal device used by a user participating in the first online conference or the second online conference; 2. The information processing system according to claim 1, wherein:

11. The determination unit determines whether the participant is the first user based on the action information stored in the action information storage unit in association with the first online conference, the conference information of which includes a conference type that matches a conference type included in the conference information of the second online conference, among conference types included in the conference information of the first online conference, and the action information of the participant based on the image data acquired by the acquisition unit.

2. The information processing system according to claim 1, wherein:

12. The determination unit determines whether the participant is the first user based on the motion information stored in the motion information storage unit in association with conference information of the first online conference that has a relatively large number of items matching the items constituting the conference information of the second online conference, and the motion information of the participant based on the image data acquired by the acquisition unit.

2. The information processing system according to claim 1, wherein:

13. an action information storage unit that stores action information of a first user generated from image data of the first user in a first online conference in which the first user participated, in association with conference information indicating the nature of the first online conference; an acquisition unit that acquires image data of a participant who is participating as the first user in a second online conference that is being held; a determination unit that determines whether the participant is the first user based on the motion information stored in the motion information storage unit in association with conference information of the first online conference that has a high similarity to conference information indicating the nature of the second online conference, and on the motion information of the participant generated from the image data acquired by the acquisition unit; and a control unit that controls the second online conference based on a determination result of the determination unit; An information processing device comprising:

14. a storage step of storing, in a motion information storage unit, motion information of the first user generated from image data of the first user in a first online conference in which the first user participated, in association with conference information indicating the nature of the first online conference; an acquisition step of acquiring image data of a participant who is participating as the first user in a second online conference that is being held; a determination step of determining whether the participant is the first user based on the motion information stored in the motion information storage unit in association with conference information of the first online conference that has a high similarity to conference information indicating the nature of the second online conference, and on the motion information of the participant generated from the image data acquired in the acquisition step; and a control procedure for controlling the second online conference based on a determination result of the determination procedure; An information processing method characterized by being executed by a computer.

15. a storage step of storing, in a motion information storage unit, motion information of the first user generated from image data of the first user in a first online conference in which the first user participated, in association with conference information indicating the nature of the first online conference; an acquisition step of acquiring image data of a participant who is participating as the first user in a second online conference that is being held; a determination step of determining whether the participant is the first user based on the motion information stored in the motion information storage unit in association with conference information of the first online conference that has a high similarity to conference information indicating the nature of the second online conference, and on the motion information of the participant generated from the image data acquired in the acquisition step; and a control procedure for controlling the second online conference based on a determination result of the determination procedure; A program characterized by causing a computer to execute the above.

Images