Picture synthesis method using network conference system

Abstract

[Problem] To provide a screen synthesis method using a web conference system, which is capable of extracting and displaying specific participants. [Solution] A screen synthesis method using a network conference system, comprising: a data display step in which the network conference system displays data for a network conference on a display unit; a participant image reception step for receiving, for a plurality of participants participating in the web conference, an image in which each participant is captured; a participant selection step for receiving a result of selection of two or more participants among the participants who have received the image in the participant image reception step; and a selected participant display step in which, for the two or more participants selected in the participant selection step, image regions of the participants are extracted from images in which the participants have been captured, and an image in accordance with the extracted image regions of the participants is displayed on a display unit together with data. In the selected participant display step, an image according to the image region of the participant is displayed on a display unit in accordance with a predetermined display mode.

Description

Technical Field

[0001] The invention relates to a picture synthesis method using a network conference system. Background Art

[0002] Japanese Patent No. 7062126 describes a network conference system using an avatar.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent No. 7062126 Summary of the invention

[0006] Problems to be solved by the invention

[0007] The object of the present invention is to provide a screen synthesis method using a network conference system, which can extract and display specific participants in the network conference system.

[0008] Technical solutions to problems

[0009] The first invention relates to a screen synthesis method using a network conference system 1. The method includes a document display step (S110), a participant image receiving step (S120), a participant selection step (S130), and a selected participant display step (S140).

[0010] The material display step ( S110 ) is a step for the network conference system 1 to display materials used for the network conference on the display unit 3 .

[0011] The participant image receiving step ( S120 ) is a step in which the network conference system 1 receives images of each participant from a plurality of participants participating in the network conference.

[0012] The participant selection step ( S130 ) is a step in which the network conference system 1 receives a result that two or more participants are selected from among the participants received in the participant image receiving step.

[0013] The participant selection and display step ( S140 ) is a step in which the network conference system 1 extracts image areas of the participants from images of the participants selected in the participant selection step, and displays images based on the extracted image areas of the participants together with data on the display unit 3 .

[0014] The selected participant display step displays an image of an image area according to a participant on the display unit 3 according to a predetermined display mode.

[0015] The preferred example of the screen synthesis method further includes a speaker identification step (S131). The speaker identification step (S131) ​​is a step in which the network conference system 1 identifies the speaker who has spoken among the participants participating in the network conference. Then, the participant selection step (S130) selects two or more participants from among the participants using information about the identified speaker.

[0016] In a preferred example of the screen synthesis method, when the network conference system 1 determines that the first participant and the second participant have had a conversation, the participant display step ( S130 ) selects the image based on the image area of ​​the first participant and the image based on the image area of ​​the second participant to be displayed on the display unit 3 .

[0017] In a preferred example of the screen synthesis method, when the network conference system 1 determines that the questioner has asked a question regarding the explanation of the presenter, the participant display step ( S130 ) selects the image based on the image area of ​​the presenter and the image based on the image area of ​​the questioner to display on the display unit 3 .

[0018] A preferred example of the screen composition method further includes a display image storage step ( S150 ).

[0019] The display image storage step ( S150 ) is a step in which the network conference system 1 stores a display image including the data displayed on the display unit 3 in the participant selection display step ( S140 ) and the image region according to the participant.

[0020] A preferred example of the screen composition method further includes a recording information correction step ( S160 ).

[0021] The recorded information correction step ( S160 ) is a step of correcting and storing an image related to the data among the display images stored in the display image storage step ( S150 ) when the data is corrected.

[0022] In an example of a screen synthesis method, the network conference system 1 includes a coding information display step (S210). The coding information display step (S210) is a step of displaying coding information on the display unit 3. The coding information display step (S210) displays the coding information corresponding to each page of the document on the display unit 3. In a preferred example of this step, when the network conference is in progress, the network conference system 1 updates the coding information to obtain the updated coding information, and the network conference system 1 displays the updated coding information on the display unit 3.

[0023] The next invention relates to a program or a non-transitory information recording medium storing the program. The program is a program for causing a computer to execute any of the above-mentioned screen combining methods.

[0024] Effects of the Invention

[0025] A screen synthesis method using a network conference system can be provided, which can extract and display specific participants in the network conference system. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] [ Figure 1 ] Figure 1 This is a flowchart for explaining a screen synthesis method.

[0027] [ Figure 2 ] Figure 2 This is a schematic diagram showing a network conference system.

[0028] [ Figure 3 ] Figure 3 is a block diagram representing a network conferencing system.

[0029] [ Figure 4 ] Figure 4 This is a conceptual diagram showing an example in which documents and participant images are displayed on the administrator screen.

[0030] [ Figure 5 ] Figure 5 This is a diagram showing a situation in which the manager selects participants.

[0031] [ Figure 6 ] Figure 6 FIG. 1 is a diagram showing an example of an image in which image areas according to participants are displayed.

[0032] [ Figure 7 ] Figure 7 This is a diagram different from the above, showing an example of an image in which image areas according to participants are displayed. DETAILED DESCRIPTION

[0033] Hereinafter, the embodiment of the present invention will be described with reference to the drawings. The present invention is not limited to the embodiment described below, and includes contents that can be appropriately modified within a range obvious to those skilled in the art from the embodiment below.

[0034] Figure 1 is a flowchart for explaining the method of picture synthesis. Figure 1 As shown, the method includes a data display step (S110), a participant image receiving step (S120), a participant selection step (S130), and a selected participant display step (S140). Figure 1 As shown, the method may further include any one or more of a speaker identification step (S131), a display image storage step (S150), and a record information correction step (S160). Figure 1The example is an example of a process, and the order may be changed appropriately, or the processes may be performed simultaneously. For example, the participant image receiving process (S120) may be performed after the data display process (S110), or the participant image receiving process (S120) may be performed after the data display process (S110). Figure 1 As shown, the method may also include a coding information display step (S210). In addition, without being limited to the method, the coding information display step (S210) may be used to store and read the display image displayed on the display unit. The method is executed by a computer or a processor.

[0035] A computer or processor has an input unit, an output unit, a control unit, a computing unit, and a storage unit, and each element is connected by a bus or the like, and is made to be able to send and receive information. For example, a program may be stored in the storage unit, and various information may also be stored. When predetermined information is input from the input unit, the control unit reads out the program stored in the storage unit. Then, the control unit appropriately reads out the information stored in the storage unit and transmits it to the computing unit. Again, the control unit appropriately transmits the input information to the computing unit. The computing unit uses the various information received to perform computing processing and stores it in the storage unit. The control unit reads out the computing results stored in the storage unit and outputs it from the output unit. In this way, various processing or processes are performed. The units or components that perform these various processing are each unit. A computer may also have a processor and a memory in which a program has been stored, and these elements realize various functions or various processes. A computer or processor may also improve the accuracy of each processing through machine learning or deep learning. In this case, the obtained data is used to make it learn a learning model, and the accuracy of machine learning or deep learning can be improved.

[0036] Figure 2 This is a schematic diagram showing a network conferencing system. Figure 2 In the example shown, a server 25 and a plurality of terminals (user terminals) 27 are connected via a network (intranet or Internet). The network conference system 1 is a system for connecting a plurality of terminals and conducting a network conference. A program for the network conference system may be installed in each terminal, or in the server.

[0037] Figure 3 is a block diagram representing a network conferencing system. Figure 3 As shown in FIG. 1 , the network conference system 1 includes a document display unit 5, a participant image receiving unit 7, a participant selection unit 9, and a selected participant display unit 11. Figure 3 As shown in FIG. 1 , the system 1 may further include one or more of a speaker identification unit 13, a display image storage unit 15, and a record information correction unit 17. Figure 3 As shown, the method may also include a coded information display unit 21 .

[0038] The material display unit 5 is an element for displaying the material used for the network conference on the display unit 3. The participant image receiving unit 7 is an element for receiving images of each participant for multiple participants participating in the network conference. The participant selection unit 9 is an element for receiving the result of selection of two or more participants from the participants received by the participant image receiving element. The selected participant display unit 11 is an element of the network conference system 1 that extracts the image area of ​​the participant from the image of the participant for two or more participants selected in the participant selection process, and displays the image based on the extracted image area of ​​the participant together with the material on the display unit 3. The selected participant display unit 11 displays the image based on the image area of ​​the participant on the display unit 3 according to a predetermined display mode. Each unit can also be read as a component, and the corresponding process is performed.

[0039] The data display process (S110) is a process for the network conference system 1 to display the data used for the network conference on the display unit 3. The display unit 3 may be a display unit (screen, etc.) of a server or a display unit (screen, etc.) of each terminal. The display unit of the server and the display unit of each terminal may display images based on the same information or images based on different information. A network conference system is a system in which multiple terminals can hold a conference at the same time through a network (the Internet or an intranet). The network conference system itself is known. Examples of network conference systems include ZOOM (registered trademark), Teams (registered trademark), Meet (registered trademark), WebEx (registered trademark), Skype (registered trademark), and LINE (registered trademark) meeting. For example, multiple people participate in a network conference. Then, a speaker (presenter) wants to share a certain data. In this way, for example, the speaker's client computer receives an instruction to read the data and reads the data from the storage unit. The client may be a personal computer (PC) or a handheld terminal such as a smartphone. Then, the client outputs the read data and the sharing instruction to the system. The system that receives the sharing instruction and the data displays the data on the display unit of the system. In addition, the system outputs information for displaying the data to the display unit of each user terminal of the participants participating in the network conference. Examples of the data are demonstration data or various data for sharing with participants. The coded information display step (S210) described later may also be performed simultaneously with the data display step (S110) or after the data display step (S110).

[0040] The participant image receiving process (S120) is a process in which the network conference system 1 receives images of each participant for multiple participants participating in the network conference. In addition, this process does not need to receive photographic images of all the multiple participants participating in the network conference. That is, in the network conference, there are also people who keep their cameras turned off, so there is no need to have photographic images for such people. For example, the photographing unit (camera) 23 of the user end of each participant can also photograph each photographer and transmit the photographed image to the system 1. In this way, the network conference system 1 can receive images of each participant for multiple participants participating in the network conference. In the network conference, if the camera is turned on, the camera will shoot the participant and transmit it to the network conference system. Then, the network conference system shares the image (including continuous images) of the person whose camera is turned on. Therefore, a general network conference system can receive images of each participant for multiple participants participating in the network conference. In addition, the photographic images of the participants can be stored in advance in the storage unit of the system 1, and when the system 1 receives information about the participant, the photographic image of the participant is read out from the storage unit.

[0041] Figure 4 1 is a conceptual diagram showing an example in which documents and participant images are displayed on the administrator screen. In this example, an image 33 related to the documents is displayed on the administrator screen 31, and a photographic image 35 of the participant is displayed. In this example, coded information 37 is displayed on the administrator screen 31. This coded information 37 can also be displayed on the display unit of the terminal of a participant other than the administrator. The system 1 stores the displayed document or participant information in association with this coded information 37, for example. In this way, by using this coded information, the participant can restore the document or participant information or the displayed screen.

[0042] The participant selection process (S130) is a process in which the network conference system 1 receives the result of selecting two or more participants from among the participants whose images are received in the participant image receiving process. For example, the speaker or the manager specifies two or more persons from among the participants included in the image displayed on the display unit. In this way, the system can also receive the result of selecting one or more participants from among the participants whose images are received in the participant image receiving process. This process can be performed automatically by the system or based on the output from the server or the terminal. In addition to the examples described below, the server automatically performs the process in which two or more participants are randomly selected from the participants. In addition, there are examples in which two or more participants are selected. However, one participant can also be selected.

[0043] Figure 5: is a diagram showing a situation where an administrator selects a participant. In this example, the display unit of the administrator's terminal becomes a participant management screen, and a plurality of participants are displayed on the screen. Then, if a photographic image of a participant is selected with a finger, the system receives the input from the finger and performs a process in which the participant is selected. In addition, the system can also specify and select a participant by a mouse or voice. For example, the participant name and participant information (such as the participant's ID) are associated and stored in the storage unit. The system analyzes the input voice, analyzes the name, and stores it appropriately in the storage unit. Then, the system reads out the stored name (the name contained in the input voice) and the stored participant name, and performs a comparison operation. As a result, the system finds the participant name that is consistent with the stored name. Then, the system can use the information of the consistent participant name as described above to read out the information about the participant stored in the storage unit and select the participant. In this way, for example, the host only needs to call the participant name, the called participant will be selected, and the image associated with the participant will be displayed on the display screen.

[0044] The participant selection process (S130) may also include a speaker identification process (S131). The speaker identification process (S131) ​​is a process in which the network conference system 1 identifies the participants who have spoken among the participants participating in the network conference. For example, it is assumed that two or more participants speak in succession. In this case, a voice is input to the user end of the participant. Then, the voice input to the user end and the user end information (participant's information) are transmitted to the system together. In this way, the system receives the voice input to the user end and the user end information (participant's information). The system stores the received voice input to the user end and the user end information (participant's information) in the storage unit. In this way, the system can identify the participants who have spoken among the participants participating in the network conference. Then, the participant selection process (S130) selects the participant using the information about the identified speaker. The system reads the user end information (participant's information) stored in the storage unit, and selects multiple participants using the read user end information (participant's information). At this time, for example, two or more participants who have completed their speeches can also be selected.

[0045] In a preferred example of the screen synthesis method, when the network conference system 1 determines that the first participant and the second participant have a conversation, the participant display step (S130) can also display an image based on the image area of ​​the first participant and an image based on the image area of ​​the second participant on the display unit 3. The system 1 analyzes the terminal to which the voice is input, and when the voice of a predetermined number of people is input within a certain period of time, it is determined that the first participant and the second participant have a conversation. In this way, the system can also display the image based on the image area of ​​the first participant and the image based on the image area of ​​the second participant on the display unit 3.

[0046] In a preferred example of the present screen synthesis method, when the network conference system 1 determines that the questioner has asked a question regarding the presenter's explanation, the network conference system 1 displays an image based on the presenter's image area and an image based on the questioner's image area on the display unit 3 in the participant selection display step (S130).

[0047] The participant selection and display process (S140) is a process in which the network conference system 1 extracts the image area of ​​the participant from the image of the participant for two or more participants selected in the participant selection process, and displays the image based on the extracted image area of ​​the participant together with the data on the display unit 3. The data in this case may also be data displayed on the display unit (a certain page of a certain data or a certain part of a certain data). The participant selection and display process displays the image based on the image area of ​​the participant on the display unit 3 according to a predetermined display mode. An example of a predetermined display mode is to display the photographed image as it is. Another example of a predetermined display mode is to display only the image based on the image area of ​​the participant on the display unit. In a general network conference, the photographed image of the participant is displayed on the display unit as it is (or the image with the background adjusted). In this example, only the image based on the image area of ​​the participant is displayed on the display unit. In this way, a display can be performed as if the selected participants are having a discussion.

[0048] An example of an image based on the image area of ​​the participant may also be an image obtained by removing the background portion from a photographic image taken by the participant's terminal. An example of an image based on the image area of ​​the participant may also be an image obtained by extracting the facial portion of the participant from a photographic image taken by the participant's terminal. It may also be an image area of ​​the participant extracted from a photographic image taken by the participant's terminal, and applying a predetermined processing on this basis. An example of the predetermined processing is processing the facial area of ​​the participant to be larger than the body area, or controlling the opening and closing of the participant's mouth in accordance with a predetermined frequency or speech, or causing the participant's eyes to open and close at a predetermined frequency. Such an image can be obtained by having the storage unit store a program for performing image processing, reading the program by instructions from the control unit, and causing the calculation unit to perform a predetermined calculation.

[0049] Figure 6 FIG. 1 is a diagram showing an example of an image in which image areas according to participants are displayed. Figure 6 In the example, the participant at the top and the participant in the center of the administrator screen are selected, and an image 39 corresponding to the image area of ​​the participant (partial image with the background removed) is extracted through image processing and displayed on the left and right sides of the screen. By doing so, it is possible to display the image 33 of the material as if two participants are discussing or talking about it.

[0050] In another embodiment of the participant selection display process (S140), the network conference system 1 displays images associated with the selected participants on the display unit 3 for one or more participants selected in the participant selection process. For example, the system 1 stores the virtual avatars associated with the participants in the storage unit in advance. Then, based on the information about the one or more participants selected in the participant selection process, the virtual avatars of the corresponding participants are read out from the storage unit. In addition, the system 1 can also appropriately operate the read virtual avatars to display them in a manner as if the virtual avatars of the participants are speaking or talking.

[0051] Figure 7 4 is a diagram showing an example of an image in which an image area according to a participant is displayed. In this example, the hair color of the participant displayed in the center of the administrator screen changes, and an exaggerated image 41 with an enlarged face portion is displayed. In this example, the avatar information of the participant displayed at the top of the administrator screen is read from the storage unit, and the avatar 43 corresponding to the participant is displayed on the screen. In this way, the system can also display the participant in an exaggerated manner or display the avatar. By designing such a display, the audience's attention can be attracted.

[0052] A preferred example of the screen composition method further includes a display image storage step ( S150 ).

[0053] The display image storage process (S150) is a process in which the network conference system 1 stores the data displayed on the display unit 3 in the participant selection display process (S140) and the display image of the image area according to the participant. Since there is a display image storage process, the participant or a third party can obtain the image displayed on the display unit in the future. Moreover, at this time, the sound when the predetermined image is displayed can also be stored in the storage unit. The participant or the third party can reproduce the display image or sound displayed on the display unit. In addition, at this time, the coding information described later is displayed on the display unit, and the display screen can also be reproduced based on the coding information.

[0054] A preferred example of the screen composition method further includes a recording information correction step ( S160 ).

[0055] The recording information correction step (S160) is a step of correcting and storing the image of the document in the display image stored in the display image storage step (S150) when the document is corrected. The storage unit of the system 1 is associated with the display image and stores information about the page or part of the document displayed on the display unit. Then, when the system 1 receives information that the page or part of the document displayed is corrected, it replaces the information of the page or part of the document in the display image stored in the storage unit with the corrected information and stores it in the storage unit as the corrected display image. In this way, when the participant or the third party wants to obtain the display image in the future, the corrected image can be obtained. Therefore, for example, in the case of a speech being wrong, the information being changed due to a revision of the law, the system being changed, etc., the display image of the latest information can be provided. In addition, if the voice data stored in the storage unit is also corrected in conjunction with it, the speech itself can be provided to the participant or the third party. Updated data. Furthermore, if one of the participants displayed on the screen wishes to cancel the display later on, information about other participants (e.g., identification information of other participants) is used to read out the images or avatars of the other participants and replace them with the image of the person who wants to cancel the display, thereby maintaining the content itself and continuing to provide the image information that can be provided.

[0056] In an example of a screen synthesis method, the network conference system 1 includes a coding information display process (S210). The coding information display process (S210) is a process of displaying coding information on the display unit 3. The coding information display process (S210) displays the coding information corresponding to each page of the data on the display unit 3. A preferred example of this process is that when the network conference is in progress, the network conference system 1 updates the coding information and obtains the updated coding information, and the network conference system 1 displays the updated coding information on the display unit 3. This coding information is associated with the data displayed on the display unit and stored in the storage unit of the system 1. Therefore, participants or third parties can obtain the data by referring to this coding information. In addition, as described above, for example, in addition to the page or part of the data, the sound data when the page or part is displayed is also stored in the storage unit in association with the coding information. In this example, by referring to the coding information, not only the page or part of the data, but also the sound when it is displayed can be obtained.

[0057] The next invention relates to a program or a computer-readable non-transitory information recording medium storing the program. The program is a program for causing a computer to execute any of the above-mentioned screen combining methods.

[0058] Industrial Applicability

[0059] This method can be used in network conferencing systems, etc.

[0060] Description of Reference Numerals

[0061] 1: Network conference system

[0062] 5: Data display department

[0063] 7: Participant image receiving unit

[0064] 9: Participant Selection

[0065] 11: Select the participant display area

[0066] 13: Speaker Identification

[0067] 15: Display image storage unit

[0068] 17: Record Information Correction Department

[0069] 21: Coded information display unit

Claims

1. A method for synthesizing images using a network conference system, characterized in that: Include: A data display step, wherein the network conference system displays the data used for the network conference on a display unit; a participant image receiving step, wherein the network conference system receives images of each of the multiple participants participating in the network conference; a participant selection step, wherein the network conference system receives a result that two or more participants are selected from among the participants who received the image in the participant image receiving step; as well as The network conference system extracts image areas of the participants from images of the participants for the two or more participants selected in the participant selection step, and displays images of the extracted image areas of the participants together with the data on the display unit. The image based on the image area of ​​the participant is an image of the facial part of the participant extracted from the image in which the participant is photographed, The selected participant display step is to display an image according to the image area of ​​the participant on the display unit according to a predetermined display mode. Participants participating in the network conference include a first participant and a second participant. When the network conference system determines that the first participant and the second participant have had a conversation, the selected participant display step displays an image based on the image area of ​​the first participant and an image based on the image area of ​​the second participant on the display unit.

2. The method for synthesizing images using a network conference system according to claim 1, characterized in that: Further including: a speaker identification step, wherein the network conference system identifies a speaker who has spoken among the participants of the network conference, The participant selection step further includes the step of selecting two or more participants from among the participants using the information on the identified speaker.

3. The method for synthesizing images using a network conference system according to claim 1, characterized in that: The first and second participants include the presenter and the questioner. When the network conference system determines that the questioner has asked a question regarding the explanation of the presenter, the selected participant display step displays an image of the image area of ​​the presenter and an image based on the image area of ​​the questioner on the display unit.

4. The method for synthesizing images using a network conference system according to claim 1, characterized in that: Further including: In a display image storage step, the network conference system stores a display image including the data displayed on the display unit in the participant selection display step and an image according to the image area of ​​the participant.

5. The method for synthesizing images using a network conference system according to claim 4, characterized in that: Further including: The recording information correction step corrects and stores, when the data is corrected, an image related to the data among the display images stored in the display image storage step.

6. The method for synthesizing images using a network conference system according to claim 1, characterized in that: Further including: In the coding information displaying step, the network conference system displays the coding information on the display unit.

7. The method for synthesizing images using a network conference system according to claim 6, characterized in that: The data includes a plurality of pages, and the coded information displaying step displays the coded information corresponding to each page displayed on the display unit on the display unit.

8. The method for synthesizing images using a network conference system according to claim 6, characterized in that: When the network conference is in progress, the network conference system updates the coding information to obtain updated coding information, and the network conference system displays the updated coding information on the display unit.

9. A program for causing a computer to execute a screen synthesis method using a network conference system, characterized in that: The method comprises: A data display step, wherein the network conference system displays the data used for the network conference on a display unit; a participant image receiving step, wherein the network conference system receives images of each of the multiple participants participating in the network conference; a participant selection step, wherein the network conference system receives a result that two or more participants are selected from among the participants received in the participant image receiving step; as well as The network conference system extracts image areas of the participants from images of the participants for the two or more participants selected in the participant selection step, and displays images of the extracted image areas of the participants together with the data on the display unit. The image based on the image area of ​​the participant is an image of the facial part of the participant extracted from the image in which the participant is photographed, The selected participant display step is to display an image according to the image area of ​​the participant on the display unit according to a predetermined display mode. Participants participating in the network conference include a first participant and a second participant. When the network conference system determines that the first participant and the second participant have had a conversation, the selected participant display step displays an image based on the image area of ​​the first participant and an image based on the image area of ​​the second participant on the display unit.

10. A computer-readable non-transitory information recording medium, characterized in that: The program according to claim 9 is stored.

Images