Data processing device, data processing method and data processing system

Abstract

To enhance presence of a remote conference.SOLUTION: A data processing apparatus 1 acquires first image data obtained by photographing one or more first participants participating in a remote conference at a first location, first voice data of the one or more first participants, a plurality of pieces of remote image data corresponding to a plurality of remote participants participating in the remote conference from a plurality of second locations, and remote voice data of the remote participants. The data processing apparatus 1 generates first composite image data obtained by combining the plurality of pieces of remote image data, and processed voice data obtained by processing the plurality of pieces of remote voice data based on the positions of the plurality of remote participants in the first composite image data. The data processing apparatus 1 transmits the first composite image data and processed voice data to one or more electronic apparatuses at the first location, and transmits the first image data and first voice data to a plurality of information terminals at the plurality of second locations.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a data processing device, a data processing method, and a data processing system. [Background technology]

[0002] Conventionally, remote conference systems for holding a conference among multiple people in remote locations have been known (see, for example, Patent Document 1). In conventional remote conference systems, the sense of realism is enhanced by creating an image in which captured images of each of the multiple conference participants are arranged on a virtual conference table. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-54192 Summary of the Invention [Problem to be solved by the invention]

[0004] In conventional remote conference systems, individual captured images of multiple conference participants are arranged on a virtual conference table, but in some cases, multiple conference participants gather in a real conference room, and some of the participants join the conference from remote locations. In such cases, if a composite image in which images captured at all locations are arranged as is around the virtual conference table is displayed in each conference room, the sense of realism is reduced.

[0005] One of the objects of the present invention is to improve the sense of realism of a remote conference. [Means for solving the problem]

[0006] A first aspect of the data processing device of the present invention is a data processing device in a system for conducting a remote conference between three or more locations, wherein an image of the remote conference is displayed at the first location via one or more electronic devices, and the data processing device includes: an acquisition unit that acquires first image data generated by imaging one or more first participants participating in the remote conference at the first location; first audio data of the one or more first participants; multiple remote image data corresponding to multiple remote participants participating in the remote conference from multiple second locations different from the first location; and multiple remote audio data of the multiple remote participants; an image data generation unit that generates first composite image data by combining the multiple remote image data; an audio data generation unit that generates processed audio data by processing the multiple remote audio data based on the positions of each of the multiple remote participants in the first composite image data; and a data transmission unit that transmits the first composite image data and the processed audio data to one or more electronic devices used by the one or more first participants participating in the remote conference at the first location to watch the remote conference, and transmits the first image data and the first audio data to multiple information terminals used by the multiple remote participants.

[0007] The image data generation unit may generate second composite image data by combining one or more remote image data, excluding one remote image data, from the plurality of remote image data with the first image data, and the data transmission unit may transmit the second composite image data, which does not include the one remote image data corresponding to the remote participant, to the information terminal used by the remote participant corresponding to the one remote image data.

[0008] The image data generation unit may generate the second composite image data by combining the plurality of remote image data at a position in the first image data where the first participant is not included.

[0009] The image data generation unit may generate the second composite image data by combining at least a portion of the remote image data in an area different from that of the first image data.

[0010] The image data generation unit may generate the second composite image data by combining at least a portion of the remote image data with an area within a predetermined range around the image data of the desk.

[0011] The data transmission unit may transmit the first image data to one or more electronic devices used by the first participant to watch the remote conference, and the image data generation unit may generate the second composite image data by synthesizing at least a portion of the data of each of the multiple remote images at a position specified in the first image data displayed on a display device installed at the first location based on the first image data acquired via the one or more electronic devices.

[0012] The audio data generation unit may generate the processed audio data corresponding to the audio of the remote participant corresponding to the remote image data, based on the position of the remote image data in the first composite image data, including first position audio data to be output to a first position speaker installed at a first position at the first location and second position audio data to be output to a second position speaker installed at a second position at the first location.

[0013] The data processing device may further have a setting reception unit that receives settings for the attributes of the remote conference, the acquisition unit acquires captured image data and avatar image data for each of the multiple remote participants, and the image data generation unit may determine whether to use the captured image data or the avatar image data as the remote image data based on the attributes of the remote conference.

[0014] A data processing method of a second aspect of the present invention is executed by a computer and includes the steps of acquiring first image data generated by imaging one or more first participants participating in a remote conference between a first location and a plurality of second locations different from the first location at a first location, first audio data of the one or more first participants, a plurality of remote image data corresponding to a plurality of remote participants participating in the remote conference from the plurality of second locations, and a plurality of remote audio data of the plurality of remote participants, generating first composite image data by combining the plurality of remote image data, generating processed audio data by processing the plurality of remote audio data based on the positions of each of the plurality of remote participants in the first composite image data, and transmitting the first composite image data and the processed audio data to one or more electronic devices used by the first participants participating in the remote conference at the first location to watch the remote conference, and transmitting the first image data and the first audio data to a plurality of information terminals used by the plurality of remote participants.

[0015] A data processing system of a third aspect of the present invention includes an electronic device installed at a first location; a plurality of information terminals installed at a plurality of second locations different from the first location; an acquisition unit that acquires first image data generated by imaging one or more first participants participating in a remote conference between the first location and the plurality of second locations at the first location, first audio data of the one or more first participants, a plurality of remote image data corresponding to a plurality of remote participants participating in the remote conference from the plurality of second locations, and a plurality of remote audio data of the plurality of remote participants; an image data generation unit that generates first composite image data by combining the plurality of remote image data; an audio data generation unit that generates processed audio data by processing the plurality of remote audio data based on the positions of each of the plurality of remote participants in the first composite image data; and a data transmission unit that transmits the first composite image data and the processed audio data to one or more of the electronic devices used by the first participants participating in the remote conference at the first location to watch the remote conference, and transmits the first image data and the first audio data to the plurality of information terminals used by the plurality of remote participants. [Effects of the Invention]

[0016] The present invention provides an advantage of being able to improve the sense of realism of a remote conference. [Brief explanation of the drawings]

[0017] [Figure 1] FIG. 1 is a diagram illustrating an overview of a data processing system. [Figure 2] FIG. 10 is a diagram schematically illustrating an example of an image displayed at each location. [Figure 3] FIG. 1 is a diagram showing an outline of the flow of image and audio data between a plurality of locations. [Figure 4] FIG. 1 is a diagram illustrating a configuration of a data processing device. [Figure 5] FIG. 10 is a diagram for explaining a method for the second participant to specify the position where the second image data is to be combined. [Figure 6] FIG. 10 is a diagram schematically illustrating an example in which an empty seat area for a second participant is created. [Figure 7] FIG. 10 is a diagram schematically illustrating an example of an image displayed at a remote site. [Figure 8] FIG. 10 is a diagram schematically illustrating an example in which an empty seat area is created. [Figure 9] 1 is a flowchart showing the flow of processing in the data processing device 1. DETAILED DESCRIPTION OF THE INVENTION

[0018] [Data Processing System S Overview] 1 is a diagram illustrating an overview of a data processing system S. The data processing system S is a system for multiple people at multiple locations to hold a remote conference. In the following description, people who take part in the remote conference are called participants.

[0019] FIG. 1 shows a first location and multiple second locations, which are connected by a network N. The network N includes, for example, the Internet or an intranet. The data processing device 1 and electronic devices at each location are connected to the network N. The network N may include at least one of wired and wireless communication. For example, the network N may be configured by wired communication, wireless communication, or may include both wired and wireless communication sections. The wireless communication may include, for example, at least one of cellular communication, wireless LAN (local area network) communication, and satellite communication. The data processing system S may include four or more locations.

[0020] At the first location, a router 21, a computer 22, a camera 23, a display 24, and speakers 25 (speakers 25L and 25R) are installed. At the first location, one or more first participants, namely participants A1, A2, A3, and A4, are seated around a table. The router 21, computer 22, camera 23, display 24, and speaker 25 are electronic devices (corresponding to first electronic devices) used by participants A1, A2, A3, and A4 at the first location to watch the remote conference.

[0021] In this embodiment, an example is given in which camera 23 has a microphone and generates first image data based on a first image captured by camera 23 within a first location and first audio data based on a first audio within the first location, but these electronic devices may be integrated, or equivalent functions may be provided by even more electronic devices.

[0022] A computer 31 (corresponding to the second electronic device) is installed at each of the second locations. The computer 31 has a camera, a microphone, and a speaker. At each of the second locations, a participant R, who is a second participant, is seated in front of the computer 31. The computer 31 generates second image data based on a second image of the participant R and second audio data based on the second audio of the participant R.

[0023] In the following, when multiple participants A1, A2, A3, and A4 are not distinguished from one another, they may be abbreviated as "participant A." On the other hand, when multiple participants R at second locations 1 to n (n is an integer of 2 or greater) are distinguished from one another, they may be abbreviated as "participants R1 to Rn."

[0024] At least one of the first and second locations may have a plurality of cameras installed to capture images of participant A or participant R from a plurality of directions. Also, at least one of the first and second locations may have a microphone that is a pin microphone used by each participant A or participant R.

[0025] The data processing device 1 is a computer, such as a cloud server, that executes various processes for conducting a remote conference between multiple locations. As will be described in detail later, the data processing device 1 transmits images and audio sent from one location to other locations. This allows participants at each location to view images of participants at multiple other locations while listening to the audio of participants at the other locations. For example, images of participants A1 to A4 at the first location are displayed on the display of a computer 31 at each of multiple second locations, and the audio of participants R1 to R4 is output from the speaker of the computer 31.

[0026] Here, for example, if participants A1 to A4 at the first location and participants R at each of multiple second locations belong to the same organization (e.g., department or company), participant R may feel uncomfortable if participants A1 to A4 and the other participants R are displayed separately on the display of computer 31 (e.g., in different areas of a display area divided into multiple sections).

[0027] Therefore, the data processing device 1 combines the first image sent from the first location with at least a portion of the second images (hereinafter also referred to as "remote images") sent from each of the second locations (hereinafter also referred to as "remote locations"), corresponding to the other participants Rj (j is one or more of 1 to n satisfying j≠i), excluding the participant Ri (i is any one of 1 to n) to whom the composite image data is to be sent.

[0028] The data processing device 1 transmits the composite image data thus synthesized to the computer 31 of the participant Ri, so that one or more participants Rj can feel as if they are participating in the remote conference from the first location together with the participants A1 to A4 at the first location. Note that at least a part of the remote image data is data of the area in the remote image data where the participant R is shown (hereinafter also referred to as "person image data").

[0029] In the following description, combining at least a portion of remote image data may be referred to as “combining remote image data.” Furthermore, “image data” may be abbreviated to “image” as appropriate.

[0030] Furthermore, the data processing device 1 generates, for example, image data of a virtual conference room (hereinafter also referred to as "virtual conference room image data") and combines at least a portion of a remote image (for example, person image data) sent from each of two or more remote locations with the generated virtual conference room image data. Furthermore, the data processing device 1 transmits the virtual conference room image combined with the remote image and the audio data sent from each of the remote locations to the first location.

[0031] The virtual conference room image data is, for example, an image that simulates the space inside a conference room at the first location. The data processing device 1, for example, synthesizes (or arranges) the person image data of each of the multiple remote participants R1 to Rn around a virtual table (or desk) included in one piece of virtual conference room image data.

[0032] By using this type of image synthesis processing, participant A at the first location can hold a conference as if each of the remote participants R1 to Rn located at different remote locations were gathered in a single conference room space and participating in the remote conference. Here, for example, by using an image of the virtual conference room image that simulates the space inside the conference room at the first location, it is expected that the effect of enhancing the sense of unity or realism of the conference between participant A at the first location and each of the remote participants R1 to Rn can be achieved.

[0033] FIG. 2 is a diagram schematically illustrating an example of an image displayed at each location. For convenience, FIG. 2 assumes that the participant A at the first location is two participants A1 and A2. FIG. 2(a) illustrates an image displayed on the display 24 at the first location where the two participants A1 and A2 are located. FIG. 2(b) illustrates an image displayed on the computer 31 at the remote location where the remote participant R1 of the two remote participants R1 and R2 is located (in other words, the video viewed by the remote participant R1). FIG. 2(c) illustrates an image displayed on the computer 31 at the remote location where the remote participant R2 is located (in other words, the video viewed by the remote participant R2). The display mode illustrated in FIG. 2 is merely an example, and the display mode may be determined based on various settings or conditions, as described below.

[0034] As shown in Fig. 2(a), the first location displays an image in which remote images (person images) of participants R1 and R2 taken at each remote location are composited around a virtual table VT in a single virtual conference room image. This display mode allows participants A1 and A2 at the first location to view an image that looks as if remote participants R1 and R2, who are located at different remote locations, are gathered in a single virtual conference room space and participating in a remote conference.

[0035] The composite (or placement) positions of the remote participants R1 and R2 in the virtual conference room image may be determined or controlled by, for example, settings or conditions such as the organization to which the remote participant Ri belongs. Information about the organization to which each remote participant Ri belongs may be displayed on the display 24 of the first location together with the person image of the remote participant Ri.

[0036] 2(b), the display of the computer 31 used by the remote participant R1 displays an image obtained by combining a first image of participants A1 and A2 taken at a first location with a remote image of another remote participant R2 taken at another remote location. The image of the remote participant R2 is combined in an area around the table RT at the first location that is different from (or does not overlap with) the area of ​​the table RT at the first location. This display mode allows the remote participant R1 to feel as if the other remote participant R2 is participating in the conference at the first location together with the participants A1 and A2 at the first location.

[0037] 2(c), the display of the computer 31 used by the remote participant R2 displays an image in which a first image taken of the participants A1 and A2 at the first location is superimposed with a remote image taken of the other participant R1 at the other remote location. The image of the remote participant R1 is superimposed, for example, in an area around the table RT at the first location that is different from (or does not overlap with) the area of ​​the participants A1 and A2. This display mode allows the remote participant R2 to feel as if the other remote participant R1 is participating in the conference at the first location together with the participants A1 and A2 at the first location.

[0038] Note that the remote image of one of the remote participants R1 and R2 does not need to be viewed by that remote participant, and therefore may be composited at a position around the table RT in the first image where it overlaps with the remote image of the other of the remote participants R1 and R2. For example, in FIG. 2(b), the remote image of the remote participant R2 may be composited at a position corresponding to the position where the remote image of the remote participant R1 is composited in FIG. 2(c). Similarly, in FIG. 2(c), the remote image of the remote participant R1 may be composited at a position corresponding to the position where the remote image of the remote participant R2 is composited in FIG. 2(b). Note that, together with the display of the remote participants R1 and R2, information indicating that the remote participants R1 and R2 are not actually at the first location (e.g., they are remotely participating at the first location) may be displayed on the display of each location.

[0039] [Data flow between multiple locations] 3 is a diagram showing an overview of the flow of image and audio data between multiple locations. A router 21 at a first location transmits first image data and first audio data generated by a camera 23 to a data processing device 1. Each of computers 31 at multiple second locations (remote locations) transmits the generated remote image data and remote audio data (i.e., second image data and second audio data) to the data processing device 1.

[0040] The data processing device 1 creates virtual conference room image data, and transmits to the first location first composite image data obtained by combining at least a portion of each remote image data with the created virtual conference room image data, and first processed audio data obtained by processing the remote audio data. The first processed audio data is, for example, stereo audio data or three-dimensional audio data obtained by processing each remote audio data based on the position of the remote image data so that the audio of the remote participant Ri can be heard from the position of the remote image data in the virtual conference room image data.

[0041] The data processing device 1 combines the first image data received from the first location with a portion of each of the remote images of the other remote participants Rj, excluding the remote participant Ri at the remote location to which the data is to be sent, from the multiple remote image data to create second composite image data to be sent to each of the remote locations.

[0042] The data processing device 1 also processes the received first audio data and the remote audio data of the other remote participants Rj excluding the remote participant Ri at the remote site to which the data is to be transmitted, to create second processed audio data to be transmitted to each of the remote sites. The second processed audio data is, for example, stereo audio data or three-dimensional audio data obtained by processing the remote audio data so that the audio of the remote participant Rj can be heard from the position of the remote image data combined with the first image data.

[0043] The data processing device 1 transmits to the first location first composite image data in which each remote image data is composited with virtual conference room image data, and first processed audio data in which the remote audio data is processed. The data processing device 1 also transmits second composite image data, second processed audio data, and first audio data to each remote location. The data processing device 1 may also transmit the first image data and remote image data before composite to the remote location. The data processing device 1 may also transmit to the remote location unprocessed remote audio data based on the position of the remote image data.

[0044] [Configuration and Operation of Data Processing Device 1] 4 is a diagram showing the configuration of the data processing device 1. The data processing device 1 has a communication unit 11, a storage unit 12, and a control unit 13. The control unit 13 has an acquisition unit 131, a setting acceptance unit 134, an image data generation unit 135, an audio data generation unit 136, and a data transmission unit 137.

[0045] The communication unit 11 has a communication interface for transmitting data to and from the router 21 or the computer 31 via the network N. The communication unit 11 inputs the first image data and first audio data received from the router 21 to the acquisition unit 131. The communication unit 11 inputs the remote image data and remote audio data received from the computer 31 to the acquisition unit 131.

[0046] The communication unit 11 inputs setting data transmitted by the router 21, the computer 31, or another information terminal to the setting reception unit 134. The setting data is, for example, data indicating attributes of the remote conference, attributes of the participants, or the format of the image to be displayed on the screen, and is data input by the organizer of the remote conference or any of the participants. The organizer of the remote conference may or may not be included in the multiple participants.

[0047] Furthermore, the communication unit 11 transmits to each location various types of data input from the data transmission unit 137. Specifically, as shown in Fig. 3, the communication unit 11 transmits data of the composite image, data of the processed audio, and the like to one or more electronic devices that are used by participants at each location to watch the remote conference.

[0048] The storage unit 12 has storage media such as a ROM (Read Only Memory), a RAM (Random Access Memory), and an SSD (Solid State Drive). The storage unit 12 stores programs executed by the control unit 13. The storage unit 12 also stores various setting data received by the communication unit 11. Furthermore, the storage unit 12 temporarily stores image data and audio data transmitted from each location.

[0049] The control unit 13 includes, for example, a CPU (Central Processing Unit). The control unit 13 executes a program stored in the storage unit 12, thereby functioning as an acquisition unit 131, a setting reception unit 134, an image data generation unit 135, an audio data generation unit 136, and a data transmission unit 137.

[0050] The acquisition unit 131 acquires first image data generated by capturing images of one or more first participants participating in the remote conference at the first location. The acquisition unit 131 also acquires first audio data including audio of the one or more first participants acquired at the first location. The acquisition unit 131 acquires the first image data and the first audio data via the communication unit 11. The acquisition unit 131 inputs the first audio data to the audio data generation unit 136, and inputs the first audio data to the image data generation unit 135.

[0051] The acquisition unit 131 also acquires multiple pieces of remote audio data based on the audio of each remote participant participating in the remote conference at multiple remote locations, and multiple pieces of remote image data corresponding to each remote participant. The remote image data may be captured image data generated by capturing an image of the remote participant at the remote location, or may be avatar image data of the remote participant created in advance. That is, the acquisition unit 131 may acquire at least one of the captured image data or avatar image data of the remote participant. For example, the acquisition unit 131 acquires either the captured image data or the avatar image data, whichever is requested by the image data generation unit 135.

[0052] The acquisition unit 131 also acquires second audio data including audio of the remote participants acquired at the remote site. The acquisition unit 131 acquires remote image data and remote audio data via the communication unit 11. The acquisition unit 131 inputs the remote image data to the image data generation unit 135 and inputs the remote audio data to the audio data generation unit 136.

[0053] The setting reception unit 134 receives various settings by acquiring setting data input by the host of the remote conference or any of the participants via the communication unit 11. The setting reception unit 134 may receive settings via a user interface such as a keyboard and mouse (not shown). The setting reception unit 134 receives, for example, settings of attributes of the remote conference. The setting reception unit 134 notifies the image data generation unit 135 of the received attributes of the remote conference.

[0054] The image data generation unit 135 generates first composite image data by combining at least a portion of each of the multiple remote image data sent from the multiple remote locations as image data to be displayed on a display at a certain location. The first composite image data is, for example, image data by combining at least a portion of each of the remote image data with the above-mentioned virtual conference room image data. The image data generation unit 135 inputs the generated image data to the data transmission unit 137.

[0055] As an example, the image data generating unit 135 generates first composite image data by combining each remote image data around a virtual table in the virtual conference room image data as shown in Fig. 2(a). Also, the image data generating unit 135 generates second composite image data by combining one or more remote image data, excluding one remote image, from the multiple remote images with the first image data sent from the first location.

[0056] As an example, the image data generation unit 135 generates second composite image data for the remote participant R1 by combining the remote image data of the remote participant R2 of the remote participants R1 and R2 into the first image data for the first location, as shown in Fig. 2(b). Also, as shown in Fig. 2(c), the image data generation unit 135 generates second composite image data for the remote participant R2 by combining the remote image data of the remote participant R1 of the remote participants R1 and R2 into the first image data for the first location.

[0057] In the second composite image data, each remote image data may be composited into an area within a predetermined range around the table in the first image data of the first location. The predetermined range may be, for example, an area in the first image data where chairs are arranged around the table.

[0058] The audio data generation unit 136 generates audio data to be output from speakers installed at each location participating in the remote conference. The audio data generation unit 136 processes the audio data of each remote participant based on the position of each remote image data in the first composite image data, for example, to generate first processed audio data corresponding to the audio to be heard by the participants at the first location.

[0059] The audio data generation unit 136 processes the audio data of the remote participant based on the position of the remote image data in the second composite image data to generate second processed audio data corresponding to the audio to be heard by the other remote participants. The audio data generation unit 136 may generate audio data including the audio of the first participant at the first location and to be heard by the remote participants at the remote locations. The audio data generation unit 136 inputs the generated processed audio data to the data transmission unit 137.

[0060] The audio data generation unit 136 generates multi-channel processed audio data, which includes multiple audio signals transmitted from multiple other locations, as audio data to be output from a speaker at a certain location. The multi-channel processed audio data includes first-position audio data and second-position audio data output from a speaker at a first location (e.g., the right side) and a speaker at a second location (e.g., the left side). The multi-channel processed audio data may include three or more pieces of audio data corresponding to three or more locations.

[0061] As an example, the audio data generation unit 136 generates first processed audio data including first position audio data to be output to a first position speaker installed at a first position at the first location and second position audio data to be output to a second position speaker installed at a second position at the first location, based on the position of the remote image in the first composite image data. Specifically, when the position of the remote image of the remote participant is closer to the first position than the second position, the audio data generation unit 136 generates first processed audio data that has been processed so that the proportion of the remote participant's voice included in the first position audio data is greater than the proportion of the remote participant's voice included in the second position audio data.

[0062] Conversely, when the position of the remote image of the remote participant is closer to the second position than the first position, the audio data generation unit 136 generates first processed audio data that has been processed so that the proportion of the audio of the remote participant contained in the second position audio data is greater than the proportion of the audio of the remote participant contained in the first position audio data.

[0063] In addition, when the position of the remote image of the remote participant is at an equal distance from the first position and the second position, the audio data generation unit 136 may generate first processed audio data that has been processed so that the proportion of the remote participant's audio contained in the first position audio data and the second position audio data is equal.

[0064] The audio data generation unit 136 may generate the first processed audio data further based on the result of identifying the depth direction position at the first location of the position of the remote image data in the first composite image data. As an example, in the example shown in FIG. 2(c), the position of the remote image data of the remote participant R2 is further back than the other remote participant R1, so the audio data generation unit 136 may generate first processed audio data that has been processed so that the audio of the remote participant R2 is quieter than that of the remote participant R1. Generating such first processed audio data by the audio data generation unit 136 enhances the sense of realism of the participants listening to the audio based on that data.

[0065] Regardless of whether the position of the remote image of the remote participant is closer to the first position than the second position or closer to the second position than the first position, the audio data generation unit 136 may generate first processed audio data that has been processed so as to increase the proportion of the audio of the second participant included in the first position audio data depending on the attributes of the remote participant. The audio data generation unit 136 may control the proportion of audio data as described above based on attributes such as the volume, voice quality, and clarity of the voice of the remote participant, for example.

[0066] Specifically, even if the remote participant R1 is positioned closer to the front than other remote participants R1 in the virtual conference room image, the audio data generation unit 136 may increase the proportion of the remote participant R1's audio if the volume of the remote participant R1's voice is below a threshold or if the condition that the voice quality is difficult to hear is met.

[0067] The above-mentioned audio data proportion control may be applied to the second processed audio data in the same manner as the first processed audio data when remote image data is combined with the first image data of the first location as shown in Figures 2(b) and 2(c).

[0068] The data transmission unit 137 transmits the image data input from the image data generation unit 135 and the audio data input from the audio data generation unit 136 to the base to which each data is to be transmitted, via the communication unit 11. Specifically, the data transmission unit 137 transmits the first composite image data and the first processed audio data to one or more electronic devices used by one or more participants in the remote conference at the first base to watch the remote conference.

[0069] The data transmission unit 137 also transmits the second composite image data to the information terminals used by the individual remote participants. That is, the data transmission unit 137 transmits the second composite image data and the second processed audio data to the computers 31 used by the remote participants to view the remote conference. The data transmission unit 137 may also transmit data for accepting settings from the organizer or participants of the remote conference to each location.

[0070] The functions of the data processing device 1 described above (for example, various functions of the control unit 13) may be realized by a plurality of data processing devices 1. For example, various processes such as generation of image data and generation of audio data may be distributed and performed by a plurality of data processing devices 1 (for example, processors mounted on each of a plurality of servers).

[0071] [Method for generating synthetic image data] The process by which the image data generating unit 135 generates the composite image data will be described in detail below. The image data generation unit 135, for example, generates virtual conference room image data in which each remote participant Ri is placed, and synthesizes at least a portion of the remote image data of each remote participant at the remote location with the generated virtual conference room image data, thereby generating first composite image data to be displayed on the display 24 at the first location, as illustrated in Figure 2(a).

[0072] In addition, the image data generation unit 135 generates second composite image data by combining at least a portion of each remote image data at a position in the first image data captured at the first location that does not include one or more first participants, as shown in Figures 2(b) and 2(c), for example.

[0073] For example, the image data generating unit 135 identifies an area in the first image data whose similarity to standard image data of a face previously stored in the storage unit 12 is equal to or greater than a threshold as a facial area, and combines the remote image data with an area different from the identified area. The image data generating unit 135 may also identify an area of ​​the participant's body, and combine the remote image data with an area different from the identified area.

[0074] The image data generation unit 135 may determine the position at which to superimpose the remote image data on the virtual conference room image data or the first image data of the first location by any method, and may superimpose the remote image data at a predetermined position by referring to superimposition position information stored in the storage unit 12. For example, the image data generation unit 135 may superimpose the remote image data on the upper right or upper left of the first image data, or may superimpose the remote image data in a region below the first image data. When the remote image data of some remote participants is superimposed in a region below the first image data, it is easier to give the participant A at the first location the impression that the locations of those some remote participants are closer to the first location than the locations of the other remote participants. Note that the image data generation unit 135 may generate the superimposed image data so that information indicating the location of each remote location (e.g., information such as the name of a prefecture or address) is displayed together with each remote image.

[0075] The image data generation unit 135 may composite the remote image data at a position designated by the organizer or participant of the remote conference. As an example, the data transmission unit 137 transmits the first image data to the computer 31, which is an information terminal used by the remote participant, and the image data generation unit 135 generates the second composite image data by compositely combining at least a portion of the remote image data at a position designated by the remote participant in an image based on the first image data displayed on the computer 31. By operating the image data generation unit 135 in this manner, the remote participant can have his or her own image displayed at a position that he or she desires. Note that the first image data transmitted by the data transmission unit 137 to the computer 31 may be image data in which portions other than the image of one or more first participants have been modified from the first image data transmitted from the first location.

[0076] FIG. 5 is a diagram illustrating a method for a remote participant to specify a position at which remote image data is to be superimposed. FIG. 5(a) is a screen for specifying a superimposition position, displayed on the computer 31 of the remote participant R1 or R2. The screen shown in FIG. 5(a) shows the positions of star marks X and Y as positions at which remote image data can be superimposed. FIG. 5(b) is second superimposed image data displayed on the display of the computer 31 used by the remote participant R2 when the remote participant R1 specifies the position of star mark X in FIG. 5(a). FIG. 5(c) is second superimposed image data displayed on the display of the computer 31 used by the remote participant R1 when the remote participant R2 specifies the position of star mark Y in FIG. 5(a).

[0077] The image data generation unit 135, for example, searches the first image data for an area whose similarity to reference image data representing a person is equal to or greater than a threshold, thereby identifying an area of ​​the image of the first participant in the first image data, and causes the computer 31 to display an image in an area other than the identified area, with a mark indicating a position where the remote image data can be composited. The image data generation unit 135 may also cause the computer 31 to display an image in an area other than the identified area, with a mark indicating a position where the remote image data can be composited, at the center of an area that is approximately the same size as the area of ​​the image of the first participant. In this way, the image data generation unit 135 presents the remote participant with a position where the second image data can be composited, allowing the remote participant to appropriately specify the position.

[0078] The data transmission unit 137 may transmit the first image data to an information terminal used by a participant registered as the organizer of the remote conference, and the image data generation unit 135 may generate the second composite image data by compositely combining at least a part of the remote image data at a specified position in an image based on the first image data displayed on the information terminal of the organizer. By operating the image data generation unit 135 in this manner, the organizer of the remote conference can arrange the images of the remote participants at positions appropriate for the relationship between the multiple participants or the purpose of the remote conference, etc.

[0079] The data transmission unit 137 may transmit the virtual conference room image data to one or more electronic devices used by one or more first participants at the first location to view the remote conference, and the image data generation unit 135 may generate first composite image data by compositely combining at least a portion of the remote image data at a specified position in the virtual conference room image displayed on the display 24, which is a display device installed at the first location, based on the virtual conference room image data acquired via the one or more electronic devices (e.g., the router 21 or the computer 22). By operating the image data generation unit 135 in this manner, the participants at the first location can place the remote image data of the remote participants in appropriate positions while viewing the virtual conference room.

[0080] In the second composite image data for the remote locations, it may be better to arrange the image of the remote participant among the images of multiple first participants, or it may be better to separate the area for displaying the image of the remote participant for each remote participant's location. As an example, when the first image data sent from the first location includes multiple first participants and the remote image sent from at least one other remote location includes one remote participant, the image data generation unit 135 generates second composite image data in which the remote image is arranged among the multiple first participants, as shown in Figures 2(b) and 2(c).

[0081] By the image data generating unit 135 generating such second composite image data, the remote participants at the remote sites who view the image based on the second composite image data can feel a greater sense of realism.

[0082] The image data generation unit 135 may generate first composite image data in which remote images are arranged in different areas of the area divided into the virtual conference room image data for each of the multiple remote participants or for each of the groups into which the multiple remote participants are divided. Alternatively, the image data generation unit 135 may generate multiple pieces of virtual conference room image data, and generate the first composite image data by allocating the multiple remote images to different pieces of virtual conference room image data and combining them.

[0083] The image data generating unit 135 may combine at least a part of the remote image in a region of the second combined image data that is different from the region of the first image data sent from the first location. For example, as shown in Figures 6(a) and 6(b), the image data generating unit 135 combines an image of a remote participant R1 or R2 at a remote location in a region of the first image that is different from the region of the first image in which multiple participants A at the first location are shown.

[0084] Based on the content of the settings received by the setting receiving unit 134, the image data generating unit 135 may determine whether to place the image of the remote participant R1 or R2 at the remote location in an area different from the area of ​​the first image in which the multiple participants A at the first location are shown, as shown in Figures 6(a) and 6(b), or to place the image of the participant at the other location in the area in which the multiple participants A at the first location are shown, as shown in Figures 2(b) and 2(c). The area in which the multiple participants are shown is, for example, the smallest oval or elliptical area that includes the images of the multiple participants.

[0085] As an example, when the image data generation unit 135 is configured to display participants from different locations in the same area, such as when a first participant A at a first location and a remote participant R2 (or R1) at a remote location belong to the same department or company, the image data generation unit 135 places a remote image of the remote participant R2 (or R1) within the area in which one or more first participants A at the first location are shown (for example, around the table RT), as shown in Figure 2(b) (or Figure 2(c)).

[0086] On the other hand, when the image data generation unit 135 is configured not to display participants from different locations in the same area, such as when the first participant A at the first location and the remote participant R2 (or R1) at the remote location belong to different departments or companies, the image data generation unit 135 places the remote image of the remote participant R2 (or R1) in an area (e.g., an area away from the table RT) different from the area in which one or more first participants at the first location are displayed, as shown in FIG. 6(a) (or FIG. 6(b)). By operating the image data generation unit 135 in this manner, the participants at each location are displayed in a manner appropriate to their affiliation, further enhancing the sense of realism of the remote participants at the remote location. In addition, the visibility of the relationships between the participants at each location is improved, which is expected to contribute to the smooth progress of the conference, for example.

[0087] [When it is difficult to specify a position in the first image data where the remote image data of the remote participant can be superimposed] In some cases, such as when the first image data includes a large number of first participants, it may be difficult to specify a position as shown in Fig. 5(a). For example, there may be no candidate positions or areas (in other words, vacant positions or vacant areas) where the remote participants can be located in the first image corresponding to the first location.

[0088] In such a case, the image data generation unit 135 may create an empty seat area for the remote participant by processing or modifying at least a portion of the first image data corresponding to the first location. As a non-limiting example, the image data generation unit 135 may virtually create an empty seat area in which the remote image of the remote participant can be placed by changing the size or shape (e.g., stretching or expanding in the length direction) of image data corresponding to the table RT in the first image data.

[0089] 7A and 7B are diagrams showing examples of how vacant areas are created. Fig. 7A shows an example in which five participants A1 to A5 are positioned around a table RT at a first location, and there are no candidate positions (or areas) around the table where remote images corresponding to the remote participants can be composited.

[0090] In this case, as shown in FIG. 7(b), for example, the image data generation unit 135 stretches image data corresponding to the table RT in the first image data of the first location (hereinafter, sometimes referred to as "table image data") in the length direction of the table RT (the direction indicated by the arrow). By this stretching, a virtual empty seat area is created around the table RT onto which the remote image of the remote participant can be composited. FIG. 7(b) shows an example in which two empty seat areas X1 and Y1 are created facing each other across the table RT. The image data generation unit 135 presents a screen to the remote participant, indicating either one of the two candidate empty seat areas X1 or Y1 as a position onto which the image of the remote participant can be composited.

[0091] 8A and 8B are schematic illustrations of an example of an image in which an empty seat area has been created, assuming that four first participants A1 to A4 are located at the first location. In FIG. 8A, a virtual table VT2 has been added to the table RT at the first location, and empty seat areas X1 and Y1 are displayed around the table VT2. When the remote participant R2 specifies the empty seat area X1 or Y1 on this screen, the image data generator 135 generates second composite image data as shown in FIG. 8B, in which the remote image of the remote participant R2 is composited into the specified area.

[0092] Such processing by the image data generation unit 135 can reduce cases in which the remote image of the remote participant cannot be composited with the first image displayed at the first location, thereby increasing the probability of the remote participant virtually appearing at the first location without compromising the realism of the conference.

[0093] The image data generating unit 135 may create an empty seat area on the condition that there is no empty seat area in the first image data, or may create an empty seat area when there is an empty seat area. As an example, the image data generating unit 135 may create an empty seat area in response to receiving an instruction to increase the empty seat area from either participant A at the first location or a remote participant at a remote location.

[0094] Furthermore, the image data generation unit 135 may place a virtual table VT2 in the first image data regardless of whether there is an empty seat area in the first image data. For example, the image data generation unit 135 may place a virtual table VT2 in the first image data so that the virtual table VT (see, for example, FIG. 2(a)) in the virtual conference room image appears visually connected to the table RT displayed on the display 24 at the first location. Alternatively, the image data generation unit 135 may place a virtual table VT2 in the first image data so that the actual table placed at the first location appears to be placed adjacent to the table RT in the virtual conference room image displayed on the display 24.

[0095] By arranging the virtual table VT2 in this way, the first participant at the first location can feel as if the first location and the virtual conference room where each remote participant is located are holding a conference in a single conference room space, further improving the sense of unity and realism of the conference.

[0096] [Processing of image data captured by remote participants] Incidentally, at the remote site, there are cases where the remote participants are facing the camera of an information terminal such as a laptop PC used for the remote conference. In such cases, the remote image data of the remote site acquired by the data processing device 1 is image data obtained by capturing an image of the remote participants from the front (hereinafter also referred to as "front image data").

[0097] If such front image data is directly combined with the first image data (or the third image data at the third location), there is a possibility that the facial or body direction of the participant at the first location will not match the facial or body direction of the remote participant in the first image data.

[0098] For example, if first image data including a first participant oriented with their eyes fixed on the display 24 at the first location is directly combined with frontal image data of a remote participant at a remote location, composite image data is generated that includes a second participant oriented with their eyes fixed in a direction away from the display 24.

[0099] In such a case, the composite image data including the remote participants positioned in an unnatural orientation is displayed at the first location, which may reduce the sense of unity or realism of the remote conference. To reduce such unnaturalness, the image data generation unit 135 may adjust the orientation of the remote participants in the composite image data by processing (e.g., transforming) the remote image data.

[0100] For example, the image data generation unit 135 transforms image data corresponding to the facial region of the remote participant in the remote image data (hereinafter also referred to as "face image data") so that the line of sight is directed toward the display 24. As an example, the image data generation unit 135 transforms the face image data so that the line of sight is directed toward the front of the remote image data. Furthermore, the image data generation unit 135 transforms image data corresponding to the body region of the remote participant in the remote image data (hereinafter also referred to as "body image data") so that the line of sight is directed toward the table based on the orientation of the table in the first image data.

[0101] In other words, the image data generating section 135 may modify the captured image data of the second participant by performing separate (for example, different) image processing on the face image and the body image included in the second image data.

[0102] By processing the second image data as described above, it is possible to reduce the unnaturalness of the remote participants in the composite image data displayed at the first location, thereby preventing a decrease in the sense of unity or realism of the remote conference.

[0103] [Use of captured images and avatar images] The images displayed on the displays installed at each location may display captured images of each participant, or may display avatar images of each participant. The image data generation unit 135 determines whether to generate composite image data including captured image data of each participant or composite image data including avatar image data of each participant, based on the contents of settings made by the organizer or participants of the remote conference accepted by the setting acceptance unit 134, for example.

[0104] The image data generation unit 135 may unify the images of multiple participants at multiple locations simultaneously displayed on the display into either a captured image or an avatar image. As an example, when it is set that the image of the first participant at the first location is to be used as the avatar image, the image data generation unit 135 generates second composite image data by combining the avatar image data of the remote participant with the first image data including the avatar image data of the first participant, even if it is not set whether the image of the remote participant at the remote location is to be used as a captured image or an avatar image.

[0105] On the other hand, when it is set that the image of the first participant at the first location is to be the captured image, even if a setting is not made as to whether the image of the remote participant at the remote location is to be the captured image or the avatar image, the image data generation unit 135 generates second composite image data by combining the first image data including the captured image data of the first participant with the captured image data of the remote participant. In this way, by the image data generation unit 135 generating composite image data consisting of either the captured image data or the avatar image data, it is possible to prevent the discomfort caused by different types of images of some participants.

[0106] The image data generating unit 135 may determine whether to use captured image data or avatar image data as remote image data based on the attributes of the remote conference set via the setting receiving unit 134. As an example, the image data generating unit 135 uses avatar image data as remote image data when the attributes of the remote conference are an internal conference, and uses captured image data when the attributes of the remote conference are a conference with people outside the company. By operating the image data generating unit 135 in this manner, for example, remote participants do not need to dress up for a remote conference when it is sufficient to display an avatar image.

[0107] [Processing flow in data processing device 1] Fig. 9 is a flowchart showing the flow of processing in the data processing device 1. The flowchart shown in Fig. 9 starts when the host of the remote conference performs an operation to start the remote conference and the setting receiving unit 134 acquires a request to start the remote conference.

[0108] When the remote conference starts, the acquisition unit 131 acquires the first image data of the first location and the remote image data of each of the multiple remote locations (S11). Although not shown in FIG. 9, the acquisition unit 131 continuously acquires the first image data and the remote image data.

[0109] During this time, the setting reception unit 134 monitors whether a request to specify a position for compositing the remote image data has been sent from the computer 31 at the remote site (S12). If the setting reception unit 134 determines that a request to specify a compositing position has been received (YES in S12), the image data generation unit 135 determines, for example, whether the remote participant corresponding to the sender of the request belongs to the same organization as the participant at the first site (S13).

[0110] If the image data generation unit 135 determines that the remote participant belongs to the same organization as the participant at the first location (YES in S13), it transmits position designation screen data based on the first image data to the computer 31 of the remote participant (S14). After transmitting the position designation screen to the computer 31, if the image data generation unit 135 acquires information indicating the position designated by the second participant from the computer 31, it determines the position indicated by the information as the composite position of the remote image (i.e., the second image) in the first image data (S15).

[0111] If a request for specifying a compositing position is not received within a predetermined period of time in S12 (NO in S12), the image data generation unit 135 determines the compositing position of the remote image (i.e., the second image) in the first image without any instructions from the remote participant (S15). As an example, if the remote participant belongs to the same organization as the participants at the first location, the image data generation unit 135 determines a position within an area in the first image data that includes image data of the remote participant (e.g., a position around the table RT at the first location) as the compositing position of the remote image data. If the remote participant does not belong to the same organization as the participants at the first location, the image data generation unit 135 determines a position within an area in the first image data that does not include an image of the first participant (e.g., a position away from the table RT at the first location) as the compositing position of the remote image data.

[0112] Next, the image data generation unit 135 generates first composite image data by combining the remote image of each remote participant Ri with the virtual conference room image, and generates second composite image data by arranging the remote image data at the combining position in the first image data (S16). Furthermore, the audio data generation unit 136 generates processed audio data based on the combining position of the remote image data (S17). The data transmission unit 137 transmits the composite image data generated by the image data generation unit 135 and the processed audio data generated by the audio data generation unit 136 to the locations corresponding to each data (S18). Note that the generation of the composite image data (S16) may be performed after the generation of the processed audio data (S17), or may be performed in parallel with the generation of the processed audio data (S17).

[0113] If the setting reception unit 134 has not received an instruction to end the remote conference from the organizer of the remote conference (NO in S19), the acquisition unit 131 acquires the first image data and multiple remote image data (i.e., second image data) (S20), and the image data generation unit 135, the audio data generation unit 136, and the data transmission unit 137 repeat the processes from S16 to S18. If the setting reception unit 134 has received an instruction to end the remote conference from the organizer of the remote conference (YES in S19), the control unit 13 ends the process.

[0114] [Variation 1] In the above description, it is assumed that there is one remote participant at each of the multiple remote locations, but there may be multiple remote participants at one remote location. In this case, the image data generation unit 135 synthesizes a portion of each of the multiple remote image data corresponding to the multiple remote participants at a position different from the position corresponding to one or more first participants in the first image data.

[0115] In addition, there may be cases where the participants at the first location and some or all of the participants at the remote locations belong to the same organization, and it is preferable that the image displayed at each remote location shows a mixture of one or more first participants at the first location and one or more remote participants at the remote locations. In such cases, the image data generation unit 135 may extract image data of one or more first participants and image data of one or more remote participants from the first image data and the remote image data, and generate composite image data to be transmitted to the remote location by combining the extracted multiple image data.

[0116] [Effects of Data Processing Device 1] As described above, in a remote conference in which one or more participants are present at each of a first location and multiple remote locations, the image data generation unit 135 generates first composite image data by combining multiple remote images captured at each of the multiple remote locations with a virtual conference room image. The audio data generation unit 136 generates processed audio data by processing the audio of the remote participants based on the position of the remote image data in the composite image data. The data transmission unit 137 then transmits the composite image data and processed audio data to one or more electronic devices at the first location so that the first participant at the first location can listen to and hear the combined image data and processed audio data. With the data processing device 1 configured in this way, the first participant at the first location can experience a sense of realism, as if all the remote participants at the multiple remote locations were gathered in the virtual conference room image and participating in the remote conference together with the first participant.

[0117] In the above-described embodiments, "image data" is not limited to two-dimensional data but may include three-dimensional data (e.g., hologram data that represents participants in a three-dimensional manner). For example, when participants are displayed (or projected) in a three-dimensional manner using three-dimensional holography technology, the image processing described in the above-described embodiments may be applied.

[0118] In the above-described embodiments, the term "site" may be replaced with a term meaning a location where people participating in a remote conference are located, such as "base," "space," "conference room," or "room." The term "organization" may be replaced with a term meaning a gathering (group) of people, such as "group," "group," "team," or "circle." Furthermore, the term "transmission" may be replaced with "output," and the term "combination (of image data)" may be replaced with "superimposition." The term "identification" may be replaced with "detection" or "identification," for example.

[0119] Furthermore, the term "part" used in the configurations exemplified in the above-described embodiments may be interchangeably read as other terms such as "means," "circuit," or "device."

[0120] Furthermore, in the above-described embodiments, terms such as "first," "second," and "third" are used for convenience to distinguish between two or more elements, and do not limit the physical quantity, superiority or inferiority, or order of the elements to which the terms refer.

[0121] Furthermore, this invention will make it possible to contribute to Goal 9 of the United Nations' Sustainable Development Goals (SDGs), which is "Build resilient infrastructure, promote inclusive and sustainable industrialization, and promote innovation and resilience."

[0122] The present invention has been described above using embodiments, but the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. For example, all or part of the device can be configured by functionally or physically distributing or integrating any unit. Furthermore, new embodiments resulting from any combination of multiple embodiments are also included in the embodiments of the present invention. The effects of the new embodiments resulting from the combination also have the effects of the original embodiments. [Explanation of symbols]

[0123] 1 Data Processing Device 11 Communications Department 12 Storage section 13 Control Unit 21 Router 22 Computer 23 Camera 24 displays 25 speakers 31 Computer 131 Acquisition Department 134 Settings Reception Section 135 Image data generation unit 136 Audio data generation unit 137 Data transmission unit

Claims

1. A data processing device in a system for conducting a remote conference among three or more locations, wherein an image of the remote conference is displayed at a first location via one or more electronic devices; an acquisition unit that acquires first image data generated by capturing an image of one or more first participants participating in the remote conference at the first location, first voice data of the one or more first participants, a plurality of remote image data corresponding to a plurality of remote participants participating in the remote conference from a plurality of second locations different from the first location, and a plurality of remote voice data of the plurality of remote participants; an image data generating unit that generates first composite image data by combining the plurality of remote image data, and generates second composite image data by combining one or more remote image data, excluding one remote image data, of the plurality of remote image data with the first image data; an audio data generation unit that generates processed audio data by processing the plurality of remote audio data based on the positions of the plurality of remote participants in the first composite image data; a data transmitting unit that transmits the first composite image data and the processed audio data to one or more electronic devices used by the one or more first participants participating in the remote conference at the first location for viewing the remote conference, transmits the first image data and the first audio data to multiple information terminals used by the multiple remote participants, transmits the second composite image data that does not include the one remote image data corresponding to the one remote participant to the information terminal used by the remote participant corresponding to the one remote image data, and transmits the first image data to one or more electronic devices used by the first participants for viewing the remote conference; and the image data generation unit generates the second composite image data by combining at least a portion of each of a plurality of remote image data at a position designated in the first image data displayed on a display device installed at the first site based on the first image data acquired via the one or more electronic devices; Data processing device.

2. the image data generation unit generates the second composite image data by combining the plurality of remote image data at positions in the first image data where the first participant is not included.

2. The data processing device according to claim 1.

3. the image data generation unit generates the second composite image data by combining at least a portion of the remote image data in an area different from the first image data.

2. The data processing device according to claim 1.

4. the image data generation unit generates the second composite image data by combining at least a portion of the remote image data with an area within a predetermined range around the image data of the desk; 4. The data processing device according to claim 3.

5. the audio data generation unit generates the processed audio data corresponding to the audio of the remote participant corresponding to the remote image data, based on a position of the remote image data in the first composite image data, including first positional audio data to be output to a first positional speaker installed at a first position at the first location and second positional audio data to be output to a second positional speaker installed at a second position at the first location; 2. The data processing device according to claim 1.

6. a setting receiving unit that receives settings of attributes of the remote conference; the acquiring unit acquires captured image data and avatar image data of each of the plurality of remote participants; the image data generation unit determines whether to use the captured image data or the avatar image data as the remote image data based on attributes of the remote conference.

2. The data processing device according to claim 1.

7. The computer executes acquiring first image data generated by capturing an image of one or more first participants participating in a remote conference between a first location and a plurality of second locations different from the first location at a first location, first audio data of the one or more first participants, a plurality of remote image data corresponding to a plurality of remote participants participating in the remote conference from the plurality of second locations, and a plurality of remote audio data of the plurality of remote participants; generating first composite image data by combining the plurality of remote image data; generating processed audio data by processing the plurality of remote audio data based on the positions of each of the plurality of remote participants in the first composite image data; transmitting the first composite image data and the processed audio data to one or more electronic devices used by the first participant participating in the remote conference at the first location for viewing the remote conference, and transmitting the first image data and the first audio data to multiple information terminals used by the multiple remote participants; transmitting the first image data to one or more electronic devices used by the first participant to view the remote conference; a step of generating second composite image data by combining one or more remote image data excluding one remote image data among the plurality of remote image data with the first image data, wherein the second composite image data is generated by combining at least a portion of each of the plurality of remote image data at a position specified in the first image data displayed on a display device installed at the first base based on the first image data acquired via the one or more electronic devices; transmitting the second composite image data, which does not include the one remote image data corresponding to the one remote image data, to the information terminal used by the remote participant corresponding to the one remote image data; A data processing method comprising:

8. an electronic device installed at a first base; a plurality of information terminals installed at a plurality of second bases different from the first base; an acquisition unit that acquires first image data generated by capturing an image of one or more first participants participating in a remote conference between the first location and the plurality of second locations at the first location, first audio data of the one or more first participants, a plurality of remote image data corresponding to a plurality of remote participants participating in the remote conference from the plurality of second locations, and a plurality of remote audio data of the plurality of remote participants; an image data generating unit that generates first composite image data by combining the plurality of remote image data, and generates second composite image data by combining one or more remote image data, excluding one remote image data, of the plurality of remote image data with the first image data; an audio data generation unit that generates processed audio data by processing the plurality of remote audio data based on the positions of the plurality of remote participants in the first composite image data; a data transmitting unit that transmits the first composite image data and the processed audio data to one or more of the electronic devices used by the first participant participating in the remote conference at the first location for viewing the remote conference, transmits the first image data and the first audio data to the multiple information terminals used by the multiple remote participants, transmits the second composite image data that does not include the one remote image data corresponding to the one remote participant to the information terminal used by the remote participant corresponding to the one remote image data, and transmits the first image data to one or more of the electronic devices used by the first participant for viewing the remote conference; and the image data generation unit generates the second composite image data by combining at least a portion of each of a plurality of remote image data at a position designated in the first image data displayed on a display device installed at the first site based on the first image data acquired via the one or more electronic devices; Data processing system.

Images