Terminal device, terminal device operation method and program

Abstract

To reduce a user's sense of incongruity in the direction from which a conversation partner's voice is heard.SOLUTION: A terminal device 20 includes a display 24 and a control unit 28. The control unit 28 forms a virtual sound source that outputs the sound emitted by an object 4B at the intersection of a line connecting the position of the mouth of an object 4B placed in a virtual space viewed from a virtual camera in a real space and the position of the head of a user 3A facing the display 24, and the screen of the display 24.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to a terminal device, an operation method for a terminal device, and a program. [Background technology]

[0002] Conventionally, there is known a technique for multiple users to communicate using a virtual space. For example, Non-Patent Document 1 discloses a virtual meeting system using a light field display. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] "Google announces Project Starline, a 3D face-to-face device with a "magic window." Stereoscopic display creates a sense of reality," [online], [Retrieved December 13, 2021], Internet<URL: https: / / japanese.engadget.com / google-project-starline-044709034.html> Summary of the Invention [Problem to be solved by the invention]

[0004] In conventional technology, the direction from which sound is output to the user is fixed, so the user may feel uncomfortable with the direction from which the sound of the conversation partner's voice or other sounds is coming depending on the position of the conversation partner as seen through the display.

[0005] In view of the above, an object of the present disclosure is to provide a technology that reduces the sense of discomfort that a user feels regarding the direction from which a sound emitted by a conversation partner is coming. [Means for solving the problem]

[0006] A terminal device according to an embodiment of the present disclosure includes: A display and The device comprises a control unit which forms a straight line connecting the real-space position of the mouth of an object placed in a virtual space as seen from a virtual camera and the position of the user's head facing the display, and forms a virtual sound source which outputs the sound emitted by the object at the intersection with the screen of the display.

[0007] A method for operating a terminal device according to an embodiment of the present disclosure includes: This includes forming a virtual sound source that outputs sound emitted by an object at the intersection of a straight line connecting the real-space position of the mouth of an object placed in a virtual space as seen from a virtual camera and the position of the user's head facing the display, and the screen of the display.

[0008] A program according to an embodiment of the present disclosure includes: On the computer, The device executes an operation including forming a virtual sound source that outputs sound emitted by an object at the intersection of a straight line connecting the real-space position of the mouth of an object placed in a virtual space as viewed from a virtual camera and the position of the user's head facing the display, and the screen of the display. Effect of the Invention

[0009] According to an embodiment of the present disclosure, it is possible to reduce the sense of discomfort felt by a user with respect to the direction from which a sound emitted by a conversation partner is coming. [Brief description of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating a schematic configuration of a provision system according to an embodiment of the present disclosure. [Diagram 2] FIG. 2 is a block diagram of the provision system shown in FIG. [Diagram 3] 2 is a sequence diagram showing an operation procedure of the providing system shown in FIG. 1. [Figure 4] 3 is a flowchart showing an operation procedure of the terminal device shown in FIG. 2. [Diagram 5] 3 is a flowchart showing an operation procedure of the terminal device shown in FIG. 2. [Figure 6] 11A and 11B are diagrams illustrating other examples of intersections at which virtual sound sources are formed. [Figure 7] 13A and 13B are diagrams illustrating still another example of the intersection where a virtual sound source is formed. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0012] As shown in FIG. 1, the provision system 1 includes at least one server device 10, a terminal device 20A, and a terminal device 20B.

[0013] Hereinafter, when there is no particular distinction between the terminal device 20A and the terminal device 20B, they are also referred to as "terminal device 20." The provision system 1 includes two terminal devices 20. However, the provision system 1 may include two or more terminal devices 20.

[0014] The server device 10 and the terminal device 20 are capable of communicating with each other via a network 2. The network 2 may be any network including a mobile communication network, the Internet, and the like.

[0015] The providing system 1 is a system for providing a virtual event. A virtual event is an event provided by utilizing a virtual space. A virtual event is an event in which a user can participate as a participant using a terminal device 20. In a virtual event, multiple participants can communicate with each other by speaking, etc. In a virtual event, each participant is represented by a three-dimensional model that represents each participant.

[0016] The server device 10 is, for example, a server computer that belongs to a cloud computing system or other computing system and functions as a server that implements various functions. The server device 10 may be configured of two or more server computers that are communicatively connected and operate in cooperation with each other.

[0017] The server device 10 executes processes necessary for providing a virtual event. For example, the server device 10 transmits information necessary for providing a virtual event to the terminal device 20 via the network 2. The server device 10 also mediates the transmission and reception of information between the terminal device 20A and the terminal device 20B.

[0018] The terminal device 20 is, for example, a desktop personal computer (PC), a tablet PC, a notebook PC, a smartphone, or the like.

[0019] The terminal device 20A is used by a user 3A. The terminal device 20B is used by a user 3B. The user 3A participates in a virtual event using the terminal device 20A. The user 3B participates in a virtual event using the terminal device 20B.

[0020] The terminal device 20 includes a display 24 and four speakers 25. However, the number of speakers 25 included in the terminal device is not limited to four. The terminal device 20 may include two or more speakers 25.

[0021] The display 24 is, for example, a Liquid Crystal Display (LCD) or an organic EL (Electro Luminescence) display, etc. The display 24 is, for example, rectangular in shape.

[0022] The four speakers 25 are arranged at the four corners of the rectangular display 24. The speakers 25 may be arranged on a frame around the display 24 or may be embedded in the display 24. However, the speakers 25 may be provided at any position on the display 24. The speakers 25 may be directional speakers. A directional speaker can output a beam of sound waves generated in a beam shape by imparting directionality to sound waves.

[0023] The operation of the terminal device 20 will be described with the terminal device 20A as an example. The terminal device 20A displays a two-dimensional image in which an object 4B is drawn on the display 24. The object 4B is a three-dimensional model representing the user 3B. The partial image 4b ​​displayed on the display 24 is a partial image in which the object 4B is drawn in a two-dimensional image. The terminal device 20A forms a virtual sound source that outputs a sound emitted by the object 4B at the intersection point P1. The virtual sound source is a virtual sound source. The virtual sound source is formed, for example, by adjusting the volume or directivity of two or more speakers 25. Here, the intersection point P1 is an intersection point between a straight line connecting the position of the mouth of the object 4B in the real space and the position of the head of the user 3A and the screen of the display 24. By forming the virtual sound source at such an intersection point P1, the sound emitted by the object 4B is output in a direction from the intersection point P1 toward the head of the user 3A. Therefore, the user 3A can feel as if the voice of the user 3B is being output from the mouth of the object 4B. With this configuration, the sense of discomfort felt by the user 3A with respect to the direction from which the sound emitted by the object 4B, the conversation partner, is coming from can be reduced compared to when the sound source outputting the sound emitted by the object 4B is fixed.

[0024] Hereinafter, the three-dimensional coordinate system of the real space is assumed to be an XYZ coordinate system based on the display 24. However, any coordinate system may be used for the three-dimensional coordinate system of the real space. In this embodiment, the X direction corresponds to the horizontal direction of the screen of the display 24. The Y direction corresponds to the vertical direction of the screen of the display 24. The Z direction corresponds to the facing direction in which the user 3A faces the display 24. The horizontal and vertical directions of the screen of the display 24 may be set appropriately according to the specifications of the display 24.

[0025] (Server device configuration) As shown in FIG. 2, the server device 10 includes a communication unit 11, a storage unit 12, and a control unit 13.

[0026] The communication unit 11 includes at least one communication module that can be connected to the network 2. The communication module is, for example, a communication module that is compatible with standards such as a wired LAN (Local Area Network) or a wireless LAN. However, the communication module is not limited to this. The communication module may be compatible with any communication standard. The communication unit 11 is connected to the network 2 via the wired LAN or wireless LAN by the communication module.

[0027] The storage unit 12 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or a combination of at least two of these. The semiconductor memory is, for example, a random access memory (RAM) or a read only memory (ROM). The RAM is, for example, a static random access memory (SRAM) or a dynamic random access memory (DRAM). The ROM is, for example, an electrically erasable programmable read only memory (EEPROM). The storage unit 12 may function as a main storage device, an auxiliary storage device, or a cache memory. The storage unit 12 stores data used in the operation of the server device 10 and data obtained by the operation of the server device 10.

[0028] The control unit 13 includes at least one processor, at least one dedicated circuit, or a combination of these. The processor is, for example, a general-purpose processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), or a dedicated processor specialized for a specific process. The dedicated circuit is, for example, an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). The control unit 13 executes processes related to the operation of the server device 10 while controlling each part of the server device 10.

[0029] The functions of the server device 10 are realized by executing a processing program according to this embodiment in a processor corresponding to the control unit 13. That is, the functions of the server device 10 are realized by software. The processing program causes a computer to execute the operations of the server device 10, thereby causing the computer to function as the server device 10. That is, the computer functions as the server device 10 by executing the operations of the server device 10 in accordance with the processing program.

[0030] A part or all of the functions of the server device 10 may be realized by a dedicated circuit corresponding to the control unit 13. In other words, a part or all of the functions of the server device 10 may be realized by hardware.

[0031] (Terminal Device Configuration) As shown in FIG. 2, the terminal device 20 includes a communication unit 21, an input unit 22, an output unit 23, a sensor unit 26, a storage unit 27, and a control unit .

[0032] The communication unit 21 is configured to include at least one communication module connectable to the network 2. The communication module is, for example, a communication module compatible with standards such as wired LAN or wireless LAN, or a communication module compatible with mobile communication standards such as LTE (Long Term Evolution), 4G (4th Generation), or 5G (5th Generation). However, the communication module is not limited to this. The communication module may be compatible with any communication standard.

[0033] The input unit 22 can receive input from a user. The input unit 22 includes at least one input interface capable of receiving input from a user. The input interface is, for example, a physical key, a capacitive key, a pointing device, a touch screen or a microphone that is integral with the display 24. However, the input interface is not limited to these.

[0034] The output unit 23 is capable of outputting data. The output unit 23 is configured to include at least one output interface capable of outputting data. The output interface includes the above-mentioned display 24 and speaker 25. However, the output unit 23 may further include an output interface other than the display 24 and speaker 25.

[0035] The sensor unit 26 includes a camera capable of capturing an image of a subject and generating a captured image, and a distance measurement sensor capable of measuring the distance to the subject. The camera is disposed at a position where it can capture an image of a user facing the display 24 as the subject. The camera continuously captures images of the subject at a frame rate of, for example, 15 to 30 fps. The distance measurement sensor is disposed at a position where it can measure the distance from the display 24 to the subject. The distance measurement sensor generates a distance image. The distance image is an image in which the pixel value of each pixel corresponds to the distance. The distance measurement sensor includes, for example, a ToF (Time Of Flight) camera, a LiDAR (Light Detection And Ranging) camera, a stereo camera, or the like.

[0036] The storage unit 27 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or a combination of at least two of these. The semiconductor memory is, for example, a RAM or a ROM. The RAM is, for example, an SRAM or a DRAM. The ROM is, for example, an EEPROM. The storage unit 27 may function as a main storage device, an auxiliary storage device, or a cache memory. The storage unit 27 stores data used in the operation of the terminal device 20 and data obtained by the operation of the terminal device 20. The storage unit 27 stores, for example, position information of each pixel on the screen of the display 24 in an XYZ coordinate system.

[0037] The control unit 28 includes at least one processor, at least one dedicated circuit, or a combination of these. The processor is, for example, a general-purpose processor such as a CPU or a GPU, or a dedicated processor specialized for a specific process. The dedicated circuit is, for example, an FPGA or an ASIC. The control unit 28 executes processes related to the operation of the terminal device 20 while controlling each part of the terminal device 20.

[0038] The functions of the terminal device 20 are realized by executing a terminal program according to this embodiment on a processor corresponding to the control unit 28. That is, the functions of the terminal device 20 are realized by software. The terminal program causes a computer to execute the operations of the terminal device 20, thereby causing the computer to function as the terminal device 20. That is, the computer functions as the terminal device 20 by executing the operations of the terminal device 20 in accordance with the terminal program.

[0039] A part or all of the functions of the terminal device 20 may be realized by a dedicated circuit equivalent to the control unit 28. In other words, a part or all of the functions of the terminal device 20 may be realized by hardware.

[0040] (Operation of the provided system) Fig. 3 is a sequence diagram showing the operation procedure of the provision system 1 shown in Fig. 1. Hereinafter, it is assumed that the user 3A sets a virtual event as a virtual event manager. In addition, it is assumed that the user 3A and the user 3B participate in the virtual event as participants.

[0041] In the process of step S1, in the terminal device 20A, the control unit 28 accepts an input of setting information from the user 3A through the input unit 22. The setting information is information for setting a virtual event. The setting information includes, for example, a schedule of the virtual event, a discussion topic, and a participant list. The participant list includes the names of the participants and their e-mail addresses. Here, the participant list includes the name of the user 3B who is a participant, and the e-mail address of the user 3B. For example, the control unit 28 accesses a site for setting a virtual event provided by the server device 10 through the communication unit 21, and acquires data of an input screen for inputting setting information. The control unit 28 displays the input screen on the display 24 and presents it to the user 3A. The user 3A looks at the input screen and inputs setting information through the input unit 22.

[0042] In the process of step S2, in the terminal device 20A, the control unit 28 transmits the setting information received by the input unit 22 to the server device 10 via the network 2 through the communication unit 21.

[0043] In the process of step S3, in the server device 10, the control unit 13 receives the setting information from the terminal device 20A via the network 2 through the communication unit 11.

[0044] In the process of step S4, in the server device 10, the control unit 13 sets the virtual event based on the setting information received in the process of step S3. For example, the control unit 13 generates authentication information. The authentication information is information for authenticating the user 3B who participates in the virtual event using the terminal device 20B. The authentication information includes a participant ID and a passcode, etc. The participant ID is identification information used when the user 3B participates in the virtual event as a participant.

[0045] In the process of step S5, in the server device 10, the control unit 13 transmits the generated authentication information to the terminal device 20B via the network 2 by the communication unit 11. For example, the control unit 13 transmits the authentication information to the terminal device 20B by attaching it to an e-mail.

[0046] In the process of step S6, in the terminal device 20B, the control unit 28 receives the authentication information from the server device 10 via the network 2 by the communication unit 21. The control unit 28 receives the authentication information attached to the e-mail.

[0047] In the process of step S7, in the terminal device 20B, the control unit 28 accepts input of authentication information and participation application information from the user 3B through the input unit 22. For example, the control unit 28 accesses a site for setting up a virtual event provided by the server device 10 through the communication unit 21, and acquires data of an input screen for inputting the authentication information and participation application information. The control unit 28 displays the input screen on the display 24 and presents it to the user 3B. The user 3B looks at the input screen and inputs the authentication information attached to the e-mail and the participation application information through the input unit 22.

[0048] In the process of step S8, in the terminal device 20B, the control unit 28 transmits the authentication information and participation application information received by the input unit 22 to the server device 10 via the network 2 through the communication unit 21.

[0049] In the process of step S9, in the server device 10, the control unit 13 receives the authentication information and the participation application information from the terminal device 20B via the network 2 by the communication unit 11. By receiving the authentication information and the participation application information, the control unit 13 completes the acceptance of the participation of the user 3B (step S10).

[0050] In the process of step S11, in the server device 10, the control unit 13 transmits, via the network 2, a notification of the start of the event to each of the terminal devices 20A and 20B through the communication unit 11.

[0051] In the process of step S12, in terminal device 20A, control unit 28 receives an event start notification from server device 10 via network 2 through communication unit 21. Upon receiving the event start notification, control unit 28 starts collecting sounds such as speech uttered by user 3A and starts capturing images of user 3A.

[0052] In the process of step S13, in terminal device 20B, control unit 28 receives an event start notification from server device 10 via network 2 through communication unit 21. Upon receiving the event start notification, control unit 28 starts collecting sounds such as speech uttered by user 3B and starts capturing images of user 3B.

[0053] In the process of step S14, the terminal device 20A and the terminal device 20B carry out a virtual event via the server device 10.

[0054] (Terminal device operation) Fig. 4 is a flowchart showing the operation procedure of the terminal device 20 shown in Fig. 2. The operation procedure shown in Fig. 4 is common to the terminal devices 20A and 20B. The operation procedure shown in Fig. 4 is an example of an operation method of the terminal device 20 according to this embodiment. The operation procedure shown in Fig. 4 is executed in the processing of step S14 as shown in Fig. 3. The following description will be given assuming that the terminal device 20B executes the operation procedure shown in Fig. 4.

[0055] In the process of step S21, the control unit 28 acquires captured image and distance image data of the user 3B by the sensor unit 26. The control unit 28 acquires sound data by collecting sounds such as speech uttered by the user 3B by the microphone of the input unit 22.

[0056] In the process of step S22, the control unit 28 generates encoded data by encoding the captured image data of the user 3B, the distance image data of the user 3B, and the sound data of the user 3B. This encoded data is used to generate the object 4B, which is a three-dimensional model representing the user 3B as described above, in the terminal device 20A. In the encoding, the control unit 28 may perform any processing (e.g., changing the resolution, trimming, etc.) on the captured image, etc.

[0057] In the process of step S23, the control unit 28 transmits the encoded data as a packet to the server device 10 via the network 2 by the communication unit 21. The encoded data is then transmitted via the server device 10 to the terminal device 20A.

[0058] In the process of step S24, the control unit 28 judges whether or not an input to interrupt image capture and sound collection or an input to exit from the virtual event has been received by the input unit 22. If the control unit 28 judges that an input to interrupt image capture and sound collection or an input to exit from the virtual event has been received (step S24: YES), the control unit 28 ends the operation procedure shown in Fig. 4. If the control unit 28 does not judge that an input to interrupt image capture and sound collection or an input to exit from the virtual event has been received (step S24: NO), the control unit 28 returns to the process of step S21.

[0059] Fig. 5 is a flowchart showing the operation procedure of the terminal device 20 shown in Fig. 2. The operation procedure shown in Fig. 5 is common to the terminal devices 20A and 20B. The operation procedure shown in Fig. 5 is an example of an operation method of the terminal device 20 according to this embodiment. The operation procedure shown in Fig. 5 is executed in the processing of step S14 as shown in Fig. 3. The following description will be given assuming that the terminal device 20A executes the operation procedure shown in Fig. 5.

[0060] In the process of step S31, the control unit 28 receives the encoded data from the terminal device 20B via the network 2 and the server device 10 through the communication unit 21.

[0061] In the process of step S32, the control unit 28 decodes the received encoded data. By decoding the encoded data, the control unit 28 acquires captured image data of the user 3B, distance image data of the user 3B, and sound data of the user 3B.

[0062] In the process of step S33, the control unit 28 generates the object 4B using the captured image and distance image data of the user 3B. For example, the control unit 28 generates a polygon model using the distance image data of the user 3B, and generates the object 4B by applying texture mapping to the polygon model using the captured image data of the user 3B. However, the generation of the object as a three-dimensional model is not limited to this. The control unit 28 may generate the object by employing any method. The control unit 28 places the generated object 4B and a virtual camera in the virtual space. The control unit 28 may appropriately adjust the position, direction, and field of view of the virtual camera based on the operation input of the user 3A received by the input unit 22.

[0063] In the process of step S34, the control unit 28 identifies the position of the mouth of the object 4B arranged in the virtual space as seen from the virtual camera. The control unit 28 may identify the position of the mouth of the object 4B by any method such as image analysis.

[0064] In the process of step S35, the control unit 28 specifies the position of the intersection P1 that forms the virtual sound source as described above with reference to Fig. 1. As described above, the intersection P1 is the intersection of the straight line that connects the position of the mouth of the object 4B in the real space and the position of the head of the user 3A with the screen of the display 24.

[0065] In the process of step S35, the control unit 28 specifies the position of the intersection P1 between the line connecting the position of the mouth of the object 4B and the position of the head of the user 3A in the XYZ coordinate system and the screen of the display 24. That is, the position of the intersection P1 is specified as a position in the XYZ coordinate system. In this case, the control unit 28 specifies the position of the XYZ coordinate system of the mouth of the object 4B from the position in the virtual space of the mouth of the object 4B specified in the process of step S34 by a process of converting the coordinate system of the virtual space into the XYZ coordinate system. In addition, the control unit 28 specifies the position of the XYZ coordinate system of the head of the user 3A based on the data of the captured image and the distance image of the user 3A acquired by the sensor unit 26. The control unit 28 specifies the position of the intersection P1 based on the specified positions of the XYZ coordinate system of the mouth of the object 4B and the XYZ coordinate system of the head of the user 3A, and the position information of the XYZ coordinate system of each pixel on the screen of the display 24.

[0066] In the process of step S36, the control unit 28 generates, by rendering, a two-dimensional image of the virtual space in which the object 4B is placed, as seen from the virtual camera.

[0067] In the process of step S37, the control unit 28 causes the display 24 to display the two-dimensional image generated in the process of step 36. The control unit 28 also forms a virtual sound source at the intersection P1 identified in the process of step S35 by appropriately adjusting the volume or directivity, etc., of two or more speakers 25. The control unit 28 causes the sound of the user 3B acquired in the process of step S32 to be output from the virtual sound source as a sound emitted by the object 4B. After executing the process of step S37, the control unit 28 ends the operation procedure shown in FIG. 5. However, when the encoded data is transmitted from the terminal device 20B to the terminal device 20A, the control unit 28 executes the operation procedure from the process of step S31 again.

[0068] In this way, a virtual sound source that outputs the sound emitted by the object 4B is formed at the intersection P1 by the processing of steps S35 to S37. The sound emitted by the object 4B is output in a direction from the intersection P1 toward the head of the user 3A. Therefore, the user 3A can feel as if the voice of the user 3B is being output from the mouth of the object 4B. With this configuration, it is possible to reduce the sense of discomfort felt by the user 3A in the direction from which the sound emitted by the object 4B, the conversation partner, is coming, compared to a case in which the sound source that outputs the sound emitted by the object 4B is fixed. Furthermore, by specifying the position of the intersection P1 as a position in the XYZ coordinate system, it is possible to further reduce the sense of discomfort felt by the user 3A.

[0069] Here, in the process of step S35, the position of the intersection P1 is specified as a position of the XYZ coordinate system. However, the position of the intersection where the virtual sound source is formed is not limited to the position of the intersection P1 of the XYZ coordinate system. The position of the intersection where the virtual sound source is formed may be specified as a position of a two-dimensional coordinate system in real space. Other examples of the intersection where the virtual sound source is formed will be described with reference to Figs. 6 and 7.

[0070] Fig. 6 is a diagram for explaining another example of the intersection where a virtual sound source is formed. In Fig. 6, the position of the head of an object 4B placed in the virtual space is indicated by a dashed line in the real space.

[0071] 6, the position of the intersection P1 where the virtual sound source is formed is specified as a position in the YZ coordinate system. The YZ coordinate system is a two-dimensional coordinate system in real space that includes a Y direction that is the vertical direction of the display 24 and a Z direction that is the facing direction in which the user 3A faces the display 24.

[0072] The control unit 28 specifies the position of an intersection P1 between a line connecting the position of the mouth of the object 4B and the position of the head of the user 3A in the YZ coordinate system and the screen of the display 24. The control unit 29 forms a virtual sound source that outputs a sound emitted by the object 4B at the intersection P1.

[0073] By specifying the position of the intersection point P1 as a position in the YZ coordinate system, in the YZ coordinate system, the sound emitted by the object 4B1 is output in a direction from the intersection point P1 toward the head of the user 3A. With this configuration, it is possible to reduce the sense of discomfort felt by the user 3A with respect to the direction from which the sound emitted by the object 4B is coming, compared to a case in which the sound source that outputs the sound emitted by the object 4B is fixed.

[0074] Fig. 7 is a diagram for explaining another example of the intersection where a virtual sound source is formed. In Fig. 7, the position of the head of an object 4B placed in the virtual space is indicated by a dashed line in the real space.

[0075] 7, the position of the intersection P1 where the virtual sound source is formed is specified as a position in the XZ coordinate system. The XZ coordinate system is a two-dimensional coordinate system in real space that includes an X direction that is the lateral direction of the display 24 and a Z direction that is the facing direction in which the user 3A faces the display 24.

[0076] The control unit 28 specifies the position of an intersection P1 between a line connecting the position of the mouth of the object 4B and the position of the head of the user 3A in the XZ coordinate system and the screen of the display 24. The control unit 28 forms a virtual sound source that outputs a sound emitted by the object 4B2 at the intersection P1.

[0077] By specifying the position of the intersection point P1 as a position in the XZ coordinate system, the sound emitted by the object 4B is output in a direction from the intersection point P1 toward the head of the user 3A in the XZ coordinate system. With this configuration, it is possible to reduce the sense of discomfort felt by the user 3A with respect to the direction from which the sound emitted by the object 4B is coming, compared to a case in which the sound source that outputs the sound emitted by the object 4B is fixed.

[0078] Here, when the processes of steps S31 to S37 described above are repeatedly executed, in the process of step S34, the control unit 28 may newly identify the position of the mouth of the object 4B when the object 4B moves. In this case, in the process of step S35, the control unit 28 may newly identify the position of the intersection point P1 using the newly identified position of the mouth of the object 4B. Also, in the process of step S37, the control unit 28 may form a virtual sound source that outputs the sound emitted by the object 4B at the newly identified intersection point P1. With this configuration, even if the object 4B moves due to the movement of the user 3B, it is possible to reduce the sense of discomfort felt by the user 3A in the direction from which the sound emitted by the object 4B is coming.

[0079] Furthermore, when the processes of steps S31 to S37 described above are repeatedly executed, in the process of step S35, the control unit 28 may newly identify the position of the head of the user 3A when the head of the user 3A moves. The control unit 28 may newly identify the position of the intersection point P1 using the newly identified position of the head of the user 3A. In this case, in the process of step S37, the control unit 28 may form a virtual sound source that outputs the sound emitted by the object 4B at the newly identified intersection point P1. With this configuration, even if the head of the user 3A moves due to the movement of the user 3A, it is possible to reduce the sense of discomfort felt by the user 3A in the direction from which the sound emitted by the object 4B is coming.

[0080] Also, in the above description, the object is described as being one. However, the object may be multiple. For example, when the participants participating in the virtual event are three or more, the object may be multiple. When the object is multiple, the control unit 28 may form multiple virtual sound sources that output the sounds emitted by the multiple objects at each intersection of the lines connecting the positions of the mouths of the multiple objects in the real space and the position of the head of the user 3A, respectively, with the screen of the display 24. In this case, the number of the multiple virtual sound sources may be the same as the number of the multiple objects. With this configuration, even if there are multiple conversation partners, it is possible to reduce the sense of discomfort felt by the user 3A in the direction from which the sounds emitted by the multiple objects are coming.

[0081] In the above, the control unit 28 may form a virtual sound source that outputs the sound emitted by the object 4B at an intersection P1 between a straight line connecting the position of the mouth of the object 4B in the real space and the center position of the face of the head of the user 3A and the screen of the display 24. The control unit 28 may select such a configuration when the face of the user 3A is facing the screen of the display 24. With such a configuration, the sound emitted by the object 4B is output in a direction toward the center of the face of the user 3A from the virtual sound source formed at the intersection P1. By outputting the sound emitted by the object 4B in a direction toward the center of the face of the user 3A, the sound emitted by the object 4B is output evenly to the two ears of the user 3A. By outputting the sound emitted by the object 4B evenly to the two ears of the user 3A, the discomfort felt by the user 3A can be further reduced.

[0082] In addition, in the above, the control unit 28 may form a virtual sound source that outputs the sound emitted by the object 4B at an intersection P1 between a straight line connecting the position of the mouth of the object 4B in the real space and the position of one of the two ears of the head of the user 3A and the screen of the display 24. The control unit 28 may select the ear closer to the screen of the display 24 from the two ears of the user 3A. The control unit 28 may specify the position of the intersection P1 using the position of the selected ear. With this configuration, the sound emitted by the object 4B is output in a direction from the virtual sound source formed at the intersection P1 toward the ear of the user 3B. By outputting the sound emitted by the object 4B in a direction toward the ear of the user 3B, the discomfort felt by the user 3A can be further reduced.

[0083] In addition, in the above, the control unit 28 may form a virtual sound source that outputs the sound emitted by the object 4B at each intersection P1 between the line connecting the position of the mouth of the object 4B in the real space and the positions of the two ears of the user 3A and the screen of the display 24. In this case, two virtual sound sources corresponding to the left and right ears of the user 3A, respectively, are formed. With such a configuration, the sound emitted by the object 4B is output to both ears of the user 3A, so that the discomfort felt by the user 3A can be further reduced.

[0084] In this manner, in the terminal device 20 according to the present embodiment, for example, the control unit 28 forms a virtual sound source that outputs the sound emitted by the object 4B at the intersection P1. With this configuration, it is possible to reduce the sense of discomfort felt by the user 3A with respect to the direction from which the sound emitted by the object 4B, the conversation partner, comes, compared to a case in which the sound source that outputs the sound emitted by the object 4B is fixed.

[0085] Although the present disclosure has been described based on the drawings and examples, it should be noted that those skilled in the art may make various modifications and alterations based on the present disclosure. Therefore, it should be noted that these modifications and alterations are included in the scope of the present disclosure. For example, the functions included in each component or step can be rearranged so as not to be logically inconsistent, and multiple components or steps can be combined or divided into one.

[0086] For example, in the above-described embodiment, the control unit 28 has been described as specifying the position of the intersection point P1 between the line connecting the position of the mouth of the object 4B in the real space and the position of the head of the user 3A facing the display 24 and the screen of the display 24. That is, the control unit 28 has been described as specifying the position of the intersection point P1 in a coordinate system in the real space. However, the control unit 28 may specify the position of the intersection point P1 in a coordinate system in the virtual space. That is, the control unit 28 may specify the position of the intersection point P1 between the line connecting the position of the mouth of the object 4B and the position of the head of the user 3A facing the display 24 and the screen of the display 24 in the coordinate system in the virtual space.

[0087] For example, in the above-described embodiment, the terminal device 20A and the terminal device 20B have been described as implementing a virtual event via the server device 10. However, the terminal device 20A and the terminal device 20B may implement a virtual event without the intervention of the server device 10. As an example, the terminal device 20A and the terminal device 20B may implement a virtual event in a state where they are connected via a P2P (Peer to Peer) type architecture.

[0088] For example, an embodiment is also possible in which a general-purpose computer functions as the terminal device 20 according to the above-described embodiment. Specifically, a program describing the processing contents for realizing each function of the terminal device 20 according to the above-described embodiment is stored in the memory of the general-purpose computer, and the program is read and executed by a processor. Therefore, the present disclosure can also be realized as a program executable by a processor, or a non-transitory computer-readable medium that stores the program. [Explanation of symbols]

[0089] 1. Provision system 2 Network 3A,3B User 4B Object 4b Partial image 10. Server device 11 Communications Department 12 Storage section 13 Control section 20, 20A, 20B Terminal equipment 21 Communications Department 22 Input section 23 Output section 24 Display 25 Speakers 26 Sensor section 27 Memory section 28 Control Unit

Claims

1. A display and Two or more speakers disposed on the display; A terminal device comprising: a control unit that adjusts the volume or directivity of the two or more speakers to form a virtual sound source that outputs the sound emitted by the object at the intersection of a straight line connecting the real space position of the mouth of an object placed in a virtual space as seen from a virtual camera and the position of the user's head facing the display and the screen of the display.

2. The control unit is When identifying a position of an intersection that forms the virtual sound source using the identified head position of the user, When the user's head moves, a new position of the user's head is identified; The terminal device according to claim 1 , further comprising: a step of: determining a new position of the intersection using a newly determined position of the user's head.

3. The control unit is When the position of the intersection that forms the virtual sound source is identified using the identified position of the mouth of the object, determining a new position of the object's mouth when the object moves; The terminal device according to claim 1 , further comprising: a step of: determining a new position of the intersection point using a newly determined position of a mouth of the object;

4. A terminal device according to any one of claims 1 to 3, wherein the control unit forms the virtual sound source at the intersection of a straight line connecting the real-space position of the object's mouth and the center position of the face of the user's head and the screen of the display.

5. The terminal device according to any one of claims 1 to 3, wherein the control unit forms the virtual sound source at the intersection of a straight line connecting the real-space position of the object's mouth and the position of one of the two ears of the user's head with the screen of the display.

6. The terminal device according to any one of claims 1 to 3, wherein the control unit forms the virtual sound source at each intersection of a straight line connecting the real-space position of the object's mouth and the positions of two ears of the user's head with the screen of the display.

7. The terminal device according to any one of claims 1 to 6, wherein the control unit forms the virtual sound source at the intersection of the straight line connecting the position of the object's mouth and the position of the user's head in a three-dimensional coordinate system in real space with the screen of the display.

8. The terminal device according to any one of claims 1 to 6, wherein the control unit forms the virtual sound source at the intersection of the straight line connecting the position of the object's mouth and the position of the user's head in a two-dimensional coordinate system in real space and the screen of the display.

9. The terminal device according to claim 8 , wherein the two-dimensional coordinate system in the real space is a coordinate system that includes a vertical direction of the display and a facing direction in which the user faces the display.

10. The terminal device according to claim 8 , wherein the two-dimensional coordinate system in the real space is a coordinate system that includes a lateral direction of the display and a facing direction in which the user faces the display.

11. The terminal device according to any one of claims 1 to 10, wherein the control unit forms a plurality of virtual sound sources that output sounds emitted by each of the plurality of objects at each intersection of a straight line connecting the real-space positions of the mouths of the plurality of objects with the position of the user's head and the screen of the display.

12. A method for operating a terminal device, comprising adjusting the volume or directivity of two or more speakers arranged on a display to form a virtual sound source that outputs sound emitted by an object at the intersection of a straight line connecting the real space position of the mouth of an object arranged in a virtual space as seen by a virtual camera and the position of the user's head facing the display, and the screen of the display.

13. On the computer, A program that executes an operation including forming a virtual sound source that outputs sound emitted by an object at the intersection of a straight line connecting the real-space position of the mouth of an object placed in a virtual space as seen from a virtual camera and the position of the user's head facing the display, and the screen of the display, by adjusting the volume or directivity of two or more speakers arranged on a display.

Images