Avatar-based conference processing method and system

Abstract

A conference processing method and system using avatars is provided. [Solution] A conference processing method according to one embodiment includes the steps of: setting up a communication session for a conference in which multiple users participate through a server; transmitting to the server an identifier of an avatar to be represented in a virtual space for the conference and coordinate information of the avatar in the virtual space; receiving from the server resources of a peripheral avatar selected based on the coordinate information; transmitting motion data of the avatar to the server via the communication session; receiving motion data of the peripheral avatar from the server via the communication session; and representing the peripheral avatar in the virtual space based on the resources of the peripheral avatar and the motion data of the peripheral avatar.

Description

[Technical field]

[0001] The following description relates to an avatar-based conference processing method and system. [Background technology]

[0002] An avatar is a character that represents one's self online, and is attracting attention as a tool for user expression because it can provide a realistic virtual environment by interacting with others in the same way as in the real world. Such avatars are widely used in various fields such as advertising, film production, game design, and teleconference.

[0003] However, in conventional technology, on services with many participants, only avatars were provided that performed actions selected by the user from pre-defined actions (avatar movements and / or facial expressions), and it was not possible to display avatars that imitate the actions of participants in real time on the service. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Korean Patent Publication No. 10-2009-0058760 Summary of the Invention [Problem to be solved by the invention]

[0005] To provide a conference processing method and system capable of expressing avatars of participants imitating the actions of the participants in a virtual space shared by the participants and proceeding with the conference by utilizing such a virtual space. [Means for solving the problem]

[0006] The present invention provides a conference processing method for a computer device including at least one processor, the conference processing method including: a step of setting, by the at least one processor, a communication session for a conference in which a plurality of users participate through a server; a step of transmitting, by the at least one processor, to the server, an identifier of an avatar to be represented in a virtual space for the conference and coordinate information of the avatar in the virtual space, by the at least one processor; a step of receiving, by the at least one processor, resources of a peripheral avatar selected based on the coordinate information from the server; a step of transmitting, by the at least one processor, motion data of the avatar to the server via the communication session, by the at least one processor; a step of receiving, by the at least one processor, motion data of the peripheral avatar from the server via the communication session, and a step of expressing, by the at least one processor, the peripheral avatar in the virtual space based on the resources of the peripheral avatar and the motion data of the peripheral avatar.

[0007] According to one aspect, the conference processing method may further include displaying the virtual space in which the avatar and the surrounding avatars are represented on a screen.

[0008] According to another aspect, the step of receiving the motion data may be characterized as receiving the motion data generated for the peripheral avatar in real time using a real-time transmission protocol.

[0009] According to another aspect, the conference processing method may further include, when receiving motion data of a new peripheral avatar, requesting resources of the new peripheral avatar from the server, and receiving the resources of the new peripheral avatar from the server.

[0010] According to another aspect, the conference processing method may further include sharing the audio of the multiple users through the server via the communication session or another communication session set separately from the communication session.

[0011] According to yet another aspect, the motion data may include data regarding at least one of poses and facial expressions of the users.

[0012] According to another aspect, the avatar pose may be configured to include a plurality of bones, and the motion data may include at least one of an index of each of the plurality of bones, rotation information in three-dimensional space for each of the plurality of bones, position information in the virtual space for each of the plurality of bones, and a current tracking state for each of the plurality of bones.

[0013] According to yet another aspect, the motion data may include coefficient values ​​calculated for a plurality of predefined points on a human face based on a face blendshape technique.

[0014] Provided is a conference processing method for a computer device including at least one processor, the conference processing method including a step of setting up a communication session for a conference in which a plurality of users participate, by the at least one processor; a step of receiving, by the at least one processor, from each of the terminals of the plurality of users, an identifier of an avatar represented in a virtual space for the conference and coordinate information of the avatar in the virtual space, by the at least one processor; a step of selecting, by the at least one processor, a peripheral avatar of a specific avatar based on the coordinate information; a step of transmitting, by the at least one processor, resources of the selected peripheral avatar to a terminal of a user corresponding to the specific avatar; a step of receiving, by the at least one processor, avatar motion data from each of the terminals of the plurality of users via the communication session, by the at least one processor; and a step of transmitting, by the at least one processor, the motion data of the selected peripheral avatar to the terminal of the user corresponding to the specific avatar via the communication session.

[0015] A computer program is provided that is recorded on a computer-readable recording medium and that, when combined with a computer device, causes the computer device to execute the method.

[0016] A computer-readable recording medium is provided on which a program for causing a computer device to execute the method is recorded.

[0017] A computer device is provided, comprising at least one processor implemented to execute computer-readable instructions, the at least one processor setting up a communication session for a conference in which a plurality of users participate through a server, transmitting to the server an identifier of an avatar to be represented in a virtual space for the conference and coordinate information of the avatar in the virtual space, receiving from the server resources of a peripheral avatar selected based on the coordinate information, transmitting motion data of the avatar to the server, receiving the motion data of the peripheral avatar from the server, and representing the peripheral avatar in the virtual space based on the resources of the peripheral avatar and the motion data of the peripheral avatar.

[0018] A computer device is provided, comprising at least one processor implemented to execute computer-readable instructions, the at least one processor setting up a communication session for a conference in which a plurality of users participate, receiving, from each of the plurality of users' terminals, an identifier of an avatar represented in a virtual space for the conference and coordinate information of the avatar in the virtual space, selecting a peripheral avatar of a specific avatar based on the coordinate information, transmitting resources of the selected peripheral avatar to a terminal of a user corresponding to the specific avatar, receiving avatar motion data from each of the plurality of users' terminals, and transmitting the motion data of the selected peripheral avatar to the terminal of the user corresponding to the specific avatar. Effect of the Invention

[0019] The avatars of the participants that mimic the movements of the participants in the virtual space are displayed in a virtual space shared by the participants, and the conference can be carried out using such a virtual space. [Brief description of the drawings]

[0020] [Figure 1] FIG. 1 illustrates an example of a network environment in accordance with an embodiment of the present invention. [Diagram 2] FIG. 2 is a block diagram illustrating an example of a computing device in accordance with an embodiment of the present invention. [Diagram 3] FIG. 2 is a diagram showing an example of a method for expressing a client's avatar in one embodiment of the present invention. [Figure 4] FIG. 13 is a diagram showing an example of a method for expressing an avatar of a server in one embodiment of the present invention. [Diagram 5] FIG. 2 is a flow diagram illustrating an example of an avatar rendering process in accordance with an embodiment of the present invention. [Figure 6] FIG. 2 is a flow diagram illustrating an example of an avatar rendering process in accordance with an embodiment of the present invention. [Figure 7] FIG. 2 is a flow diagram illustrating an example of an avatar rendering process in accordance with an embodiment of the present invention. [Figure 8] FIG. 2 is a flow diagram illustrating an example of an avatar rendering process in accordance with an embodiment of the present invention. [Figure 9] FIG. 2 is a diagram showing an example of an avatar bone structure in one embodiment of the present invention. [Figure 10] FIG. 1 illustrates an example of a conference using avatars in accordance with an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings.

[0022] A conference processing system according to an embodiment of the present invention may include a computer device implementing at least one client and a computer device implementing at least one server, and a conference processing method according to an embodiment of the present invention may be executed by at least one computer device included in the conference processing system. In this case, a computer program according to an embodiment of the present invention may be installed and executed in the computer device, and the computer device may execute the conference processing method according to an embodiment of the present invention under the control of the executed computer program. The above-mentioned computer program may be recorded in a computer-readable recording medium in combination with the computer device to cause the computer to execute the conference processing method.

[0023] Fig. 1 is a diagram showing an example of a network environment in one embodiment of the present invention. The network environment in Fig. 1 shows an example including a plurality of electronic devices 110, 120, 130, 140, a plurality of servers 150, 160, and a network 170. Such Fig. 1 is merely an example for explaining the invention, and the number of electronic devices and the number of servers should not be limited to those shown in Fig. 1. Furthermore, the network environment in Fig. 1 is merely an example of an environment applicable to this embodiment, and the environment applicable to this embodiment should not be limited to the network environment in Fig. 1.

[0024] The electronic devices 110, 120, 130, 140 may be fixed terminals or mobile terminals realized by computer devices. Examples of the electronic devices 110, 120, 130, 140 include a smartphone, a mobile phone, a navigation system, a computer, a notebook PC (Personal Computer), a digital broadcasting terminal, a PDA (Personal Digital Assistant), a PMP (Portable Multimedia Player), a tablet PC, etc. As an example, FIG. 1 shows a smartphone as an example of the electronic device 110, but in the embodiment of the present invention, the electronic device 110 may mean one of various physical computer devices that can communicate with other electronic devices 120, 130, 140 and / or servers 150, 160 via a network 170 using a substantially wireless or wired communication method.

[0025] The communication method is not limited, and may include not only a communication method using a communication network (for example, a mobile communication network, a wired Internet, a wireless Internet, or a broadcasting network) that the network 170 may include, but also short-range wireless communication between devices. For example, the network 170 may include any one or more of a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), a broadband network (BBN), the Internet, and the like. Furthermore, the network 170 may include any one or more of a network topology including, but not limited to, a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree or hierarchical network, and the like.

[0026] Each of the servers 150, 160 may be realized by one or more computer devices that communicate with the multiple electronic devices 110, 120, 130, 140 via the network 170 to provide instructions, codes, files, content, services, etc. For example, the server 150 may be a system that provides services (such as an instant messaging service, a game service, a group calling service (or an audio conference service), a messaging service, an email service, a social networking service, a map service, a translation service, a financial service, a payment service, a search service, a content providing service, etc.) to the multiple electronic devices 110, 120, 130, 140 connected via the network 170.

[0027] 2 is a block diagram showing an example of a computer device according to an embodiment of the present invention. Each of the electronic devices 110, 120, 130, and 140 and the servers 150 and 160 described above may be realized by a computer device 200 shown in FIG.

[0028] Such a computer device 200 may include a memory 210, a processor 220, a communication interface 230, and an input / output interface 240, as shown in FIG. 2. The memory 210 is a computer-readable recording medium and may include a random access memory (RAM), a read only memory (ROM), and a permanent mass storage device such as a disk drive. Here, a permanent mass storage device such as a ROM or a disk drive may be included in the computer device 200 as a separate permanent storage device separate from the memory 210. The memory 210 may also store an operating system and at least one program code. Such software components may be loaded into the memory 210 from a computer-readable recording medium separate from the memory 210. Such separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, a memory card, etc. In another embodiment, the software components may be loaded into the memory 210 through the communication interface 230 rather than from a computer-readable recording medium. For example, the software components may be loaded into the memory 210 of the computing device 200 based on a computer program installed by a file received over the network 170 .

[0029] The processor 220 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. The instructions may be provided to the processor 220 by the memory 210 or the communication interface 230. For example, the processor 220 may be configured to execute instructions received according to program code stored in a storage device such as the memory 210.

[0030] The communication interface 230 may provide a function for the computer device 200 to communicate with other devices (for example, the above-mentioned recording device) via the network 170. For example, requests, instructions, data, files, etc. generated by the processor 220 of the computer device 200 according to program codes recorded in a recording device such as the memory 210 may be transmitted to other devices via the network 170 under the control of the communication interface 230. Conversely, signals, instructions, data, files, etc. from other devices may be received by the computer device 200 via the communication interface 230 of the computer device 200 via the network 170. The signals, instructions, data, etc. received via the communication interface 230 may be transmitted to the processor 220 or the memory 210, and the files, etc. may be recorded on a recording medium (the above-mentioned permanent recording device) that the computer device 200 may further include.

[0031] The input / output interface 240 may be a means for interfacing with the input / output device 250. For example, the input device may include a device such as a microphone, a keyboard, or a mouse, and the output device may include a device such as a display or a speaker. As another example, the input / output interface 240 may be a means for interfacing with a device in which functions for input and output are integrated into one, such as a touch screen. At least one of the input / output devices 250 may be configured as a single device together with the computer device 200. For example, a touch screen, a microphone, a speaker, and the like may be realized in a form in which they are included in the computer device 200, such as a smartphone.

[0032] Also, in other embodiments, computing device 200 may include fewer or more components than those of Figure 2. However, most prior art components need not be explicitly illustrated. For example, computing device 200 may be implemented to include at least some of the input / output devices 250 described above, and may further include other components such as a transceiver, a database, etc.

[0033] FIG. 3 is a diagram showing an example of a conference processing method of a client in an embodiment of the present invention. The conference processing method according to the present embodiment may be executed by a computer device 200 that realizes a client device. Here, the client device may be a subject that receives the provision of a conference service from a server under the control of a client program installed in the client device. Here, the client program may correspond to an application for providing the conference service. In this case, the processor 220 of the computer device 200 may be realized to execute control instructions according to the operating system code and at least one computer program code included in the memory 210. Here, the processor 220 may control the computer device 200 so that the computer device 200 executes steps 310 to 380 included in the method of FIG. 3 according to the control instructions provided by the code recorded in the computer device 200.

[0034] In step 310, the computer device 200 may set up a communication session for a conference in which multiple users participate through a server. For example, a communication session for the terminals of multiple users participating in the conference may be set up by a method in which a user opens a conference using a user interface provided by a conference service and then invites other users to the conference.

[0035] In step 320, the computer device 200 may share the voices of the multiple users through the server via a communication session or another communication session set separately from the communication session. For example, the server may receive the voices of the multiple users, mix the received voices, and transmit the mixed voices to the multiple users, thereby sharing the voices. In this case, the mixed voice transmitted to a specific user may be audio data obtained by mixing the remaining voices except for the voice of the specific user. Depending on the embodiment, a communication protocol for transmitting and receiving the voice may be different from a communication protocol for transmitting and receiving the data described below. For example, RTP (Real-time Transport Protocol) may be used for transmitting and receiving the voice, and RTSP (Real-Time Streaming Protocol) may be used for transmitting and receiving the data described below. Such step 320 may be performed after step 310, or may be performed in parallel with steps 330 to 380 described below. Depending on the embodiment, step 320 may be omitted.

[0036] In step 330, the computer device 200 may transmit to the server an identifier of an avatar represented in the virtual space for the conference and coordinate information of the avatar in the virtual space. At this time, each participant participating in the conference transmits the identifier and coordinate information of his / her avatar to the server, so that the server can grasp the position of each avatar for the conference.

[0037] In step 340, the computer device 200 may receive resources of the neighboring avatars selected based on the coordinate information from the server. For example, the server may select, for a particular avatar, avatars existing within a preset distance from the particular avatar as neighboring avatars, and may transmit the resources of the selected neighboring avatars to the computer device 200. This is because the computer device 200 does not need to represent avatars of all the participants in the conference, but only avatars that may be displayed on the screen. Here, the resources of the avatars may include data for representing the corresponding avatar on the screen.

[0038] In step 350, the computing device 200 may transmit the motion data of the avatar to the server via the communication session. As an example, the computing device 200 may obtain the motion data of the user by analyzing an image captured by a camera included in the computing device 200, and may transmit the motion data of the user as the motion data of the corresponding avatar to the server. The motion data may include data on at least one of the pose and facial expression of the corresponding user. In another embodiment, the motion data may include data on a motion selected by the corresponding user on the computing device 200 from a number of preset motions. In still another embodiment, the motion data may be extracted from images or videos stored on the computing device 200 or on the web.

[0039] On the other hand, the pose of the avatar may include a plurality of bones. In this case, the motion data may include at least one of the following information: indexes of each of the plurality of bones, rotation information in a three-dimensional space of each of the plurality of bones, position information in a virtual space of each of the plurality of bones, and a current tracking state of each of the plurality of bones. The computer device 200 may use such motion data to reflect the user's pose or a pose desired by the user in the pose of the avatar. A pose or a facial expression in one frame is static, but a change in a pose or a facial expression in multiple frames leads to a movement of the avatar. Furthermore, the motion data may include coefficient values ​​calculated for a plurality of points predefined for a human face based on a face blendshape technique. In this case, the computer device 200 may use such motion data to reflect the user's facial expression or a facial expression desired by the user in the facial expression of the avatar.

[0040] In step 360, the computer device 200 may receive motion data of the peripheral avatars through the communication session. As an example, the computer device 200 may receive motion data generated for the peripheral avatars in real time using a real-time transmission protocol. To generate a video screen for a conference, the motion data may be generated for each frame of the video. For example, if a video for a conference is to be provided at 10 frames per second (fps), the motion data may be received 10 times per second.

[0041] In step 370, the computer device 200 may represent the peripheral avatars in the virtual space based on the resources of the peripheral avatars and the motion data of the peripheral avatars. The resources of the avatars are used to represent the avatars in the virtual space, and the motion of the avatars is controlled using the motion data received in real time, so that an avatar reflecting the motion of a participant in real time can be represented in the virtual space.

[0042] On the other hand, with the appearance or movement of a new avatar, from the viewpoint of computer device 200, it also comes to receive motion data of a new peripheral avatar. In this case, computer device 200 requests resources of the new peripheral avatar from the server and receives the resources of the new peripheral avatar from the server, thereby making it possible to express the new peripheral avatar in the virtual space.

[0043] In step 380, the computer device 200 may display the virtual space in which the avatar and the surrounding avatars are represented on a screen. Thus, the conference participants can proceed with the conference while using the virtual space in which their own avatars are represented. At this time, the participants' poses, movements including facial expressions, or movements desired by the participants can be applied to the avatars in real time.

[0044] FIG. 4 is a diagram showing an example of a conference processing method of a server according to an embodiment of the present invention. The conference processing method according to this embodiment may be executed by a computer device 200 that realizes a server. Here, the server may be an entity that provides a conference service to a number of client devices in which a client program is installed. Also, the client program may correspond to an application for the above-mentioned service. In this case, the processor 220 of the computer device 200 may be realized to execute control instructions according to the operating system code and at least one computer program code included in the memory 210. Here, the processor 220 may control the computer device 200 so that the computer device 200 executes steps 410 to 470 included in the method of FIG. 4 according to the control instructions provided by the code recorded in the computer device 200.

[0045] In step 410, the computer device 200 may set up a communication session for a conference in which multiple users will participate. As an example, after a user opens a conference using a user interface provided by the conference service, the user may invite other users to the conference, and the computer device 200 may set up a communication session for the conference for participants including the user and the other invited users.

[0046] In step 420, the computer device 200 may mix voices received from a plurality of users through a communication session or another communication session set separately from the communication session, and provide the mixed voice to the plurality of users. In this case, when mixing voices to be provided to a particular user, the computer device 200 may generate a mixed voice to be provided to the particular user by mixing the remaining voices excluding the voice of the particular user.

[0047] In step 430, the computer device 200 may receive, from each of the terminals of the multiple users, an identifier of an avatar to be displayed in the virtual space for the conference and coordinate information of the avatar in the virtual space, which enables the computer device 200 to grasp the positions of the avatars of all the participants in the conference.

[0048] In step 440, the computer device 200 may select a peripheral avatar of the specific avatar based on the coordinate information. As described above, the computer device 200 may select an avatar existing within a preset distance from the specific avatar as a peripheral avatar for the specific avatar, and may transmit the resource of the selected peripheral avatar to the terminal of the user corresponding to the specific avatar. This is because, from the viewpoint of one user's terminal, it is not necessary to represent the avatars of all the participants of the conference, but only the avatars that may be displayed on the screen.

[0049] In step 450, the computer device 200 may transmit resources of the selected peripheral avatar to the user's terminal corresponding to the specific avatar. Here, the resources of the avatar may include data for displaying the corresponding avatar on a screen. Thus, the corresponding user's terminal can display the selected peripheral avatar in a virtual space using such resources.

[0050] In step 460, the computer device 200 may receive avatar motion data from each of the multiple user terminals via the communication session. As described above, each user terminal may obtain user motion data by analyzing images captured by a camera included in the corresponding terminal, and may transmit such user motion data to the server as corresponding avatar motion data. Such motion data may include data on at least one of the pose and facial expression of the corresponding user. In another embodiment, the motion data may include data on a motion selected by the corresponding user on the computer device 200 from a number of preset motions. In still another embodiment, the motion data may be extracted from images or videos stored on the computer device 200 or on the web.

[0051] On the other hand, the pose of the avatar may be configured to include a plurality of bones. In this case, the motion data may include at least one of the following information: indexes of each of the plurality of bones, rotation information of each of the plurality of bones in a three-dimensional space, position information of each of the plurality of bones in a virtual space, and a current tracking state of each of the plurality of bones. The motion data may also include coefficient values ​​calculated for a plurality of predefined points on a human face based on a face blend shape technique.

[0052] In step 470, the computer device 200 may transmit motion data of the selected peripheral avatar to a user terminal corresponding to a specific avatar through a communication session. In this case, each user terminal may reflect the pose and / or facial expression of the corresponding user, or the pose and / or facial expression desired by the corresponding user, in the peripheral avatar using the transmitted motion data. A pose or facial expression in one frame is static, but a change in pose in multiple frames leads to the movement of the avatar. To this end, in step 460, the computer device 200 may receive motion data generated from each of the multiple user terminals in real time using a real-time transmission protocol, and in step 470, may transmit the motion data of the selected peripheral avatar to the user terminal using the real-time transmission protocol. Such real-time transmission of motion data makes it possible to provide a conference service in which an avatar participates, in which the user's motion is reflected in real time, rather than a simple conference service in which an avatar participates.

[0053] On the other hand, as described above, when a new avatar appears or moves, from the viewpoint of the user's terminal, the user's terminal may receive the motion data of the new peripheral avatar. In this case, the user's terminal may request the resource of the new peripheral avatar from the server. For example, the computer device 200 may receive a resource request including an identifier of the new peripheral avatar from the user's terminal as a result of the user's terminal receiving the motion data of the new peripheral avatar. In this case, the computer device 200 may obtain the resource of the new peripheral avatar based on the identifier of the new peripheral avatar, and may transmit the obtained resource to the user's terminal. Therefore, the user's terminal can express the new peripheral avatar in the virtual space by receiving the resource of the new peripheral avatar from the server.

[0054] 5-8 are flow diagrams illustrating an example of an avatar rendering process in one embodiment of the present invention, showing User 1 (510), User 2 (520), Application Server (AS) 530, Voice Mixer (VMX) 540, Motion Mixer (MMX) 550, and Character Manager (CM) 560.

[0055] Here, each of User 1 (510) and User 2 (520) may be a terminal that is substantially a physical device used by the user to utilize the service, and such a terminal may be realized in the form of the computer device 200 described with reference to Fig. 2. For example, User 1 (510) may be realized in the form of the computer device 200 described with reference to Fig. 2, and may execute operations for a conference processing method by the processor 220 included in the computer device 200 under the control of an application installed and executed in the computer device 200 to receive the conference service. User 1 (510) and User 2 (520) who receive the conference service through such an application may be clients of the conference service.

[0056] Moreover, the AS 530, the VMX 540, the MMX 550, and the CM 560 may each be a software module implemented in a separate physical device or in one physical device. The physical device in which the AS 530, the VMX 540, the MMX 550, and / or the CM 560 is implemented may also be implemented in the form of the computer device 200 described with reference to Fig. 2. Such an AS 530, the VMX 540, the MMX 550, and the CM 560 may be at least a part of a server system for providing a conference service.

[0057] 5, the preparation process 570 may include an avatar information transmission process 571, a resource request process 572, and resource transfer processes 573 and 574. In the avatar information transmission process 571, the user 1 (510) may transmit to the AS 530 an avatar identifier and coordinate information of the avatar in the virtual space.

[0058] In a resource request step 572, the AS 530 may transmit avatar identifiers and coordinate information to the CM 560 to request resources for the neighboring avatars. The CM 560 may obtain coordinate information of all avatars represented in the virtual space by recording avatar identifiers and avatar coordinate information transmitted from each of the users (in this embodiment, user 1 (510) and user 2 (520)) participating in the conference. In this case, the CM 560 may select avatars located within a preset distance based on the avatar coordinate information of user 1 (510) as neighboring avatars, and may check the resources of the selected neighboring avatars.

[0059] In resource transmission steps 573 and 574, the CM 560 may transmit the resources of the peripheral avatars to the user 1 (510) via the AS 530. Thus, the user 1 (510) can express the peripheral avatars located around his / her avatar.

[0060] Although FIG. 5 shows an example in which a preparation process 570 is performed for user 1 (510), such a preparation process 570 may be performed for each participant of the conference.

[0061] 6, a voice communication process 600 may include voice transmission processes 610 and 620 and voice reception processes 630 and 640. Such a voice communication process 400 may be selectively utilized for participants to perform voice dialogue with each other. In other words, it may be omitted when voice dialogue between participants is not provided.

[0062] In voice transmission steps 610 and 620, user 1 (610) and user 2 (620) may transmit their respective voices to the VMX 540. Of course, voice transmission is premised on the assumption that the voices of the conference participants are recognized. As an example, if the voice of user 2 (620) is not recognized, the voice transmission step 620 from user 2 (620) to the VMX 540 may be omitted.

[0063] In the audio receiving steps 630 and 640, user 1 (610) and user 2 (620) may receive mixed audio from the VMX540. Here, mixed audio may mean audio in which the remaining audio, excluding the user's own audio, is mixed. For example, if users A, B, C, D, and E simultaneously transmit audio to the VMX540, the VMX540 may transmit to user A audio in which the audio of users B, C, D, and E, excluding user A, is mixed.

[0064] As described above, such a voice communication process 600 may be selectively utilized for participants to have a voice dialogue with each other. Also, a video communication process 700 in FIG. 7 and a new avatar processing process 800 in FIG. 8 described below may be executed in parallel with such a voice communication process 600.

[0065] Referring to FIG. 7, a video communication process 800 may include motion data sending processes 810, 820 and motion data receiving processes 830, 840.

[0066] In motion data transmission steps 810 and 820, user 1 (510) and user 2 (520) may transmit their motion data to the MMX 550. Such motion data may be obtained from images captured by the cameras of user 1 (510) and user 2 (520), respectively. Such motion data may include data on at least one of the pose and facial expression of the user. In another embodiment, the motion data may include data on a motion selected by the user from a number of preset motions. In still another embodiment, the motion data may be extracted from images or videos stored on the user's terminal or on the web.

[0067] In motion data receiving steps 830 and 840, user 1 (510) and user 2 (520) may receive motion data of peripheral avatars from MMX 550. For example, if the identifier of the avatar of user 1 (510) is CID1 and the identifier of the avatar of user 2 (520) is CID2, user 1 (510) may receive motion data of the peripheral avatar selected for CID1, and user 2 (520) may receive motion data of the peripheral avatar selected for CID2.

[0068] At this time, each of User 1 (510) and User 2 (520) can express not only their own avatar but also the surrounding avatars in the virtual space for the conference based on the resources and motion data of the surrounding avatars. In addition, it is possible to apply the motions (poses and / or facial expressions) of the conference participants to the corresponding avatars in real time using dynamic data. To this end, as described above, the process of transmitting and receiving the motion data may be performed using RTSP (Real-Time Streaming Protocol), and the motion data may be framed for RTSP.

[0069] Referring to FIG. 8, a new avatar processing process 800 may include a new avatar detection process 810, resource request processes 820, 830, and resource transmission processes 840, 850.

[0070] In the new avatar detection process 810, when the user 1 (510) receives the motion data of the surrounding avatars, the user 1 (510) may detect the appearance of a new surrounding avatar based on the avatar identifier held by the user 1 (510).

[0071] In resource request steps 820 and 830, the user 1 (510) may request resources for a new nearby avatar from the AS 530. Such a resource request may be transmitted from the AS 530 to the CM 560.

[0072] In resource transfer steps 840 and 850, the CM 560 may transfer the requested resources of the new surrounding avatar from the AS 530 to User 1 (510).

[0073] The user 1 (510) can express a new avatar as a nearby avatar by the new avatar processing process 800. Although an example in which the new avatar processing process 800 is executed for the user 1 (510) is described in Fig. 8, the new avatar processing process 800 may be executed for all users in the conference who detect a new nearby avatar.

[0074] Table 1 below shows an example of a data structure for expressing a pose as motion data, and Fig. 9 shows an example of an avatar bone structure in one embodiment of the present invention. From the perspective of one frame of a video in which an avatar is expressed, as long as the avatar's pose in that frame is expressed, the avatar's motion can be realized by the avatar's pose connected by linking such frames.

[0075] [Table 1]

[0076] In this manner, the avatar pose may be comprised of multiple bones, and the motion data may include at least one of the following information: an index for each of the multiple bones, rotation information in three-dimensional space for each of the multiple bones, position information in virtual space for each of the multiple bones, and a current tracking state for each of the multiple bones.

[0077] For example, when sending motion data at 10 fps (frames per second), motion data is sent 10 times per second, and each motion data may include a bone index, rotation information for each bone, position information for each bone, and information on the tracking state of each bone. In the case of an avatar composed of 11 bones as in the embodiment shown in Fig. 8, motion data sent at one time may include 11 bone indexes, 11 pieces of rotation information, 11 pieces of position information, and 11 pieces of tracking state.

[0078] Meanwhile, as described above, the motion data may further include data on the facial expression of the avatar in addition to the pose of the user. To this end, the motion data may include coefficient values ​​calculated for a plurality of points predefined for a human face based on a face blendshape technique. For example, 52 face points may be defined as the plurality of points, and the coefficient values ​​may be calculated to have values ​​from 0.0 to 1.0. For example, for a point called "eyes," a value of 0.0 may correspond to closed eyes, and a value of 0.1 may correspond to wide open eyes. The number of times that the motion data for such facial expressions is transmitted may also be determined according to the set fps.

[0079] FIG. 10 is a diagram showing an example of a conference held using avatars in an embodiment of the present invention. FIG. 10 shows an example of a conference screen 1000. It is not necessary for the avatars of all participants participating in the conference to be displayed on the conference screen 1000, and at least a part of the virtual space corresponding to the peripheral avatars 1020, 1030, and 1040 set based on the position of the avatar 1010 of the participant who will display the corresponding conference screen 1000 may be displayed on the conference screen 1000. At this time, the avatars 1010, 1020, 1030, and 1040 may be applied in real time with actions including the poses and expressions of the participants or actions desired by the participants based on the respective action data of the avatars 1010, 1020, 1030, and 1040 transmitted to the server in real time. In other words, it is possible to realize a conference in which the actions of the user are applied in real time, rather than simply displaying the avatars in the virtual space for the conference.

[0080] Thus, according to an embodiment of the present invention, avatars of participants that mimic the movements of the participants in the virtual space can be displayed in a virtual space shared by the participants, and the conference can be conducted using such a virtual space.

[0081] The above-described systems or devices may be realized by hardware components or a combination of hardware and software components. For example, the devices and components described in the embodiments may be realized using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or various devices capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications that run on the OS. The processing device may also respond to the execution of the software and access, record, manipulate, process, and generate data. For convenience of understanding, a single processing device may be described as being used, but one skilled in the art will understand that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, the processing device may include multiple processors or one processor and one controller. Other processing configurations, such as parallel processors, are also possible.

[0082] The software may include computer programs, codes, instructions, or a combination of one or more of these, and may configure or instruct the processing device to perform a desired operation, either individually or collectively. The software and / or data may be embodied in any type of machine, component, physical device, virtual device, computer storage medium, or device to be interpreted by or provide instructions or data to the processing device. The software may be distributed and stored or executed in a distributed manner on computer systems connected by a network. The software and data may be stored on one or more computer-readable storage media.

[0083] The method according to the embodiment may be realized in the form of program instructions executable by various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The medium may be one that continuously records a computer-executable program, or one that temporarily records the program for execution or download. The medium may be one that is a recording or storage means in the form of a single or multiple hardware devices, and is not limited to a medium that is directly connected to a certain computer system, but may be one that is distributed over a network. Examples of the medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and ROMs, RAMs, flash memories, and the like, which are configured to record program instructions. Other examples of the medium include recording media or storage media managed by application stores that distribute applications, or sites, servers, and the like that supply or distribute various other software. Examples of the program instructions include not only machine language codes such as those generated by compilers, but also high-level language codes that are executed by a computer using an interpreter or the like.

[0084] Although the embodiments have been described based on the limited embodiments and drawings as above, those skilled in the art will be able to make various modifications and variations from the above description. For example, the described techniques may be performed in an order different from that described, and / or the components of the described systems, structures, devices, circuits, etc. may be combined or combined in a manner different from that described, or may be counterbalanced or replaced by other components or equivalents, and still achieve suitable results.

[0085] Therefore, different embodiments are within the scope of the appended claims provided they are equivalent to the claims.

Claims

1. A computer program recorded on a computer-readable recording medium for causing the computer to execute a conference processing method in combination with a computer device, The conference processing method includes: establishing, via a server, a communications session for a conference involving multiple users; transmitting to a server an identifier of an avatar represented in a virtual space for the conference and coordinate information of the avatar in the virtual space; receiving resources of the nearby avatars selected based on the coordinate information from the server; analyzing an image captured by a camera included in the computing device to obtain motion data of a user to which the avatar corresponds; transmitting the user's motion data obtained by analyzing the image as the avatar's motion data to the server via the communication session; receiving motion data of the peripheral avatar from the server via the communication session; expressing the peripheral avatar in the virtual space based on the resource of the peripheral avatar and the motion data of the peripheral avatar; and sharing the voices of the plurality of users through the server via another communication session that is set separately from the communication session; Including, the avatar pose is configured to include a plurality of bones; the motion data includes at least one of information on an index of each of the plurality of bones, rotation information in three-dimensional space of each of the plurality of bones, position information in the virtual space of each of the plurality of bones, and a current tracking state of each of the plurality of bones as data on a pose of the corresponding avatar; a communication protocol of the communication session for communicating the motion data is RTSP (Real-Time Streaming Protocol); The communication protocol of the other communication session for communicating audio is RTP (Real-time Transport Protocol). Computer program.

2. The conference processing method includes: displaying the virtual space in which the avatar and the peripheral avatars are represented on a screen; 2. The computer program of claim 1, further comprising:

3. The step of receiving the operational data includes: receiving the motion data generated for the peripheral avatar in real time using a real-time transmission protocol; 2. A computer program according to claim 1, characterized in that

4. The conference processing method includes: When receiving motion data of a new neighboring avatar, requesting resources of the new neighboring avatar from the server; and receiving resources of the new neighboring avatar from the server; 2. The computer program of claim 1, further comprising:

5. The motion data includes coefficient values ​​calculated for a plurality of predefined points on a human face based on a facial blendshape technique.

2. A computer program according to claim 1, characterized in that

6. 1. A method of conference processing in a computer device including at least one processor, comprising: establishing, by said at least one processor, a communications session for a conference involving a plurality of users through a server; transmitting, by the at least one processor, to a server, an identifier of an avatar to be represented in a virtual space for the conference and coordinate information of the avatar in the virtual space; receiving, by the at least one processor, resources of the selected surrounding avatars from the server based on the coordinate information; analyzing, by the at least one processor, images captured by a camera included in the computing device to obtain motion data of a user to which the avatar corresponds; transmitting, by the at least one processor, motion data of the user obtained by analyzing the images as motion data of the avatar to the server via the communication session; receiving, by said at least one processor, motion data of said peripheral avatar from said server via said communication session; expressing the peripheral avatar in the virtual space based on the resource of the peripheral avatar and the motion data of the peripheral avatar by the at least one processor; sharing, by the at least one processor, the voices of the plurality of users through the server via another communication session that is established separately from the communication session; Including, the avatar pose is configured to include a plurality of bones; the motion data includes at least one of information on an index of each of the plurality of bones, rotation information in three-dimensional space of each of the plurality of bones, position information in the virtual space of each of the plurality of bones, and a current tracking state of each of the plurality of bones as data on a pose of the corresponding avatar; a communication protocol of the communication session for communicating the motion data is RTSP (Real-Time Streaming Protocol); The communication protocol of the other communication session for communicating audio is RTP (Real-time Transport Protocol). Conference handling methods.

7. 1. A method of conference processing in a computer device including at least one processor, comprising: establishing, by said at least one processor, a communications session for a conference involving a plurality of users; receiving, by the at least one processor, from each of the terminals of the plurality of users, an identifier of an avatar to be represented in a virtual space for the conference and coordinate information of the avatar in the virtual space; selecting, by the at least one processor, neighboring avatars of a particular avatar based on the coordinate information; transmitting, by the at least one processor, resources of the selected peripheral avatar to a terminal of a user corresponding to the particular avatar; receiving, by the at least one processor, avatar motion data from each of the plurality of users' terminals via the communication session; transmitting, by the at least one processor, motion data of the selected peripheral avatar to a terminal of a user corresponding to the particular avatar via the communication session; and mixing, by the at least one processor, audio received from the plurality of users via another communication session set separately from the communication session, and providing the audio to the plurality of users; Including, the avatar pose is configured to include a plurality of bones; the motion data is obtained by analyzing images captured by cameras included in each of the multiple users' terminals, and includes at least one of information on an index of each of the multiple bones, rotation information of each of the multiple bones in three-dimensional space, position information of each of the multiple bones in the virtual space, and a current tracking state of each of the multiple bones as data on a pose of the corresponding avatar; a communication protocol of the communication session for communicating the motion data is RTSP (Real-Time Streaming Protocol); The communication protocol of the other communication session for communicating audio is RTP (Real-time Transport Protocol). Conference handling methods.

8. The step of receiving the avatar motion data via the communication session comprises: Utilizing a real-time transmission protocol, receiving in real time the motion data generated from each of the terminals of the plurality of users; The step of transmitting the motion data of the selected peripheral avatar includes: Transmitting the motion data of the selected peripheral avatar to the user's terminal using the real-time transmission protocol.

8. The conference processing method according to claim 7, further comprising:

9. receiving, by the at least one processor, a resource request including an identifier of the new peripheral avatar from the user's terminal by the user's terminal receiving motion data of the new peripheral avatar; obtaining, by the at least one processor, resources of the new peripheral avatar based on an identifier of the new peripheral avatar; and transmitting, by the at least one processor, the acquired resources to a terminal of the user. The conference processing method of claim 7 further comprising:

10. Representing the selected peripheral avatar in the virtual space based on the resources of the selected peripheral avatar and the motion data of the selected peripheral avatar in the user's terminal, and displaying the virtual space in which the user's avatar and the selected peripheral avatar are represented on a screen of the terminal.

8. The conference processing method according to claim 7, further comprising:

Images