Live communication system using characters
The live communication system enhances emotional bonding between users and characters by enabling synchronized physical interactions, addressing the lack of reality in virtual characters through motion detection and hand coordination.
Patent Information
- Application Number
- JP2023100943
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2023-06-20
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2039-08-01
AI Technical Summary
Characters in computer games lack the sense of reality and emotional connection that humans possess, limiting the depth of communication between users and these virtual entities.
A live communication system that enables one-on-one interaction between a distributor acting as a spokesperson for a character and a viewer, using motion detection and hand coordination to simulate handshakes and other physical interactions, enhancing the emotional bond through synchronized movements and gestures.
Facilitates richer and more intimate communication experiences by allowing viewers to physically interact with characters, thereby strengthening their emotional attachment and sense of connection.
Smart Images

Figure 0007584576000001 
Figure 0007584576000002 
Figure 0007584576000003
Abstract
Description
[Technical field]
[0001] The present invention relates to visual communication technology over a communication network. [Background technology]
[0002] With the spread of smartphones, communication via SNS (Social Networking Service) has become common. SNS allows you to easily share a variety of content, including not only text but also photos and videos. SNS is a tool that makes you feel a strong bond (connection) between people.
[0003] On the other hand, in the field of computer games, there are many games in which the user plays while having a conversation with a character (computer). Users often have strong feelings for characters that do not exist in reality. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] JP 2012-231281 A [Patent Document 2] Patent application No. 2019-002271 Summary of the Invention [Problem to be solved by the invention]
[0005] Characters are virtual entities created by game designers and do not have the sense of reality that humans have. On the other hand, characters have a unique charm that stems from their sense of non-existence. The inventors thought that by incorporating both the unique charm of characters and the "realistic connection" that is unique to humans, they could propose a new method of communication.
[0006] Specifically, we consider a conversation between a streamer (human) and a viewer (human). The streamer acts as a spokesperson for the character, just like a voice actor in a television anime, and converses with the viewer through the character. If we consider this type of communication, we believe that if the viewer can have a relationship with the character that goes beyond words, the viewer's attachment to the character will be further strengthened.
[0007] The present invention was completed based on the above recognition, and its main purpose is to provide technology for deepening non-verbal communication between characters and viewers, in particular, in communication via characters. [Means for solving the problem]
[0008] A live communication system according to an embodiment of the present invention includes a distribution device operated by a distributor and a viewing device operated by a viewer. The broadcasting device includes an imaging unit that images the broadcaster, a motion information generation unit that generates motion information indicating characteristics of the broadcaster's motion from the captured image of the broadcaster, a first hand detection unit that detects first hand coordinates corresponding to the position of the broadcaster's hand, a first message acquisition unit that accepts a first message from the broadcaster, and a transmission unit that transmits the motion information, the first hand coordinates, and the first message. The viewing device includes a receiving unit that receives the movement information, the first hand coordinate, and the first message, a second hand detection unit that detects the second hand coordinate corresponding to the position of the viewer's hand, a viewing character display unit that displays a moving image of a character on the screen of the viewing device and moves the character according to the movement information, a first message output unit that outputs a first message in accordance with the movement of the character, and a hand display unit that displays a first hand which is an image of the distributor's hand and a second hand which is an image of the viewer's hand on the screen of the viewing device based on the first hand coordinate and the second hand coordinate. The hand display unit displays a handshake between the first hand and the second hand when the distance between the first hand coordinates and the second hand coordinates is within a first threshold value. Effect of the Invention
[0009] According to the present invention, rich communication via characters can be easily realized. [Brief description of the drawings]
[0010] [Figure 1] FIG. 2 is a hardware configuration diagram of a live communication system. [Diagram 2] FIG. 1 is a schematic diagram for explaining a conversation service provided by a live communication system. [Diagram 3] FIG. 2 is an external view of a conversation booth. [Figure 4] FIG. 2 is a functional block diagram of a distribution device and a server. [Diagram 5] FIG. 2 is a functional block diagram of the viewing device. [Figure 6] 11 is a screen diagram of a viewer conversation screen displayed on the viewing device. FIG. [Figure 7] FIG. 13 is a screen diagram of a distributor conversation screen displayed on the distribution device. [Figure 8] FIG. 11 is a sequence diagram of a process for displaying a character on both the distribution device and the viewing device. [Figure 9] 13 is a diagram showing the screen of the viewing device before the viewer shakes hands with the character. FIG. [Figure 10] 13 is a diagram showing a screen of a viewing device when a viewer shakes hands with a character. [Figure 11] FIG. 1 is a schematic diagram for explaining a handshake movement. [Figure 12] 13 is a diagram showing the screen of the viewing device at the start of a conversation with a character. [Figure 13] 13 is a diagram showing the screen of the viewing device when a viewer touches a character. [Figure 14] FIG. 11 is a sequence diagram showing a process when a good is presented to a character. [Figure 15] 13 is a diagram showing a screen of the viewing device when a character is provided with goods. FIG. [Figure 16] FIG. 1 is an external view of a digital signage device. [Figure 17] FIG. 2 is a functional block diagram of the digital signage device. [Figure 18] FIG. 13 is a diagram showing the screen of the viewing device when playing a hand game with a character. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0011] FIG. 1 is a diagram showing the hardware configuration of a live communication system 100. As shown in FIG. In the live communication system 100, a plurality of distribution devices 104a, 104b, . . . 104m (hereinafter collectively referred to as "distribution device 104" when they are referred to collectively or when there is no particular distinction) and a plurality of viewing devices 108a, 108b, . . . 108n (hereinafter collectively referred to as "viewing device 108" when they are referred to collectively or when there is no particular distinction) are connected to a server 102 via the Internet 106. A smartphone is assumed as the distribution device 104, but it may also be a general-purpose computer such as a laptop PC. A large touch pad is assumed as the viewing device 108, but it may also be another information device such as a smartphone.
[0012] A user who uses the distribution device 104 is called a "distributor." A user who uses the viewing device 108 is called a "viewer." The distributor is the provider of the service, and the viewer is the recipient of the service.
[0013] A digital signage device 300 is connected to the server 102. The digital signage device 300 broadcasts live or records and plays back the conversation between the distributor and the viewer. Details will be described later with reference to Fig. 16 and subsequent figures. The digital signage device 300 is not a required component for the live communication system 100.
[0014] FIG. 2 is a schematic diagram for explaining a conversation service provided by the live communication system 100. As shown in FIG. In the live communication system 100 disclosed in Patent Document 2 by the present applicant, a large number of viewing devices 108 are simultaneously connected to one distribution device 104, and the system has been described on the premise of one-to-many communication. In the live communication system 100 of the present embodiment, a distributor 110 and a viewer 112 converse with each other through one-to-one communication.
[0015] A character is assigned in advance to the distributor 110. A distribution character 114a assigned to the distributor 110 is displayed on the distribution device 104. A viewing character 114b is also displayed on the viewing device 108. The distribution character 114a and the viewing character 114b are exactly the same character. Hereinafter, when there is no particular need to distinguish between the distribution character 114a and the viewing character 114b, they will simply be referred to as "character 114."
[0016] When the distributor 110 speaks in front of the distribution device 104, the voice data is transmitted to the viewing device 108. The distribution device 104 detects the movement of the distributor 110's mouth, and also moves the mouth of the distribution character 114a. In conjunction with this, the viewing device 108 also moves the mouth of the viewing character 114b. To the viewer 112, it appears as if the viewing character 114b is speaking of its own volition, rather than the distributor. In a sense, the distributor 110 serves as a "voice actor" and speaks for the character 114. Hereinafter, the voice message input by the distributor 110 is referred to as a "first message."
[0017] When the viewer 112 speaks in front of the viewing device 108, the voice data is transmitted to the distribution device 104 together with a captured image (moving image) of the viewer 112 himself / herself. Hereinafter, the voice message input by the viewer 112 is referred to as a "second message." In Patent Document 2, it was described that the viewer 112 inputs the second message by text, but in this embodiment, the distributor 110 and the viewer 112 converse with each other by voice. The distributor 110 can see the appearance of the viewer 112 (described later in relation to FIG. 7), but the viewer 112 only sees the viewed character 114b played by the distributor 110.
[0018] The individuality of the character 114 is formed by the basic settings of the character 114 (such as the appearance of the character 114 and the variety of facial expressions) and the voice quality and conversational ability of the distributor 110.
[0019] The viewer 112 usually enjoys one-to-many conversations with the characters 114 on a smartphone or the like, and develops feelings for the characters 114 (see Patent Document 2). In this embodiment, a "handshake event (an opportunity to meet one-to-one)" with the characters 114 that the viewer 112 has developed feelings for is assumed. The viewer 112 visits a venue (hereinafter referred to as an "event venue") set up by the operator of the live communication system 100. At the event venue, the viewer 112 can enjoy one-to-one voice conversations with the characters 114. However, the "time limit" for the conversation is set to one minute. In the conversation, the viewer can enjoy not only the conversation but also a handshake by using a method described below.
[0020] FIG. 3 is an external view of conversation booth 302. At the event venue, a conversation booth 302 is prepared for each character 114. The viewers 112 line up at the conversation booth 302 corresponding to their favorite character 114. The viewers 112 wear nameplates. When it is their turn, the viewers 112 enter the conversation booth 302. The conversation booth 302 is a semi-closed space separated by a partition, and is configured to make it easy for the viewers 112 to concentrate on the conversation with the character 114. A viewing device 108 is installed in front of the conversation booth 302, and a viewing character 114b is displayed on the viewing device 108. In the conversation booth 302, the viewers 112 can monopolize the conversation time with the idol-like character 114 who usually has a conversation with a large group of other viewers 112.
[0021] A camera 304 for capturing an image of the viewer 112 is attached to the top of the viewing device 108. The captured image of the viewer 112 is transferred to the distribution device 104. A present box 306 is provided at the bottom of the viewing device 108. The viewer 112 can insert presents such as hats and bouquets of flowers (hereinafter referred to as "goods") into an insertion slot 308 of the present box 306. A tag is attached to the goods (physical object), and a detection device 310 inside the insertion slot 308 detects the ID of the goods (hereinafter referred to as "goods ID"). The insertion of goods will be described later with reference to Figs. 14 and 15.
[0022] FIG. 4 is a functional block diagram of the distribution device 104 and the server 102. Each component of the distribution device 104 and the server 102 is realized by hardware including computing units such as a CPU (Central Processing Unit) and various coprocessors, storage devices such as memory and storage, and wired or wireless communication lines connecting them, and software stored in the storage devices and supplying processing instructions to the computing units. The computer programs may be composed of device drivers, an operating system, various application programs located at higher layers than those, and libraries that provide common functions to these programs. Each block described below represents a functional block, not a hardware-based configuration. The same applies to the audiovisual device 108 and the digital signage device 300.
[0023] (Distribution device 104) The distribution device 104 includes a user interface processing unit 116, an imaging unit 118, a data processing unit 120, a communication unit 122, and a data storage unit 124. The communication unit 122 is responsible for communication processing. The imaging unit 118 images the distributor 110. The user interface processing unit 116 accepts operations from the distributor 110 and is responsible for processing related to the user interface, such as image display and audio output. The data storage unit 124 stores various data including image data of the character 114. The data processing unit 120 executes various processes based on data acquired by the communication unit 122, the user interface processing unit 116, and the imaging unit 118 and data stored in the data storage unit 124. The data processing unit 120 also functions as an interface for the user interface processing unit 116, the imaging unit 118, the communication unit 122, and the data storage unit 124.
[0024] The communication unit 122 includes a transmitting unit 148 and a receiving unit 150 .
[0025] The user interface processing unit 116 includes an input unit 126 that accepts input from the distributor 110 and an output unit 128 that outputs various information to the distributor 110 . The input unit 126 includes a motion detection unit 130 and a first message acquisition unit 132. The motion detection unit 130 detects facial expressions and body movements of the distributor 110 via the imaging unit 118. The first message acquisition unit 132 acquires a first message (voice message) from the speech of the distributor 110. The action detection unit 130 includes a first hand detection unit 134. The first hand detection unit 134 detects the movement of the distributor 110's hand.
[0026] The output section 128 includes a delivery character display section 138 , a second message output section 140 , a hand display section 142 , a decoration section 312 , an object display section 314 and a live image output section 332 . The delivery character display unit 138 displays the delivery character 114a. The second message output unit 140 outputs a second message (audio message) by the viewer 112. The hand display unit 142 displays an image (described later) imitating the hands of the character 114 (the distributor 110) and the viewer 112 on the screen. The decoration unit 312 displays various decoration images as visual effects on a given occasion, such as when the character 114 and the viewer 112 shake hands. The handshakes and decoration images will be described later in relation to Figs. 9, 10, 11, 13, etc. The object display unit 314 displays an image (hereinafter referred to as a "goods image") corresponding to a good given as a present by the viewer 112 on the screen. The display of the goods image (virtual object) on the screen will be described later in relation to Fig. 15. The live image output unit 332 displays a moving image of the captured image of the viewer 112 (described later in relation to Fig. 7).
[0027] The data processing unit 120 includes a motion information generating unit 144 . The motion information generating unit 144 generates "motion information" from the detection result of the motion detecting unit 130. The motion information is numerical data indicating the motion state of the distributor 110, such as a feature vector indicating the facial expression of the distributor 110 (e.g., a numerical value indicating the angle of the eyebrows, the opening degree of the eyes and mouth, and a numerical value indicating the relative positional relationship (front, back, left, right) between the distributor 110 and the distribution device 104).
[0028] (Server 102) The server 102 includes a communication unit 152 , a live control unit 154 , and a data storage unit 156 . The communication unit 152 is responsible for communication processing. The data storage unit 156 stores various data. The live control unit 154 mediates and manages communication between the distribution device 104 and the viewing device 108 based on the data acquired by the communication unit 152 and the data stored in the data storage unit 156. The live control unit 154 also functions as an interface between the communication unit 152 and the data storage unit 156. The data storage unit 156 includes a character storage unit 158 that stores data on a plurality of characters 114. The data storage unit 156 also registers a goods ID and a goods image in association with each other.
[0029] FIG. 5 is a functional block diagram of the viewing device 108. The viewing device 108 includes a user interface processing unit 162 , a data processing unit 166 , a communication unit 164 , an imaging unit 316 and a data storage unit 168 . The communication unit 164 is responsible for communication processing. The user interface processing unit 162 accepts operations from the viewer 112 and is responsible for processing related to the user interface, such as image display and audio output. The imaging unit 316 captures images of the viewer 112 via the camera 304. The data storage unit 168 stores various data including image data of the character 114. Both the distribution device 104 and the viewing device 108 store data of the character 114. The data processing unit 166 executes various processes based on data acquired by the communication unit 164 and the user interface processing unit 162 and data stored in the data storage unit 168. The data processing unit 166 also functions as an interface for the user interface processing unit 162, the communication unit 164, the imaging unit 316, and the data storage unit 168.
[0030] The communication unit 122 includes a transmitting unit 186 and a receiving unit 190 .
[0031] The user interface processing unit 162 includes an input unit 170 that receives input from the viewer 112 and an output unit 172 that outputs various information to the viewer 112 . The input unit 170 includes a second message acquisition unit 174, a second hand detection unit 318, an object insertion unit 320, and an object identification unit 322. The second message acquisition unit 174 acquires a second message (voice message) from the speech of the viewer 112. The second hand detection unit 318 detects the hand movement of the viewer 112. The object insertion unit 320 detects the insertion of a good (physical object) into the insertion slot 308. The object identification unit 322 identifies the type of good by reading the good ID of the good inserted into the insertion slot 308. When the good ID is identified, the transmission unit 186 of the communication unit 164 transmits the good ID to the server 102.
[0032] The output section 172 includes a viewing character display section 180 , a first message output section 182 , a hand display section 324 , a decoration section 326 , an object display section 328 and a curtain display section 330 . The viewing character display unit 180 displays the viewing character 114b on the viewing device 108. The first message output unit 182 outputs the first message by the distributor 110 as audio from the viewing device 108. The hand display unit 324 displays images of the hands of the character 114 (distributor 110) and the viewer 112 on the screen. The decoration unit 326 displays various decoration images. The object display unit 328 displays images of goods corresponding to goods given as a present by the viewer 112 on the screen. The curtain display unit 330 displays images imitating curtains at the start and end of a conversation. Curtains will be described later in relation to FIG. 12.
[0033] The data processing unit 166 includes a conversation management unit 290. The conversation management unit 290 manages the conversation time between the viewer 112 and the character 114. The conversation management unit 290 ends the conversation between the viewer 112 and the character 114 when a predetermined time limit, for example, one minute, has elapsed.
[0034] FIG. 6 is a diagram showing a viewer conversation screen 196 displayed on the viewing device 108. The viewing character display unit 180 displays the viewing character 114b selected as the conversation partner as a moving image. The viewer 112 enjoys a conversation with the viewing character 114b on the viewing device 108. The viewer 112 cannot see the distributor 110. The viewer 112 can enjoy the sensation of having a conversation with the viewing character 114b itself, not with the distributor 110. The first message output unit 182 of the viewing device 108 outputs a voice (first message) from the distributor 110 who is in charge of the viewing character 114b. The viewer 112 speaks to the viewing character 114b by voice (second message). The live control unit 154 of the server 102 relays the first message and the second message.
[0035] FIG. 7 is a diagram showing the distributor conversation screen 206 displayed on the distribution device 104. The distributor conversation screen 206 shown in FIG. 7 is displayed on the distribution device 104 of the distributor 110 in charge of the distribution character 114a (corresponding to the viewing character 114b). The distributor conversation screen 206 is divided into a viewer display area 336 and a character display area 334. The distribution character display unit 138 displays the distribution character 114a as a moving image in the character display area 334. The viewing character 114b and the distribution character 114a are the same character and move in the same way. The distributor 110 talks with the viewer 112 while checking the movement of the distribution character 114a, which is his / her alter ego, on the distribution device 104. The distributor 110 inputs a first message by voice to the distribution device 104. The transmission unit 148 transmits the first message to the server 102, and the live control unit 154 of the server 102 distributes the first message to the viewing device 108.
[0036] The second message output unit 140 outputs the voice (second message) from the viewer 112. The captured image of the viewer 112 is transmitted from the viewing device 108 to the distribution device 104 via the server 102. The live image output unit 332 displays the captured image (moving image) of the viewer 112 in the viewer display area 336. The distributor 110 can talk with the viewer 112 while watching the appearance (attitude and facial expression) of the viewer 112.
[0037] The camera 208 (imaging unit 118) of the broadcasting device 104 captures an image of the broadcaster 110. The motion detection unit 130 detects the facial expression and motion of the broadcaster 110 from the captured image, and the motion information generation unit 144 generates motion information. The broadcast character display unit 138 determines the motion of the broadcast character 114a based on the motion information. The viewing character 114b is also displayed as a moving image on the viewing device 108 based on the same motion information.
[0038] As described above, the viewer 112 converses with the viewing character 114b by voice. The distributor 110 converses with the viewer 112 as the viewing character 114b. The viewer 112 cannot see the distributor 110, but the distributor 110 converses with the viewer 112 while seeing both the viewer 112 and the viewing character 114a.
[0039] FIG. 8 is a sequence diagram of a process for displaying character 114 on both distribution device 104 and viewing device 108. The motion detection unit 130 of the broadcasting device 104 detects the motion of the broadcaster 110 (S10). The motion information generation unit 144 generates motion information (S12). The motion information is numerical data indicating the facial expression of the broadcaster 110, the angle of the body, the distance of the broadcaster 110 from the imaging unit 118, and the like (S12).
[0040] The transmission unit 148 of the distribution device 104 transmits the action information to the viewing device 108 via the server 102 (S14). The distribution character display unit 138 generates a moving image of the distribution character 114a based on the action information (S16) and causes the distribution device 104 to display the moving image of the distribution character 114a (S18).
[0041] Meanwhile, the viewing character display unit 180 of the viewing device 108 also generates a moving image of the viewing character 114b based on the received action information (S20), and causes the viewing device 108 to display the moving image of the viewing character 114b (S22).
[0042] The delivered character 114a and the viewed character 114b are the same character and their movements are determined based on the same motion information, so that the delivered character 114a and the viewed character 114b have the same appearance and perform the same movements on the delivery device 104 and the viewing device 108, respectively.
[0043] The distribution device 104 does not transmit the moving image (character image) of the character 114 itself to the viewing device 108, but transmits only the motion information that defines the motion of the character 114. Since the amount of data transmitted is smaller than when transmitting the moving image itself, it is easier to quickly link the motions of the distributed character 114a and the viewed character 114b. In other words, since the difference between the time until the motion information is reflected as the motion of the distributed character 114a and the time until the same motion information is reflected as the motion of the viewed character 114b is small, the viewer 112 can enjoy a pleasant conversation with the character 114 (distributor 110).
[0044] The first message (voice message) and the motion information may be transmitted from the distribution device 104 to the viewing device 108 in an integrated manner. In this case, the motion of the viewing character 114b and the voice message are less likely to be out of sync. In addition, transmitting motion information for determining a moving image, rather than the moving image itself, is also effective in reducing the communication load on the Internet 106.
[0045] When the distributor 110 tilts his / her body left or right, the motion detection unit 130 detects the tilt of the body. The motion information generation unit 144 includes the tilt angle and tilt direction in the motion information. The viewing character display unit 180 tilts the viewing character 114b according to the motion information. The delivery character display unit 138 also tilts the delivery character 114a according to the same motion information. In this way, when the distributor 110 tilts his / her body left or right, the characters 114 on both the delivery side and the viewing side tilt their bodies to match the distributor 110. When the distributor 110 tilts his / her body forward or backward, the characters 114 move their bodies forward or backward in the same manner. The motion detection unit 130 may recognize the forward or backward tilt of the distributor 110 from the size of the distributor 110 reflected in the captured image.
[0046] It is generally said that human psychology is reflected in the inclination of the body. For example, it is said that when a person is interested in a conversation, they tend to lean forward (as if they are about to listen), and when they are upset, their body sways from side to side. By reflecting the body movements of the distributor 110 in the character 114, the viewer 112 can sense the psychology of the character 114 (distributor 110).
[0047] When the distributor 110 changes his / her facial expression, the motion detection unit 130 detects the facial expression of the distributor 110, specifically, the size of the eyes, the shape of the mouth, the angle of the eyebrows, the depth and direction of wrinkles between the eyebrows, etc. The motion information generation unit 144 generates a feature vector that defines the facial expression based on these detection values. The feature vector becomes part of the motion information. When the distributor 110 closes one eye (winks), the viewing character display unit 180 makes the viewing character 114b wink. The same is true for the delivery character 114a.
[0048] In addition, by numerically expressing various facial expressions such as sticking out the tongue or turning up the corners of the mouth using feature vectors (motion information), the facial expressions of the distributor 110 can also be reflected in the character 114. When the distributor 110 smiles, the character 114 also smiles. When the distributor 110 purses his lips, the character 114 also purses his lips. By linking the facial expressions of the distributor 110 and the character 114, the viewer 112 can feel the presence of the character 114 even closer to him.
[0049] FIG. 9 is a diagram showing the screen of the viewing device 108 before the viewer 112 shakes hands with the character 114. When the viewer 112 touches the screen of the viewing device 108 (hereinafter, simply referred to as the "viewing screen"), the second hand detection unit 318 of the viewing device 108 detects the coordinates of the touch point (hereinafter, simply referred to as the "second hand coordinates 346"). When the distributor 110 touches the screen of the distribution device 104 (hereinafter, simply referred to as the "distribution screen"), the first hand detection unit 134 of the distribution device 104 detects the coordinates of the touch point (hereinafter, simply referred to as the "first hand coordinates 340"). The viewing device 108 transmits the second hand coordinates 346 to the distribution device 104, and the distribution device 104 transmits the first hand coordinates 340 to the viewing device 108. It is assumed that the same coordinate system is set for the viewing screen and the distribution screen. For example, when the first hand coordinates 340 on the distribution screen are (x1, y1), the first hand coordinates 340 on the viewing screen are also (x1, y1). The same applies to the second hand coordinates 346.
[0050] The hand display unit 324 of the viewing device 108 displays a first hand 342 at first hand coordinates 340. The first hand 342 is an image that imitates the hand of the viewed character 114b (distributor 110). In FIG. 9, the first hand 342 is displayed separately from the image of the viewed character 114b, but the hand of the viewed character 114b may be moved to the position of the first hand coordinates 340. The hand display unit 324 also displays a second hand 344 at second hand coordinates 346, which is the point touched by the viewer 112. The second hand 344 is an image that imitates the hand of the viewer 112. The hand display unit 142 of the broadcasting device 104 also displays a first hand 342 and a second hand 344 on the broadcasting screen.
[0051] When both distributor 110 and viewer 112 are touching the distribution screen and the viewing screen, respectively, they are in a "handshake possible state." On the other hand, when either one of them removes their hand from the screen, they are in a "handshake not possible state." When in the handshake possible state, hand display unit 142 periodically calculates the distance between first hand coordinate 340 and second hand coordinate 346 (hereinafter referred to as "hand distance d"). The same is true for hand display unit 142 of distribution device 104.
[0052] FIG. 10 is a diagram showing the screen of the viewing device 108 when the viewer 112 shakes hands with the character 114. When a handshake is possible and the hand distance d is within the first threshold, the hand display unit 324 displays a handshake image 338 of the second hand 344 and the first hand 342 shaking hands. That is, the distributor 110 and the viewer 112 bring their fingers (touch points) close to each other while still in a touching state, thereby establishing a "handshake." When the handshake is established, the decoration unit 326 displays a star-shaped decoration image 360 around the first hand 342 and the second hand 344. The decoration image 360 creates the effect of the handshake being achieved.
[0053] In this way, by bringing the distributor 110 and the viewer 112 closer to each other in their touch points, a handshake that transcends the "dimensions" of the viewed character 114b (two-dimensional) and the viewer 112 (three-dimensional) becomes possible. By giving the viewer 112 the sensation of not only talking but also holding the hand of the character 114, or of the character 114 holding his / her hand, the encounter between the viewer 112 and the character 114 can be made even more exciting. A handshake image 338 is also displayed on distribution device 104 by a similar processing method.
[0054] FIG. 11 is a schematic diagram for explaining the handshake movement. First hand coordinate 340 at time t1 is expressed as "first hand coordinate 340(t1)". At time t1, a handshake between first hand coordinate 340(t1) and second hand coordinate 346(t1) is established. Also, a hand distance d(t1) between first hand coordinate 340(t1) and second hand coordinate 346(t1) is assumed to be within a second threshold. Even after the handshake is established, the handshake continues as long as the hand distance d is within the second threshold.
[0055] Hand display unit 142 displays handshake image 338 (first hand 342 and second hand 344) at handshake point 348 (t1), which is the midpoint between first hand coordinate 340 (t1) and second hand coordinate 346 (t1). If first hand coordinate 340 and second hand coordinate 346 are close enough, a handshake between first hand 342 and second hand 344 is established even if second hand coordinate 346 and first hand coordinate 340 do not completely match.
[0056] Next, it is assumed that the first hand coordinate 340 and the second hand coordinate 346 are displaced at time t2. It is assumed that the hand distance d(t2) at time t2 is also within the second threshold. The handshake is maintained, and the handshake point 348 is displaced. The handshake image 338 moves from the handshake point 348(t1) to the handshake point 348(t2), so that the "movement" of the handshake is expressed. When the handshake image 338 moves a predetermined distance or more, or when the handshake image 338 moves at a predetermined speed or more, the decoration unit 326 adds the decoration image 360. According to this control method, the character 114 (distributor 110) and the viewer 112 can continue the handshake by moving the first hand 342 and the second hand 344 together in unison, in other words, by moving their hands in a timely manner so that the hand distance d between the first hand coordinate 340 and the second hand coordinate 346 does not become larger than the second threshold. Also, by making more decorative images 360 appear as the handshake continues, the handshake scene can be made more exciting.
[0057] The first threshold and the second threshold may be the same, or the second threshold may be set to be greater than the first threshold as described above. In this case, it is possible to realize a sense of operation in which, unless the first hand coordinate 340 and the second hand coordinate 346 are brought close enough to each other, it is not possible to shake hands, but once handshakes are made, it is difficult to undo them. With such settings, it becomes easier for the viewer 112 to continue shaking hands with the character 114.
[0058] Since the same display is also made on the distribution device 104, the distributor 110 also shares the feeling of shaking hands with the viewer 112. In particular, since the distributor 110 can visually check the appearance and expression of the viewer 112 in the viewer display area 336, the distributor 110 can sense the fun and joy that the viewer 112 feels when shaking hands with the character 114.
[0059] FIG. 12 is a diagram showing the screen of the viewing device 108 at the start of a conversation with the character 114. When viewer 112 enters conversation booth 302, curtain display unit 330 of viewing device 108 displays curtain 350 on the viewing screen. Viewing character 114b is initially hidden by curtain 350. When the viewer calls out to character 114, when a human sensor detects entry into conversation booth 302, or when distributor 110 inputs an opening command, curtain display unit 330 displays a video of curtain 350 opening. Character 114 is displayed behind curtain 350. The curtain 350 opens to reveal the desired character 114, creating the illusion of viewer 112 meeting character 114.
[0060] The curtain display unit 330 executes the performance of opening the curtain 350 for a predetermined opening time. In this embodiment, the opening time is three seconds. When the curtain 350 opens or closes, the output unit 172 of the audiovisual device 108 may play predetermined opening music.
[0061] After the opening, the conversation management unit 290 measures the conversation time. When the time limit (e.g., one minute) has elapsed, the conversation management unit 290 instructs the curtain display unit 330 to close the curtain 350. The curtain display unit 330 gradually blocks the character 114 by displaying a moving image of the curtain 350 closing. The curtain display unit 330 executes the performance of closing the curtain 350 over a predetermined closing time. In this embodiment, the closing time is set to 10 seconds.
[0062] Making the closing time longer than the opening time can make the viewer 112 feel a sense of regret for parting. By slowly closing the curtain and gradually hiding the character 114, a time of regret for parting with the character 114 can be created, and the viewer 112 can be made to feel a desire to see the character 114 again. When the curtain 350 closes completely, the character 114 is hidden and the conversation time ends. When the conversation time ends, the broadcaster 110 leaves the viewing device 108, and the next broadcaster 110 enters the conversation booth 302. Note that distribution device 104 may also be provided with a curtain display unit, and curtain 350 may be displayed on the distribution screen using a similar control method.
[0063] FIG. 13 is a diagram showing the screen of the viewing device 108 when the viewer 112 touches the character 114. The viewer 112 can touch the viewed character 114b, not limited to shaking hands. The viewer 112 can "pat" the viewed character 114b with the second hand 344 (the hand of the viewer 112) by touching the head of the viewed character 114b and rubbing the head. More specifically, the second hand detection unit 318 detects the second hand coordinates 346 at the time of touching, and determines whether the detection point of the second hand coordinates 346 is in the head area of the viewed character 114b. When the head area is touched and the second hand coordinates 346 makes a reciprocating motion, the second hand detection unit 318 determines that a "pat" operation is being performed.
[0064] When the viewer 112 strokes the viewed character 114b, the decoration unit 326 displays a heart-shaped decoration image 360 near the head. The more the viewer 112 strokes the viewed character 114b, the more decoration images 360 the decoration unit 326 displays. The viewer 112 can express affection for the character 114 by stroking the viewed character 114b with the second hand 344. When the viewer 112 strokes the head of the viewed character 114b, the viewed character display unit 180 changes the movement and facial expression of the viewed character 114b. For example, the reaction of the character 114 to the stroking operation is expressed by movements and facial expressions such as closing the eyes, smiling, tickling, and shrugging the shoulders.
[0065] In addition, the viewer 112 may touch and swipe the cheek of the viewed character 114b to perform an action of pulling the cheek of the viewed character 114b. In addition, an area that should not be touched on the viewed character 114b (hereinafter referred to as an "inappropriate area") may be set, such as the eyes or the chest. When the viewer 112 touches the inappropriate area with the second hand 344, the conversation management unit 290 may shorten the time limit of the conversation, or may terminate the conversation and close it. The conversation management unit 290 may shorten the conversation time every time a touch on an inappropriate area is detected. The decoration unit 326 may display a decoration image 360 indicating a warning when a touch on an inappropriate area is detected. According to such a control method, the viewer 112 will no longer touch the viewed character 114b indiscriminately, and the viewer 112 can be made to respect the personality of the character 114. Viewer 112's efforts to interact with character 114 in a way that does not offend the character contribute to making the character 114 seem more real.
[0066] As described above, when the conversation time ends, the curtain 350 slowly closes. The viewer 112 can express his or her sadness by touching the character 114 disappearing behind the curtain 350 through the gap in the curtain 350.
[0067] FIG. 14 is a sequence diagram showing the process when a gift is given to the character 114. The viewer 112 can also give goods to his / her favorite character 114. At the event venue, various goods (physical objects) such as bouquets of flowers, hats, ribbons, sweets, etc. are sold. These goods are physical entities, and have tags attached thereto in which goods IDs are recorded. These tags may be RFID (Radio Frequency IDentifier) tags or stickers in which QR codes (registered trademarks) are recorded.
[0068] The viewer 112 presents the purchased goods to the character 114 by inserting the goods into the gift box 306. The object insertion unit 320 first detects the passing of the goods using the detection device 310 (S30). The object identification unit 322 optically reads the goods ID (S32). The transmission unit 186 of the viewing device 108 transmits the goods ID to the server 102 (S34). The communication unit 152 of the server 102 transmits data of the goods image (virtual object) to the distribution device 104 and the viewing device 108 (S36, S37).
[0069] The object display unit 314 of the distribution device 104 displays the goods image on the distribution screen (S38). Similarly, the object display unit 328 of the viewing device 108 displays the goods image on the viewing screen (S40). The live control unit 154 of the server 102 registers the gifted goods as a purchase history in association with the goods ID, the date and time of provision, the user ID of the viewer 112 who provided the goods, and the character ID of the provided distributor 110 (character 114) (S42). Through the above process, the goods are presented from the viewer 112 to the character 114.
[0070] FIG. 15 is a diagram showing the screen of the viewing device 108 when a good is offered to the character 114. It is assumed that the viewer 112 purchases a hat (goods as a physical entity) and presents it to the character 114. At this time, the object display unit 314 displays the hat image 212 (goods image as a virtual entity). In FIG. 15, the viewed character 114b is wearing the presented hat image 212. If the character 114 wears the hat that the viewer 112 presented, the viewer 112 can feel a sense of satisfaction as if his sincerity was conveyed to the character 114. Since the three-dimensional hat is converted into the two-dimensional hat image 212, the viewer 112 can feel a greater "connection beyond dimensions" with the character 114. Also, by presenting goods, the viewer 112 can enjoy the change in the fashion of his favorite character 114. The viewer 112 may present multiple types of goods to the character 114 at the same time.
[0071] A portion of the purchase price of the goods is allocated to the distributor 110 who plays the character 114. In addition, the goods actually purchased may be given to the distributor 110. By purchasing goods, the viewer 112 can express his / her affection and feelings of support for the character 114. By using the merchandise income as an incentive, the distributor 110 can be encouraged to "make an effort to be loved," such as by improving his / her conversation skills.
[0072] FIG. 16 is an external view of the digital signage device 300. As shown in FIG. The digital signage device 300 is a large-scale advertisement display device targeted at an unspecified number of passersby. The digital signage device 300 in this embodiment will be described as a free-standing structure that can be set up in a station, a square, or the like.
[0073] As described in relation to FIG. 1, the digital signage device 300 is connected to the server 102. The digital signage device 300 broadcasts a live conversation between the distributor 110 (character 114) and the viewer 112. The digital signage device 300 displays the character 114 in the same display format as the viewing device 108. A conversation display area 362 is provided at the bottom of the screen, and both the first message (voice of the distributor 110) and the second message (voice of the viewer 112) are displayed in text. By visually recognizing the digital signage device 300, passersby can enjoy the conversation between the character 114 (distributor 110) and the viewer 112. Publishing the conversation on the digital signage device 300 is also an opportunity for the distributor 110 to appeal to many people about the appeal of the character 114. However, when publishing the conversation between the viewer 112 and the character 114 on the digital signage device 300, it is a prerequisite to obtain the consent of the viewer 112.
[0074] FIG. 17 is a functional block diagram of the digital signage device 300. As shown in FIG. The digital signage device 300 includes a communication unit 352 , a display unit 354 , and a data processing unit 356 . The communication unit 352 is responsible for communication processing. The display unit 354 displays images. The data processing unit 356 executes various processes based on data acquired by the communication unit 352. The data processing unit 356 also functions as an interface between the communication unit 352 and the display unit 354.
[0075] The communication unit 352 includes a transmitting unit 358 and a receiving unit 368 .
[0076] The data processing unit 356 includes a character conversion unit 366. The live control unit 154 of the server 102 also transmits the first message and the second message to the digital signage device 300. The character conversion unit 366 converts the first message and the second message from voice to a character string (text).
[0077] The display unit 354 includes a character display unit 370 and a message display unit 364. The character display unit 370 receives operation information from the server 102 and displays the character 114. The message display unit 364 displays various messages converted into characters in the conversation display area 362.
[0078] <Summary> The live communication system 100 has been described above based on the embodiment. According to this embodiment, in the conversation booth 302, the viewer 112 can not only enjoy a one-to-one conversation with the character 114, but can also perform operations that mimic a "handshake," thereby further increasing the intimacy between the viewer 112 and the character 114 (distributor 110). A handshake is established when the broadcaster 110 and the viewer 112 touch the screen at the same time and bring their fingers close to each other. In order to establish a handshake, an implicit communication between the viewer 112 and the character 114 (distributor 110) is necessary. When a handshake is established, the broadcaster 110 can have a unique sensation, as if they are in sync with the character 114 and have communicated with each other.
[0079] Furthermore, when a handshake is established, the character 114 (distributor 110) and the viewer 112 can shake their hands by moving their fingers so that they do not move too far apart. The more the handshake is shaken, the more decorative images 360 are displayed. As the viewer 112 and the character 114 (distributor 110) continue to shake hands in sync, the viewer 112 can feel an even greater sense of intimacy with the character 114.
[0080] In this embodiment, the curtain 350 is used to express the start and end of the conversation. By opening the curtain 350 to reveal the character 114, it is possible to effectively heighten the viewer 112's sense of anticipation for the precious opportunity to have a one-on-one conversation with the character 114. In addition, since the curtain 350 slowly closes when the conversation ends, it is possible to express the viewer 112's reluctance to part with the character 114. The viewer 112 is likely to feel the desire to say something to the character 114, who is hidden by the curtain 350. In addition, this reluctance to part can arouse the viewer 112's desire to meet the character 114 again.
[0081] The viewer 112 can enjoy physical contact other than a handshake by touching the character 114 himself. On the other hand, if the viewer 112 touches an inappropriate area of the character 114, a penalty such as a shortened conversation time is imposed. The distributor 110 may express his "rejection" by refusing to shake hands with the viewer 112 who has behaved in an inappropriate manner. This control method can encourage the viewer 112 to interact with the character 114 in a way that respects the personality of the other person.
[0082] By broadcasting live or recording and displaying the conversation between the character 114 and the viewer 112 on the digital signage device 300, it is possible to widely convey the appeal of the live communication system 100 to users who are not interested in or have not heard of live communication conversations with the character 114. Also, for the distributor 110, it is an opportunity to widely promote the appearance and conversational ability of the character 114, so the digital signage device 300 is considered to be useful in increasing the pride of the distributor 110 in his / her job.
[0083] The viewer 112 can purchase goods and give them to the character 114. A goods image (two-dimensional object) corresponding to the goods (three-dimensional object) is displayed on the viewing device 108. This provides the viewer 112 with an opportunity to show affection for the character 114, who is a resident of the virtual world, by giving the real-world goods to the character 114.
[0084] The present invention is not limited to the above-mentioned embodiment and modified examples, and the components can be modified without departing from the gist of the present invention. Various inventions can be formed by appropriately combining multiple components disclosed in the above-mentioned embodiment and modified examples. In addition, some components can be deleted from all the components shown in the above-mentioned embodiment and modified examples.
[0085] [Variations] <Hand games> FIG. 18 is a diagram showing the screen of the audiovisual device 108 when playing a hand game with the character 114. In the present embodiment, it has been described that the distributor 110 and the viewer 112 display the first hand 342 and the second hand 344 by touching the distribution screen and the viewing screen, respectively. As a modified example, the positions of the hands of the distributor 110 and the viewer 112 may be detected by image recognition, rather than touching the distribution screen, etc. The distributor 110 and the viewer 112 may wear white gloves, for example. In this case, the first hand detection unit 134 and the second hand detection unit 318 can more reliably recognize the positions of the hands of the distributor 110 and the viewer 112 from the captured image.
[0086] The first hand detection unit 134 detects the hand of the distributor 110 by image recognition, and specifies the first hand coordinates 340 based on the detected position of the hand. A technique for converting the position of an object shown in a captured image into coordinates on the screen by projecting it onto the screen is possible by application of a known technique. Similarly, the second hand detection unit 318 detects the position of the hand of the viewer 112 by image recognition, and specifies the second hand coordinates 346.
[0087] The first hand detection unit 134 performs image recognition of not only the position of the hand of the distributor 110 but also the shape of the hand (the state of the fingers). The second hand detection unit 318 also performs image recognition of the shape of the hand of the viewer 112. The hand display unit 142 etc. changes the shapes of the first hand 342 and the second hand 344 based on the hand shapes of the distributor 110 and the viewer 112, respectively. According to such a control method, for example, the viewer 112 can enjoy playing hand games such as "rock, paper, scissors" with the character 114 (distributor 110).
[0088] The face direction of the viewer 112 can be seen by the distributor 110. The face direction of the distributor 110 is reflected as the movement of the character 114. If not only the hand shapes but also the face directions can be mutually recognized, it is also possible to play games using the body such as "rock, paper, scissors." The winner of the hand game may be determined independently by the distributor 110 and the viewer 112, or a "win / loss determination unit" may be provided in the server 102. For example, in the case of rock-paper-scissors, the win / loss determination unit recognizes "rock," "scissors," or "paper" from the hand shapes of the distributor 110 and the viewer 112, respectively, and determines whether the character 114 (distributor 110) or the viewer 112 has won.
[0089] Also, depending on whether the viewer 112 wins or loses, merits or demerits may be given to the viewer 112. For example, if the viewer wins the hand game, the conversation time may be extended, the character 114 (the distributor 110) may utter lines requested by the viewer 112, or the character 114 may be given a novelty such as a key chain.
[0090] <Photo with Character 114> The distributor 110 may take a commemorative photo with the character 114. The imaging unit 316 of the viewing device 108 captures an image of the viewer 112. The data processing unit 166 of the viewing device 108 may include an "image generation unit." The image generation unit generates a commemorative photo of the two of them by combining the captured image of the viewer 112 with the character 114. The transmission unit 186 of the viewing device 108 may transmit the commemorative photo to the mobile terminal of the viewer 112.
[0091] Not only commemorative photographs but also short movies may be created by shooting moving images. The viewer 112 may request the character 114 to pose for the commemorative photograph or to say lines that he / she would like to see. A short movie showing the viewer 112 and the character 114 may be presented to the viewer 112 as a memento of his / her visit by combining the moving image of the character 114 with the captured image (moving image) of the viewer 112.
[0092] <How to move your hands> The viewer 112 may express his / her involvement with the character 114 by the way he / she moves his / her hand when touching the character 114. For example, when the viewer 112 slowly moves his / her hand from side to side while touching the viewing screen, the second hand 344 may express a state of gently stroking the character 114. Specifically, the second hand detection unit 318 may detect the movement of the viewer 112's hand, and may determine that the hand movement speed is within a predetermined range as "stroking". In addition, when the viewer 112 moves his / her hand quickly, the second hand 344 may express a state of hitting the character 114. In this case, the character 114 may be penalized as an inappropriate behavior. The decoration unit 326 may display various decoration images 360 on the screen depending on not only the place where the character 114 is touched but also the way in which it is touched.
[0093] <Other> The live control unit 154 may register the character 114 who has behaved inappropriately on a blacklist. The live control unit 154 may register the viewer 112 who came to meet the character 114, the goods provided by the viewer 112, and whether or not the viewer 112 has behaved in an inappropriate manner. When starting a conversation, the live control unit 154 transmits and presents various attribute information regarding the viewer 112 to the distributor 110. By referring to this attribute information, the distributor 110 can confirm various personal information, such as whether the viewer 112 is a viewer 112 who has come to meet the character 114 for the first time or whether the viewer 112 gave a present last time, and therefore can respond appropriately to a wide variety of viewers 112 (fans). For example, if the viewer 112 gave the character 114 a hat in the last conversation, the viewer 112 may feel relieved by thanking the character 114 for the hat, knowing that the character 114 remembered him.
[0094] Similarly, for viewers 112 who have engaged in inappropriate behavior (such as touching inappropriate areas or making unpleasant remarks), character 114 (distributor 110) can convey his / her displeasure to the viewers 112 by refusing to shake hands or shortening the conversation time. This method fosters good manners among viewers 112 (fans) toward character 114 (idol), which makes it easier for viewers 112 to respect character 114 as an individual. In addition, this kind of manners is also considered to be effective in further increasing the sense of reality and importance of character 114.
[0095] In the present embodiment, the first hand 342 is displayed separately from the image of the character 114. The hand display unit 324 or the like may express a handshake between the viewer 112 and the character 114 by moving the character 114 itself. Also, the viewer 112 may touch the hand of the character 114 to complete the handshake.
[0096] The output unit 172 of the viewing device 108 may output a sound effect when a handshake is established, or may provide a tactile sensation to the viewer 112 using a built-in vibrator. The output unit 172 may also operate a lighting device (not shown) installed in the conversation booth 302 to change the amount of light or color of lighting when a handshake is established, thereby creating the effect of a handshake. The output unit 172 may vary the effects and other effects for each character 114, or for the number or duration of handshakes. The decoration unit 326 may display multiple types of decoration images 360 for each character 114.
[0097] The distribution device 104 may be equipped with a "text conversion unit." The text conversion unit of the distribution device 104 may convert the first message (audio) and the second message (audio) into text and display them on the distribution screen. With such a control method, the distributor 110 can continue the conversation while checking the content of the conversation with the viewer 112. The live control unit 154 may record the conversation between the character 114 (distributor 110) and the viewer 112 as text information, and provide the distributor 110 with a record of the past conversation when the conversation starts. By providing the record of the past conversation, the distributor 110 can more easily respond to a large number of viewers 112 in a detailed manner.
[0098] In this embodiment, a handshake has been described, but it is not limited to a handshake, and may be a clap of hands or a high-five. The viewer 112 places the palm of his / her hand on the viewing screen. The distributor 110 also places the palm of his / her hand on the distribution screen. When the hand coordinates of the two people approach within a predetermined range, the hand display unit 324 or the like may display an animation of a clap of hands, a high-five, or the like. At this time, the decoration unit 326 may also display a decoration image 360.
[0099] <Other application examples> The live communication system 100 can also be used for customer service in unmanned stores. For example, the viewing device 108 may be installed in a self-checkout register, which is currently becoming common in convenience stores and the like. A store clerk (distributor 110) in a remote location interacts with a visitor (viewer 112) as a character 114. The visitor can enjoy shopping while interacting with the character 114. The visitor may also receive advice from the character 114 (distributor 110) about shopping for clothes or the like. By applying the live communication system 100, the visitor can feel the warmth of the customer service even in an unmanned store. In addition, since there is no need for the distributor 110 to be present in the store, there is an advantage for the store side in that it can reduce the operational burden.
[0100] Not limited to stores, conversation services through the character 114 from remote locations such as nursing homes and kindergartens are also possible. By combining the extraordinary nature of the character 114 with the reality of the distributor 110, the ease of interacting with the character 114 and the personality of the distributor 110 are combined. Another advantage is that the distributor 110 can serve multiple viewers 112 at the same time.
[0101] A remote class may also be considered using the live communication system 100. For example, an English teacher may be a broadcaster 110, and students may be viewers 112. The broadcaster 110 (English teacher) may provide an English lesson service to viewers 112 (students) from a remote location as a character 114.
[0102] The live communication system 100 also enables discussions between characters 114. For example, assume that user X plays character 114x and user Y plays character 114y. Character 114y is displayed on the communication terminal of user X, and character 114x is displayed on the communication terminal of user Y. With such a control method, user X and user Y can enjoy anonymous conversation while wearing the "mask" of character 114. Other possibilities include remote parties (joint parties). Since users meet through characters 114, there is a possibility that they can have a conversation centered on personality rather than appearance. [Explanation of symbols]
[0103] 100 live communication system, 102 server, 104 distribution device, 106 Internet, 108 viewing device, 110 distributor, 112 viewer, 114 character, 114a distribution character, 114b viewing character, 116 user interface processing unit, 118 imaging unit, 120 data processing unit, 122 communication unit, 124 data storage unit, 126 input unit, 128 output unit, 130 motion detection unit, 132 first message acquisition unit, 134 first hand detection unit, 138 distribution character display unit, 140 second message output unit, 142 hand display unit, 144 motion information generation unit, 148 transmission unit, 150 reception unit, 152 communication unit, 154 live control unit, 156 data storage unit, 158 character storage unit, 162 user interface processing unit, 164 communication unit, 166 data processing unit, 168 Data storage unit, 170 Input unit, 172 Output unit, 174 Second message acquisition unit, 180 Viewing character display unit, 182 First message output unit, 186 Transmission unit, 190 Reception unit, 196 Viewer conversation screen, 206 Distributor conversation screen, 208 Camera, 212 Hat image, 290 Conversation management unit, 300 Digital signage device, 302 Conversation booth, 304 Camera, 306 Present box, 308 Insertion slot, 310 Detection device, 312 Decoration unit, 314 Object display unit, 316 Imaging unit, 318 Second hand detection unit, 320 Object insertion unit, 322 Object identification unit, 324 Hand display unit, 326 Decoration unit, 328 Object display unit, 330 Curtain display unit, 332 Live image output unit, 334 Character display area, 336 Viewer display area, 338 handshake image, 340 first hand coordinates, 342 first hand, 344 second hand, 346 second hand coordinates, 348 handshake point, 350 curtain, 352 communication unit, 354 display unit, 356 data processing unit, 358 transmission unit, 360 decoration image, 362 conversation display area, 364 message display unit, 366 character conversion unit, 368 reception unit, 370 character display unit, d hand distance
Claims
1. A distribution device operated by a distributor and a viewing device operated by a viewer, The distribution device includes: An imaging unit that images a broadcaster; a motion information generating unit that generates motion information indicating characteristics of the motion of the distributor from a captured image of the distributor; a first input detection unit that detects a first input coordinate corresponding to a position of an input by a distributor; a transmission unit that transmits the motion information and the first input coordinates, The viewing device includes: a receiving unit that receives the motion information and the first input coordinates; a second input detection unit that detects second input coordinates corresponding to a position of a viewer's input; a viewing character display unit that displays a moving image of a character on a screen of the viewing device and moves the character in accordance with the action information; a predetermined display unit that determines whether the first input coordinates and the second input coordinates satisfy a predetermined positional relationship, and performs a predetermined display on a screen of the viewing device when the predetermined positional relationship is satisfied, A live communication system characterized in that the specified display unit of the viewing device determines whether the specified positional relationship is satisfied only when input from both the distributor and the viewer is maintained.
2. A system including a distribution device operated by a distributor and a viewing device operated by a viewer, The distribution device includes: An imaging unit that images a broadcaster; a motion information generating unit that generates motion information indicating characteristics of the motion of the distributor from a captured image of the distributor; a first input detection unit that detects a first input coordinate corresponding to a position of an input by a distributor; a transmission unit that transmits the motion information and the first input coordinates, The viewing device includes: a receiving unit that receives the motion information and the first input coordinates; a second input detection unit that detects second input coordinates corresponding to a position of a viewer's input; a viewing character display unit that displays a moving image of a character on a screen of the viewing device and moves the character in accordance with the action information; a predetermined display unit that determines whether the first input coordinates and the second input coordinates satisfy a predetermined positional relationship, and performs a predetermined display on a screen of the viewing device when the predetermined positional relationship is satisfied, A live communication system characterized in that the specified display unit of the viewing device cancels the specified display when, after the specified display has started, the first input coordinates and the second input coordinates no longer satisfy the specified positional relationship.
3. A system including a distribution device operated by a distributor and a viewing device operated by a viewer, The distribution device includes: An imaging unit that images a broadcaster; a motion information generating unit that generates motion information indicating characteristics of the motion of the distributor from a captured image of the distributor; a first input detection unit that detects a first input coordinate corresponding to a position of an input by a distributor; a transmission unit that transmits the motion information and the first input coordinates, The viewing device includes: a receiving unit that receives the motion information and the first input coordinates; a second input detection unit that detects second input coordinates corresponding to a position of a viewer's input; a viewing character display unit that displays a moving image of a character on a screen of the viewing device and moves the character in accordance with the action information; a predetermined display unit that determines whether the first input coordinates and the second input coordinates satisfy a predetermined positional relationship, and performs a predetermined display on a screen of the viewing device when the predetermined positional relationship is satisfied, the first input detection unit of the broadcasting device detects the first input coordinates based on a touch point when a broadcaster touches a screen of the broadcasting device; A live communication system, characterized in that the second input detection unit of the viewing device detects the second input coordinates based on a touch point when a viewer touches a screen of the viewing device.
4. A system including a distribution device operated by a distributor and a viewing device operated by a viewer, The distribution device includes: An imaging unit that images a broadcaster; a motion information generating unit that generates motion information indicating characteristics of the motion of the distributor from a captured image of the distributor; a first input detection unit that detects a first input coordinate corresponding to a position of an input by a distributor; a transmission unit that transmits the motion information and the first input coordinates, The viewing device includes: a receiving unit that receives the motion information and the first input coordinates; a second input detection unit that detects second input coordinates corresponding to a position of a viewer's input; a viewing character display unit that displays a moving image of a character on a screen of the viewing device and moves the character in accordance with the action information; a predetermined display unit that determines whether the first input coordinates and the second input coordinates satisfy a predetermined positional relationship, and performs a predetermined display on a screen of the viewing device when the predetermined positional relationship is satisfied, A live communication system characterized in that the viewing character display unit of the viewing device changes the reaction of the character depending on both or either of the touched area and touch manner of the character when a viewer touches the character on the screen of the viewing device.
5. A system including a distribution device operated by a distributor and a viewing device operated by a viewer, The distribution device includes: An imaging unit that images a broadcaster; a motion information generating unit that generates motion information indicating characteristics of the motion of the distributor from a captured image of the distributor; a first input detection unit that detects a first input coordinate corresponding to a position of an input by a distributor; a transmission unit that transmits the motion information and the first input coordinates, The viewing device includes: a receiving unit that receives the motion information and the first input coordinates; a second input detection unit that detects second input coordinates corresponding to a position of a viewer's input; a viewing character display unit that displays a moving image of a character on a screen of the viewing device and moves the character in accordance with the action information; a predetermined display unit that determines whether the first input coordinates and the second input coordinates satisfy a predetermined positional relationship, and performs a predetermined display on a screen of the viewing device when the predetermined positional relationship is satisfied, The viewing device is a live communication system further characterized in that it includes a conversation management unit that shortens the conversation time between the broadcaster and the viewer when a specified part of the character is touched during a conversation between the broadcaster and the viewer via the character.
6. A system including a distribution device operated by a distributor and a viewing device operated by a viewer, The distribution device includes: An imaging unit that images a broadcaster; a motion information generating unit that generates motion information indicating characteristics of the motion of the distributor from a captured image of the distributor; a first input detection unit that detects a first input coordinate corresponding to a position of an input by a distributor; a transmission unit that transmits the motion information and the first input coordinates, The viewing device includes: a receiving unit that receives the motion information and the first input coordinates; a second input detection unit that detects second input coordinates corresponding to a position of a viewer's input; a viewing character display unit that displays a moving image of a character on a screen of the viewing device and moves the character in accordance with the action information; a predetermined display unit that determines whether the first input coordinates and the second input coordinates satisfy a predetermined positional relationship, and performs a predetermined display on a screen of the viewing device when the predetermined positional relationship is satisfied, The viewing device includes: an object insertion unit that detects an insertion of a physical object by a viewer; an object identification unit for detecting an ID of the thrown physical object; and an object display unit that displays a virtual object corresponding to the detected ID on a screen of the viewing device.
7. A live communication system as described in any one of claims 1 to 6, characterized in that the specified display unit of the viewing device determines that the specified positional relationship is satisfied when the distance between the first input coordinate and the second input coordinate is within a first threshold value.
8. The function of capturing images of the broadcaster, A function of generating motion information indicating characteristics of the motion of the distributor from a captured image of the distributor; A function of detecting a first input coordinate corresponding to a position of an input by a distributor; a first program that causes a distribution device to perform a function of transmitting the motion information and the first input coordinates; receiving the motion information and the first input coordinate; detecting a second input coordinate corresponding to a position of the viewer's input; a function of displaying a moving image of a character on a screen of a viewing device and moving the character in accordance with the motion information; a second program for causing the viewing device to perform a function of determining whether or not the first input coordinates and the second input coordinates satisfy a predetermined positional relationship, and, if the predetermined positional relationship is satisfied, performing a predetermined display on a screen of the viewing device; A live communication program characterized in that the function of performing the specified display determines whether the specified positional relationship is satisfied only when input from both the distributor and the viewer is maintained.
9. A function for capturing an image of a broadcaster; A function of generating motion information indicating characteristics of the motion of the distributor from a captured image of the distributor; A function of detecting a first input coordinate corresponding to a position of an input by a distributor; a first program causing a distribution device to perform a function of transmitting the motion information and the first input coordinates; receiving the motion information and the first input coordinate; detecting a second input coordinate corresponding to a position of the viewer's input; a function of displaying a moving image of a character on a screen of a viewing device and moving the character in accordance with the motion information; a second program for causing the viewing device to perform a function of determining whether or not the first input coordinates and the second input coordinates satisfy a predetermined positional relationship, and, if the predetermined positional relationship is satisfied, performing a predetermined display on a screen of the viewing device; A live communication program characterized in that the function of performing the specified display cancels the specified display when the first input coordinate and the second input coordinate no longer satisfy the specified positional relationship after the specified display has started.
10. A function for capturing an image of a broadcaster; A function of generating motion information indicating characteristics of the motion of the distributor from a captured image of the distributor; A function of detecting a first input coordinate corresponding to a position of an input by a distributor; a first program causing a distribution device to perform a function of transmitting the motion information and the first input coordinates; receiving the motion information and the first input coordinate; detecting a second input coordinate corresponding to a position of the viewer's input; a function of displaying a moving image of a character on a screen of a viewing device and moving the character in accordance with the motion information; a second program for causing the viewing device to perform a function of determining whether or not the first input coordinates and the second input coordinates satisfy a predetermined positional relationship, and, if the predetermined positional relationship is satisfied, performing a predetermined display on a screen of the viewing device; the function of detecting the first input coordinates detects the first input coordinates based on a touch point when a broadcaster touches a screen of the broadcasting device; A live communication program, characterized in that the function of detecting the second input coordinates detects the second input coordinates based on a touch point when a viewer touches the screen of the viewing device.
11. A function for capturing an image of a broadcaster; A function of generating motion information indicating characteristics of the motion of the distributor from a captured image of the distributor; A function of detecting a first input coordinate corresponding to a position of an input by a distributor; a first program causing a distribution device to perform a function of transmitting the motion information and the first input coordinates; receiving the motion information and the first input coordinate; detecting a second input coordinate corresponding to a position of the viewer's input; a function of displaying a moving image of a character on a screen of a viewing device and moving the character in accordance with the motion information; a second program for causing the viewing device to perform a function of determining whether or not the first input coordinates and the second input coordinates satisfy a predetermined positional relationship, and, if the predetermined positional relationship is satisfied, performing a predetermined display on a screen of the viewing device; A live communication program characterized in that the function of moving the character changes the reaction of the character depending on both or either of the touched part and the touch manner of the character when a viewer touches the character on the screen of the viewing device.
12. A function for capturing an image of a broadcaster; A function of generating motion information indicating characteristics of the motion of the distributor from a captured image of the distributor; A function of detecting a first input coordinate corresponding to a position of an input by a distributor; a first program causing a distribution device to perform a function of transmitting the motion information and the first input coordinates; receiving the motion information and the first input coordinate; detecting a second input coordinate corresponding to a position of the viewer's input; a function of displaying a moving image of a character on a screen of a viewing device and moving the character in accordance with the motion information; a second program for causing the viewing device to perform a function of determining whether or not the first input coordinates and the second input coordinates satisfy a predetermined positional relationship, and, if the predetermined positional relationship is satisfied, performing a predetermined display on a screen of the viewing device; The second program is a live communication program that further causes the viewing device to exhibit a function of shortening the conversation time between the broadcaster and the viewer when a specified part of the character is touched during a conversation between the broadcaster and the viewer via the character.
13. A function for capturing an image of a broadcaster; A function of generating motion information indicating characteristics of the motion of the distributor from a captured image of the distributor; A function of detecting a first input coordinate corresponding to a position of an input by a distributor; a first program causing a distribution device to perform a function of transmitting the motion information and the first input coordinates; receiving the motion information and the first input coordinate; detecting a second input coordinate corresponding to a position of the viewer's input; a function of displaying a moving image of a character on a screen of a viewing device and moving the character in accordance with the motion information; a second program for causing the viewing device to perform a function of determining whether or not the first input coordinates and the second input coordinates satisfy a predetermined positional relationship, and, if the predetermined positional relationship is satisfied, performing a predetermined display on a screen of the viewing device; The second program is A function to detect the insertion of physical objects by viewers; detecting an ID of the inserted physical object; and a function of displaying a virtual object corresponding to the detected ID on a screen of the viewing device.
Citation Information
Patent Citations
Magnetic recording medium having zinc-containing auxiliary layer and its production
JP1988042024A
Avatar display device in virtual space communication system, avatar displaying method and storage medium
JP2001160154A
Video content viewing system and video content distribution service
JP2012231281A
Social touch control device
JP2016212564A
Method and device for assisting communication in virtual space and program enabling computer to execute method
JP2018032384A
