Computer program, method and server device

Abstract

To provide a computer program, a server device, a terminal device, and a display method for displaying an image based on motions of a performer by means of novel techniques.SOLUTION: A terminal device 20 or a server device 30 includes: a storage unit for storing various pieces of information deemed to be required for distributing images and / or receiving images; a sensor unit for acquiring image data regarding a body of a performer per unit time; an amount-of-change acquisition unit for acquiring, based on the acquired measurement data regarding the body of the performer, an amount of change per unit time for each of a plurality of body parts of the performer; a referential value acquisition unit for acquiring, based on the measurement data, a referential value based on the amount of change per unit time for the plurality of body parts of the performer; an identifying unit for, when an event whose referential value exceeds a threshold is detected, identifying at least one upper ranked part whose amount of change per unit time is great; and determining unit for determining any one of specific motions associated with the identified, at least one upper ranked part whose amount of change per unit is great, as a motion performed by the performer.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The technology disclosed in this application relates to a computer program, a method, and a server device for displaying an image based on the actions of a performer (user) in various applications. [Background technology]

[0002] One service known as an Animoji service uses technology that controls the facial expressions of a virtual character displayed in an application based on the facial expressions of the performer (Non-Patent Document 1). In this service, users can change the facial expression of an avatar displayed in a messenger application by changing their facial expression while looking at a smartphone equipped with a camera that detects deformations in the shape of the face.

[0003] Another known service is called "Custom Cast" (Non-Patent Document 2). In this service, a user assigns one of a large number of prepared facial expressions to each of a number of flick directions on the smartphone screen. Furthermore, when a video is distributed, the user can make an avatar displayed in the video express a desired facial expression by flicking the screen in the direction corresponding to the desired facial expression.

[0004] Each of the above non-patent documents 1 and 2 is incorporated herein by reference in its entirety. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] "How to use Animoji on iPhone X and later", [online], October 24, 2018, Apple Japan, Inc., [Retrieved January 10, 2020], Internet (URL: https: / / support.apple.com / ja-jp / HT208190) [Non-Patent Document 2] "Custom Cast", [online], October 3, 2018, Dwango Co., Ltd., [Retrieved January 10, 2020], Internet (URL: https: / / customcast.jp / ) Summary of the Invention [Problem to be solved by the invention]

[0006] In an application that displays a virtual character (such as an avatar), it is sometimes desired to have the character express an impressive facial expression. Impressive facial expressions include, for example, the following three examples. A first example is a facial expression in which the shape of the face is unrealistically deformed like a cartoon. This facial expression includes, for example, a facial expression in which both eyes protrude from the face. A second example is a facial expression in which symbols, shapes and / or colors are added to the face. This facial expression includes, for example, a facial expression with tears spilling, a facial expression with a bright red face, an angry facial expression with triangular eyes, and the like. A third example is a facial expression that expresses emotions including joy, anger, sadness and happiness. Impressive facial expressions are not limited to these examples.

[0007] However, first, the technology described in Patent Document 1 changes the facial expression of the virtual character so as to follow the change in the shape of the user's (performer's) face. Therefore, the technology described in Patent Document 1 has difficulty in expressing the above-mentioned impressive facial expressions in the facial expression of the virtual character, which are difficult for the user's face to actually express.

[0008] Next, in the technology described in Patent Document 2, it is necessary to pre-assign facial expressions to be expressed by a virtual character to each of a plurality of flick directions. For this reason, the user (performer) needs to recognize all of the prepared facial expressions. Furthermore, the total number of facial expressions that can be assigned to a plurality of flick directions and used at one time is limited to less than 10, which is not sufficient.

[0009] Therefore, some embodiments disclosed in the present application provide a computer program, a server device, a terminal device, and a display method for displaying an image based on the movements of a performer in a new manner. [Means for solving the problem]

[0010] A computer program according to one embodiment can be executed by at least one processor to cause the processor to function as follows: "to retain information associating each of at least one specific movement with at least one top part of a plurality of parts of an actor's body having the greatest amount of change, to identify at least one top part of the actor's body having the greatest amount of change per unit time using measurement data on the actor's body, and to determine, as a detected movement, any one of the at least one specific movement associated with the identified at least one top part of the body having the greatest amount of change per unit time using the information."

[0011] A computer program according to another embodiment can cause the processor to "function, when executed by at least one processor, to store information associating each of at least one specific movement with at least one top part of a plurality of parts of an actor's body having the greatest amount of change, to identify at least one top part of the actor's body having the greatest amount of change per unit time using measurement data on the actor's body transmitted by the actor's terminal device, and to determine, as a detected movement, any one of the at least one specific movement associated with the identified at least one top part of the body having the greatest amount of change per unit time using the information."

[0012] A method according to one embodiment can be "a method executed by at least one processor that executes computer-readable instructions, wherein the processor executes the instructions to store information correlating each of at least one specific movement with at least one top part of a plurality of parts of the performer's body having the greatest amount of change, identify at least one top part of the performer's body having the greatest amount of change per unit time using measurement data on the performer's body, and determine, using the information, as a detected movement, any one of the at least one specific movement that is associated with the identified at least one top part of the body having the greatest amount of change per unit time."

[0013] A server device in one embodiment is equipped with at least one processor, which, by executing computer-readable instructions, stores information correlating each of at least one specific movement with at least one top part of a plurality of parts of an actor's body having the greatest amount of change, identifies at least one top part of the actor's body having the greatest amount of change per unit time using measurement data on the actor's body, and uses the information to determine, as a detected movement, any one of the at least one specific movement that is associated with the identified at least one top part of the body having the greatest amount of change per unit time. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a block diagram showing an example of a configuration of a communication system according to an embodiment. [Diagram 2] FIG. 2 is a block diagram illustrating an example of a hardware configuration of the terminal device 20 (server device 30) illustrated in FIG. [Diagram 3] FIG. 3 is a block diagram illustrating an example of functions of the terminal device 20 (server device 30) illustrated in FIG. [Figure 4] FIG. 4 is a flow diagram showing an example of operations performed in the entire communication system 1 shown in FIG. [Diagram 5] FIG. 5 is a flow diagram showing an example of the operations related to generation and transmission of moving images among the operations shown in FIG. [Figure 6] FIG. 6 is a diagram illustrating an example of correspondence information used in the communication system illustrated in FIG. [Figure 7A] FIG. 7A is a diagram showing an example of a moving image displayed by the terminal device 20 or the like used in the communication system shown in FIG. [Figure 7B] FIG. 7A is a diagram showing another example of a moving image displayed by the terminal device 20 or the like used in the communication system shown in FIG. [Figure 7C] FIG. 7C is a diagram showing yet another example of a moving image displayed by the terminal device 20 or the like used in the communication system shown in FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Various embodiments of the present invention will be described below with reference to the accompanying drawings. Note that common components in the drawings are given the same reference numerals. Also, please note that components shown in one drawing may be omitted in another drawing for convenience of explanation. Furthermore, please note that the attached drawings are not necessarily drawn to scale.

[0016] 1. Examples of communication systems In short, in the communication system disclosed in the present application, a terminal device etc. provided opposite a distribution user (performer) can transmit images (moving images and / or still images) generated based on the actions of this distribution user to a terminal device etc. of each viewing user via a server device etc.

[0017] FIG. 1 is a block diagram showing an example of a configuration of a communication system according to an embodiment. As shown in FIG. 1, the communication system 1 may include one or more terminal devices 20 connected to a communication network 10, and one or more server devices 30 connected to the communication network 10. In FIG. 1, three terminal devices 20A to 20C are illustrated as examples of the terminal devices 20, and three server devices 30A to 30C are illustrated as examples of the server devices 30. However, one or more terminal devices 20 other than these may be connected to the communication network 10 as the terminal devices 20. In addition, one or more server devices 30 other than these may be connected to the communication network 10 as the server devices 30.

[0018] Furthermore, the communication system 1 may include one or more studio units 40 connected to the communication network 10. Note that in Fig. 1, two studio units 40A and 40B are illustrated as examples of the studio units 40. However, one or more studio units 40 other than these may be connected to the communication network 10 as the studio units 40.

[0019] In the "first aspect," in the communication system 1 shown in FIG. 1, a terminal device 20 (e.g., terminal device 20A) operated by a performer and executing a predetermined application (such as an application for video distribution) can acquire data related to the body of the performer facing the terminal device 20A and / or audio data related to the speech and / or singing uttered by the performer. Furthermore, this terminal device 20A can transmit an image of a virtual character generated according to the acquired data to a server device 30 (e.g., server device 30A) via the communication network 10. Furthermore, the server device 30A can distribute the image of the virtual character received from the terminal device 20A via the communication network 10 to one or more other terminal devices 20 that are executing a predetermined application (such as an application for video viewing) and have transmitted a request for image distribution.

[0020] In this specification, a "specific application" or a "particular application" may be one or more applications, or a combination of one or more applications and one or more middleware.

[0021] In the "second aspect," in the communication system 1 shown in Fig. 1, a server device 30 (e.g., server device 30B) installed in, for example, a studio or other location can acquire data on the body of a performer in the studio or other location and / or audio data on speech and / or singing uttered by the performer. Furthermore, this server device 30B can distribute, via the communication network 10, an image of a virtual character generated according to the acquired data to one or more terminal devices 20 that have executed a specific application (such as an application for watching videos) and transmitted a request for image distribution.

[0022] In the "third aspect," in the communication system 1 shown in Fig. 1, for example, a studio unit 40 installed in a studio or other location can acquire data on the body of a performer in the studio or other location and / or audio data on speech and / or singing uttered by the performer. Furthermore, this studio unit 40 can generate an image of a virtual character generated according to this data and transmit it to the server device 30. Furthermore, the server device 30 can distribute the image acquired (received) from the studio unit 40 via the communication network 10 to one or more terminal devices 20 that have executed a predetermined application (such as an application for watching videos) and transmitted a request for image distribution.

[0023] The communication network 10 may include, but is not limited to, a mobile phone network, a wireless LAN, a landline telephone network, the Internet, an intranet, and / or an Ethernet.

[0024] The terminal device 20 can execute an installed specific application to perform operations such as acquiring data on the body of the performer and / or audio data on the speech and / or singing uttered by the performer. Furthermore, the terminal device 20 can execute operations such as transmitting an image of a virtual character generated according to the acquired data to the server device 30 via the communication network 10. Alternatively, the terminal device 20 can execute an installed web browser to receive and display a web page from the server device 30 and perform similar operations.

[0025] The terminal device 20 may be any terminal device capable of performing such operations, and may include, without being limited to, a smartphone, a tablet, a mobile phone (feature phone), and / or a personal computer.

[0026] In the "first aspect," the server device 30 can function as an application server by executing a specific application installed thereon. This allows the server device 30 to perform operations such as receiving an image of a virtual character from each terminal device 20 via the communication network 10 and distributing the received image to each terminal device 20 via the communication network 10. Alternatively, the server device 30 can perform a similar operation via a web page transmitted to each terminal device 20 by executing a specific application installed thereon and functioning as a web server.

[0027] In the "second aspect", the server device 30 can execute a specific application installed therein to function as an application server. This allows the server device 30 to execute operations such as acquiring data on the body of a performer in a studio or other location where the server device 30 is installed and / or voice data on speech and / or singing uttered by the performer. Furthermore, the server device 30 can execute operations such as distributing an image of a virtual character generated according to the acquired data to each terminal device 20 via the communication network 10. Alternatively, the server device 30 can execute the same operation via a web page transmitted to each terminal device 20 by executing a specific application installed therein to function as a web server. Furthermore, the server device 30 can execute a specific application installed therein to function as an application server. This allows the server device 30 to execute operations such as acquiring (receiving) an image of a virtual character whose facial expression has been changed according to data on the body of a performer in the studio or other location and / or voice data on speech and / or singing uttered by the performer from a studio unit 40 installed in a studio or other location. Furthermore, the server device 30 can execute operations such as distributing this image to each terminal device 20 via the communication network 10.

[0028] The studio unit 40 can function as an information processing device that executes a specific application installed therein. This allows the studio unit 40 to acquire data on the body of a performer in the studio where the studio unit 40 is installed or in other locations, and / or audio data on the speech and / or singing uttered by the performer. Furthermore, the studio unit 40 can transmit an image of a virtual character generated according to the acquired data to the server device 30 via the communication network 10.

[0029] 2. Hardware configuration of each device Next, an example of the hardware configuration of each of the terminal device 20, the server device 30, and the studio unit 40 will be described. 2-1. Hardware configuration of terminal device 20 An example of the hardware configuration of each terminal device 20 will be described with reference to Fig. 2. Fig. 2 is a block diagram showing a schematic example of the hardware configuration of the terminal device 20 (server device 30) shown in Fig. 1. In Fig. 2, the reference characters in parentheses are written in relation to each server device 30, as described later.

[0030] 2, each terminal device 20 can mainly include a central processing unit 21, a main memory device 22, an input / output interface device 23, an input device 24, an auxiliary memory device 25, and an output device 26. These devices are connected to each other by a data bus and / or a control bus.

[0031] The central processing unit 21 may be referred to as a "CPU", and can perform operations on instructions and data stored in the main memory 22, and can store the results of the operations in the main memory 22. Furthermore, the central processing unit 21 can control an input device 24, an auxiliary memory device 25, an output device 26, and the like via an input / output interface device 23. The terminal device 20 may include one or more such central processing units 21.

[0032] The main memory device 22 may be referred to as "memory" and can store instructions and data received from the input device 24, the auxiliary memory device 25, and the communication network 10 (such as the server device 30) via the input / output interface device 23, as well as the results of calculations by the central processing unit 21. The main memory device 22 can include, without being limited to, a RAM (random access memory), a ROM (read only memory), and / or a flash memory.

[0033] The auxiliary storage device 25 is a storage device having a larger capacity than the main storage device 22. The auxiliary storage device 25 stores instructions and data (computer programs) constituting the specific application, web browser, etc., and is capable of transmitting these instructions and data (computer programs) to the main storage device 22 via the input / output interface device 23 under the control of the central processing unit 21. The auxiliary storage device 25 may include, but is not limited to, a magnetic disk device and / or an optical disk device.

[0034] The input device 24 is a device that inputs data from the outside, and may include, but is not limited to, a touch panel, a button, a keyboard, a mouse, and / or a sensor, etc. The sensor may include, but is not limited to, a first sensor including one or more cameras, etc., and / or a second sensor including one or more microphones, etc., as described below.

[0035] The output device 26 may include, but is not limited to, a display device, a touch panel, and / or a printer device.

[0036] In such a hardware configuration, the central processing unit 21 can sequentially load instructions and data (computer programs) constituting a specific application stored in the auxiliary storage device 25 into the main storage device 22 and perform operations on the loaded instructions and data. This allows the central processing unit 21 to control the output device 26 via the input / output interface device 23, and also to transmit and receive various information to and from other devices (such as the server device 30 and other terminal devices 20) via the input / output interface device 23 and the communication network 10.

[0037] Thus, the terminal device 20 can execute the installed specific application to obtain data on the performer's body and / or audio data on the speech and / or singing uttered by the performer, and transmit an image of a virtual character generated according to the obtained data to the server device 30 via the communication network 10 (including various operations described in detail later).Alternatively, the terminal device 20 can execute the installed web browser to receive and display a web page from the server device 30 and execute a similar operation.

[0038] It should be noted that the terminal device 20 may include one or more microprocessors and / or a graphics processing unit (GPU) instead of or in addition to the central processing unit 21.

[0039] 2-2. Hardware configuration of server device 30 An example of the hardware configuration of each server device 30 will also be described with reference to Fig. 2. The hardware configuration of each server device 30 may be the same as the hardware configuration of each terminal device 20 described above. Therefore, the reference characters for the components of each server device 30 are shown in parentheses in Fig. 2.

[0040] 2, each server device 30 can mainly include a central processing unit 31, a main memory device 32, an input / output interface device 33, an input device 34, an auxiliary memory device 35, and an output device 36. These devices are connected to each other by a data bus and / or a control bus.

[0041] The central processing unit 31, main memory device 32, input / output interface device 33, input device 34, auxiliary memory device 25 and output device 36 may be approximately the same as the central processing unit 21, main memory device 22, input / output interface device 23, input device 24, auxiliary memory device 25 and output device 26 included in each of the terminal devices 20 described above, respectively.

[0042] In such a hardware configuration, the central processing unit 31 can sequentially load instructions and data (computer programs) constituting a specific application stored in the auxiliary storage device 35 into the main storage device 32 and perform operations on the loaded instructions and data. This allows the central processing unit 31 to control the output device 36 via the input / output interface device 33, and also to transmit and receive various information to and from other devices (such as the terminal devices 20) via the input / output interface device 33 and the communication network 10.

[0043] As a result, in the "first aspect", the server device 30 can function as an application server by executing a specific application installed thereon. As a result, the server device 30 can execute operations such as receiving an image of a virtual character from each terminal device 20 via the communication network 10 and distributing the received image to each terminal device 20 via the communication network 10 (including various operations described in detail later). Alternatively, the server device 30 can execute a specific application installed thereon and function as a web server. As a result, the server device 30 can execute the same operations via a web page transmitted to each terminal device 20.

[0044] In the "second aspect", the server device 30 can execute a specific application installed therein to function as an application server. This allows the server device 30 to execute operations such as acquiring data on the body of a performer in a studio or other location where the server device 30 is installed and / or voice data on speech and / or singing uttered by the performer. Furthermore, the server device 30 can execute operations such as distributing an image of a virtual character generated according to the acquired data to each terminal device 20 via the communication network 10 (including various operations described in detail later). Alternatively, the server device 30 can execute a specific application installed therein to function as a web server. This allows the server device 30 to execute similar operations via a web page transmitted to each terminal device 20.

[0045] Furthermore, in the "third aspect," the server device 30 can execute a specific installed application to function as an application server. This allows the server device 30 to execute operations such as acquiring (receiving) from the studio unit 40 via the communication network 10 an image of a virtual character generated according to data on the body of a performer in a studio or other location where the studio unit 40 is installed and / or audio data on speech and / or singing uttered by the performer. Furthermore, the server device 30 can also execute operations such as distributing this image to each terminal device 20 via the communication network 10 (including various operations described in detail later).

[0046] It should be noted that the server device 30 may include, instead of or in addition to the central processing unit 31, one or more microprocessors and / or a graphics processing unit (GPU).

[0047] 2-3.Hardware configuration of Studio Unit 40 The studio unit 40 can be implemented by an information processing device such as a personal computer. Although not shown, the studio unit 40 can mainly include a central processing unit, a main memory device, an input / output interface device, an input device, an auxiliary memory device, and an output device, similar to the above-mentioned terminal device 20 and server device 30. These devices are connected to each other by a data bus and / or a control bus.

[0048] The studio unit 40 can execute a specific application installed therein and function as an information processing device. This allows the studio unit 40 to acquire data on the body of a performer in the studio where the studio unit 40 is installed or in other locations and / or audio data on the speech and / or singing uttered by the performer. Furthermore, the studio unit 40 can transmit an image of a virtual character generated according to the acquired data to the server device 30 via the communication network 10.

[0049] 3. Functions of each device Next, an example of the functions of each of the terminal device 20, the server device 30, and the studio unit 40 will be described. 3-1. Functions of terminal device 20 An example of the functions of the terminal device 20 will be described with reference to Fig. 3. Fig. 3 is a block diagram that illustrates an example of the functions of the terminal device 20 (server device 30) illustrated in Fig. 1.

[0050] 3, the terminal device 20 may include a storage unit 100, a sensor unit 110, a change amount acquisition unit 120, a discrimination unit 140, a determination unit 150, an image generation unit 160, a display unit 170, a user interface unit 180, and a communication unit 190. The terminal device 20 may further include a reference value acquisition unit 130.

[0051] (1) Storage section 100 The storage unit 100 can store various information required for image distribution and / or image reception. In particular, the storage unit 100 can store correspondence information. In the correspondence information, each of at least one predetermined specific motion is associated with at least one part of the performer's body that has the largest amount of change among multiple parts of the performer's body.

[0052] Here, the at least one predetermined specific action may include, for example, at least one of the actions exemplified below. (A) At least one of various facial expressions that are difficult for the performer to actually express (hereinafter referred to as "special facial expressions"). (B) At least one of various movements using the body (such as making a heart shape with both hands and expressing semaphore signals using both hands).

[0053] The special facial expression shown in (A) above may include, for example, at least one of the facial expressions (A1) to (A3) shown below. (A1) Facial expression in which the face shape is unrealistically distorted like a cartoon (A2) Facial expressions with symbols, shapes and / or colors added to the face (A3) Facial expressions expressing emotions including joy, anger, sadness and happiness

[0054] Furthermore, the multiple parts of the performer's body may include, without being limited to, the right eye, left eye, right eyebrow, left eyebrow, nose, mouth, right ear, left ear, chin, right cheek, left cheek, neck, right shoulder, left shoulder, right hand, left hand, chest, and / or a part of any of these parts. Here, if the part is the right eye, for example, the right end of the right eye, the left end of the right eye, the center of the right eye, the upper edge of the right eye, and / or the lower edge of the right eye, etc.

[0055] Furthermore, the correspondence information may also associate each of at least one predetermined specific movement with at least one top-ranking part among multiple parts of the performer's body having the greatest amount of change, and with an order based on the magnitude of the amount of change in the at least one top-ranking part having the greatest amount of change. Specific examples of the corresponding information will be described later.

[0056] (2) Sensor unit 110 The sensor unit 110 may include sensors such as various types of cameras and / or microphones, and at least one processor for processing information acquired by such sensors. The sensor unit 110 may acquire data (such as images and / or sounds) related to the body of the performer facing the sensor unit 110, and may further perform information processing on the data.

[0057] Specifically, for example, the sensor unit 110 can first obtain image data on the performer's body for each unit time using various types of cameras. Here, the unit time can be set or changed to any length at any timing by the user / performer or the like via the user interface unit 180. Furthermore, the sensor unit 110 can measure the positions of each of a plurality of parts of the performer's body for each unit time using the image data thus obtained. Here, the plurality of parts can include, without being limited to, the right eye, left eye, right eyebrow, left eyebrow, nose, mouth, right ear, left ear, chin, right cheek, left cheek, neck, right shoulder, left shoulder, right hand, left hand, chest, and / or any part of these parts. Note that various methods well known to those skilled in the art can be used as a method for measuring the positions of a plurality of parts of the performer's body for each unit time using image data obtained by a camera.

[0058] For example, in one embodiment, the sensor unit 110 may include, as sensors, an RGB camera that captures visible light and a near-infrared camera that captures near-infrared light. As such a camera, for example, the True Depth camera of the iPhone X (registered trademark) may be used. Note that, as the True Depth camera, the camera disclosed at https: / / developer.apple.com / documentation / arkit / arfaceanchor may be used. The contents of this website are incorporated herein in their entirety by reference.

[0059] With respect to the RGB camera, the sensor unit 110 can generate data that associates an image captured by the RGB camera with a time code (a code indicating the time when the image was captured) and records it for each unit of time. This data can be, for example, an MPEG file.

[0060] Furthermore, the sensor unit 110 can generate data in which a predetermined number (e.g., 51) of depth-indicating values ​​(e.g., floating-point values) acquired by the near-infrared camera are associated with the time code and recorded for each unit time. This data can be, for example, a TSV file. Here, a TSV file is a file format in which multiple pieces of data are recorded with tabs separating the data.

[0061] Concerning the near-infrared camera, specifically, first, a dot projector can emit an infrared laser including a dot (point) pattern onto the body of the performer. Furthermore, the near-infrared camera can capture the infrared dots projected onto and reflected from the body of the performer, and generate an image of the infrared dots thus captured. The sensor unit 110 can compare an image of a dot pattern emitted by a dot projector registered in advance with an image captured by the near-infrared camera, and calculate the depth of each point (each part) using the positional deviation of each point (e.g., each of 51 points and 51 parts) in both images. Here, the depth of each point (each part) can be the distance between each point (each part) and the near-infrared camera. The sensor unit 110 can generate data in which a numerical value indicating the depth thus calculated is associated with a time code as described above and recorded for each unit time.

[0062] This allows the sensor unit 110 to obtain, for each unit time, a moving image such as an MPEG file and the position (coordinates, etc.) of each part as data (measurement data) relating to the performer's body in association with the time code.

[0063] In another embodiment, the sensor unit 110 can use a technology called Argumented Faces. For Argumented Faces, information disclosed at https: / / developers.google.com / ar / develop / java / augmented-faces / can be used. The information disclosed at this website is incorporated herein by reference in its entirety.

[0064] By using Argumented Faces, the sensor unit 110 can obtain the following information for each unit time using images captured by the camera. (1) The physical center of the performer's skull, (2) a facial mesh that includes hundreds of vertices that make up the actor's face and is defined relative to the center location; and (3) The positions (coordinates) of the performer's face parts (e.g., right cheek, left cheek, tip of nose) identified based on (1) and (2) above

[0065] By using this technique, the sensor unit 110 can obtain the positions (coordinates) of multiple parts of the performer's upper body (face, etc.) every unit time.

[0066] The sensor unit 110 can obtain an audio signal by performing known signal processing on audio data relating to the speech and / or singing of a performer output from a microphone or the like. The audio signal may be, for example, an MPEG file or the like. Good too.

[0067] (3) Change amount acquisition unit 120 The change amount acquiring section 120 can acquire the amount of change per unit time of each of a plurality of parts of the performer's body based on the data (measurement data) related to the performer's body acquired by the sensor section 110. Specifically, for example, the change amount acquiring section 120 can obtain the difference between the position (coordinates) acquired in a first unit time for a part called the right cheek and the position (coordinates) acquired in a second unit time occurring next (immediately after) the first unit time. This allows the change amount acquiring section 120 to acquire the amount of change in the part called the right cheek between the first unit time and the second unit time. That is, the change amount acquiring section 120 can acquire the amount of change per unit time of the part called the right cheek. The change amount acquiring section 120 can similarly acquire the amount of change per unit time for other parts as well.

[0068] The amount of change per unit time for each part can be expressed, for example, by a floating point between 0 and 1.0. The unit time may be fixed, variable, or a combination of these. The unit time may also be a time equivalent to one frame.

[0069] (4) Reference value acquisition unit 130 The reference value acquisition unit 130 acquires a reference value based on the amount of change per unit time of multiple parts of the performer's body, based on data (measurement data) related to the performer's body acquired by the sensor unit 110. In one embodiment, the reference value R can be calculated using the following formula. TIFF0007662224000001.tif10141, where x i is the amount of change per unit time of the i-th part among the multiple parts of the performer's body. N is the total number of parts (2 or more). The value calculated by this formula is called the root mean square.

[0070] The reference value indicates the amount of change per unit time of a plurality of parts of the performer's body. When the reference value is large, it can be estimated that the performer's body has performed at least one of the predetermined specific movements. On the other hand, when the reference value is small, it can be estimated that the performer's body has not performed any of the predetermined at least one specific movements.

[0071] The reference value may be the root mean square value itself shown in the above formula, or may be a value obtained by multiplying this root mean square value by an arbitrary coefficient.

[0072] In another embodiment, the reference value R is expressed as x i It may be an average value of TIFF0007662224000002.tif8134Note that, here too, x i is the amount of change per unit time of the i-th part among the multiple parts of the performer's body, and N is the total number of parts (2 or more).

[0073] (4) Identification unit 140 The identification unit 140 can use the amount of change per unit time of each of the multiple parts of the performer's body acquired by the change amount acquisition unit 120 to identify at least one part among the multiple parts that has the largest amount of change per unit time.

[0074] In one embodiment, the identification unit 140 can identify at least one site with a top large amount of change per unit time when an event is detected in which the reference value calculated by the reference value acquisition unit 130 exceeds the threshold value. The top at least one site with a large amount of change may include N sites.

[0075] (5) Determination unit 150 The determination unit 150 can determine, in the above-mentioned correspondence information stored in the memory unit 100, any one of the specific movements associated with at least one of the parts having the largest amount of change per unit time identified by the identification unit 140, as the movement (detected movement) performed by the performer.

[0076] Furthermore, when the storage unit 100 stores correspondence information that associates each of at least one predetermined specific motion with at least one body part having the largest amount of change among a plurality of body parts of the performer, and an order based on the amount of change of the at least one body part having the largest amount of change, the determination unit 150 can also perform the following operation. Specifically, the determination unit 150 can determine, as the motion (detected motion) performed by the performer, any one of the specific motions that matches at least one body part having the largest amount of change per unit time identified by the identification unit 140 in the correspondence information stored in the storage unit 100 and matches the order based on the amount of change.

[0077] A specific example of the operation performed by the determination unit 150 will be described later.

[0078] (6) Image generation unit 160 The image generating unit 160 can generate a moving image including an animation of a virtual character that normally follows the movement of the body of the performer, using the data (measurement data) related to the body of the performer acquired by the sensor unit 110. For example, the image generating unit 160 can generate a moving image in which the virtual character simply blinks while maintaining a straight face, a moving image in which the virtual character simply looks down while maintaining a straight face, and a moving image in which the virtual character moves its mouth and eyes in accordance with the movement of the face of the performer, etc. The generation of such a moving image can be realized by the image generating unit 160 detecting, for example, a change in the position (coordinates) of a specific part (eyes, etc.) based on the data (measurement data) related to the body of the performer acquired by the sensor unit 110, and drawing the specific part based on such a change. The generation of such a moving image can also be realized by using various rendering techniques well known to those skilled in the art.

[0079] Furthermore, when the determination unit 150 determines that any of the specific actions among at least one predetermined specific action has been executed (when any of the specific actions is determined as a detection action), the image generation unit 160 can perform the following action. Specifically, the image generation unit 160 can generate a moving image including an animation of a virtual character that moves based on the determined detection action. For example, the image generation unit 160 can first hold in advance information on how to change and / or move the virtual character in association with each of the above-mentioned specific actions (A1), (A2), (A3) and / or (B). Next, the image generation unit 160 can generate a moving image including an animation in which the virtual character is changed and / or moved by using information corresponding to the determined detection action among the information held in advance in this way. Moreover, the generation of such a moving image can also be realized by using various rendering techniques well known to those skilled in the art.

[0080] The image generating unit 160 can store in the storage unit 100 a file (such as an MPEG file) storing the moving image thus generated. In an embodiment based on VR (Virtual Reality), the image generating unit 160 can generate an image (moving image) in which a virtual character operating as described above is arranged in a virtual space formed by CG or the like. In an embodiment based on AR (Augmented Reality) or MR (Mixed Reality), the image generating unit 160 can generate an image (moving image) in which a virtual character operating as described above is arranged in a real space.

[0081] (7) Display section 170 The display unit 170 may include, for example, a touch panel and / or a display panel, etc. Such a display unit 170 may play and display a file that contains a moving image stored in the storage unit 100.

[0082] (8) User interface unit 180 The user interface unit 180 may include a touch panel, a mouse, and / or a keyboard, etc. Such a user interface unit 180 may generate information indicating the content of an operation performed by a performer (user).

[0083] (9) Communications Department 190 The communication unit 190 can communicate various information required for distributing and / or receiving an image with the server device 30 via the communication network 10. In particular, when the terminal device 20 operates as the terminal device 20 of a performer (distributing user), the communication unit 190 can transmit moving images stored in the storage unit 100 to the server device 30. When the terminal device 20 operates as the terminal device 20 of a viewing user, the communication unit 190 can receive moving images distributed by the terminal device 20 of the broadcasting user via the server device 30.

[0084] 3-2. Functions of the server device 30 A specific example of the functions of the server device 30 will be described with reference to Fig. 3. For example, some of the functions of the terminal device 20 described above can be used as the functions of the server device 30. Therefore, the reference characters for the components of the server device 30 are shown in parentheses in Fig. 3.

[0085] First, in the above-mentioned "second aspect," the server device 30 may have the same components as the memory unit 200 to the communication unit 290 described in relation to the terminal device 20, respectively, except for the differences described below.

[0086] The sensors included in the sensor unit 210 may be placed facing the performers in a space where the performers perform their acts, such as in a studio where the server device 30 is installed or in another location. Similarly, the display, touch panel, etc. included in the display unit 270 may also be placed facing the performers or near the performers in a space where the performers perform their acts.

[0087] The communication unit 290 can distribute files containing videos stored in the storage unit 200 in association with each performer to the multiple terminal devices 20 via the communication network 10. Each of the multiple terminal devices 20 can execute a specific application (e.g., an application for watching videos) installed therein and transmit a signal (request signal) to the server device 30 requesting the distribution of a desired video. In response to this signal, each terminal device 20 can receive the desired video from the server device 30 via the specific application.

[0088] In addition, various information (such as files storing videos) stored in the memory unit 200 may be stored in one or more other server devices (storage) 30 that can communicate with the server device 30 via the communication network 10.

[0089] On the other hand, in the above-mentioned "first aspect", the sensor unit 210 to the image generating unit 260 used in the above-mentioned "second aspect" can be used as options. In addition to operating as described above, the communication unit 290 can store a file containing a moving image transmitted by each terminal device 20 and received from the communication network 10 in the storage unit 200 and distribute the file to the multiple terminal devices 20.

[0090] On the other hand, in the "third embodiment", the sensor unit 210 to the image generating unit 260 used in the above-mentioned "second embodiment" can be used as options. In addition to operating as described above, the communication unit 290 can store a file containing a moving image transmitted by the studio unit 40 and received from the communication network 10 in the storage unit 200 and distribute the file to a plurality of terminal devices 20.

[0091] 3-3. Functions of Studio Unit 40 3, the studio unit can perform the same operations as the terminal device 20 or the server device 30. However, the communication unit 190 (290) can transmit the moving image generated by the image generation unit 160 (260) and stored in the storage unit 100 (200) to the server device 30 via the communication network 10.

[0092] In particular, the sensors included in the sensor section 110 (210) may be placed facing the performers in a space where the performers perform their acts, such as in a studio where the studio unit 40 is installed or in other locations. Similarly, the display, touch panel, etc. included in the display section 170 (270) may also be placed facing the performers or near the performers in a space where the performers perform their acts.

[0093] 4. Overall Operation of Communication System 1 Next, a specific example of the operation of the entire communication system 1 having the above-mentioned configuration will be described with reference to Fig. 4. Fig. 4 is a flow diagram showing an example of the operation performed in the entire communication system 1 shown in Fig. 1.

[0094] First, in step (hereinafter referred to as "ST") 402, in the first mode, the terminal device 20 (its image generating section 160) can generate a moving image including animation of a virtual character using data (measurement data) related to the performer's body. In the second mode, the server device 30 (its image generating section 260) can execute the same operation. In the third mode, the studio unit 40 can execute the same operation.

[0095] In ST404, in the first aspect, terminal device 20 (communication section 190 thereof) can transmit the generated moving image to server device 30. In the second aspect, server device 30 (communication section 290 thereof) can either not execute ST404 or transmit the generated moving image to another server device 30. In the third aspect, studio unit 40 can transmit the generated moving image to server device 30. Specific examples of the operations executed in ST402 and ST404 will be described later with reference to FIG. 5 etc.

[0096] In ST406, in the first aspect, the server device 30 (the communication unit 290 thereof) can transmit the video received from the terminal device 20 to the other terminal device 20. In the second aspect, the server device 30 (or another server device 30) (the communication unit 290 thereof) can transmit the video received from the terminal device 20 to the other terminal device 20. In the third aspect, the server device 30 (the communication unit 290 thereof) can transmit the video received from the studio unit 40 to the other terminal device 20.

[0097] In ST408, in the first and third aspects, the other terminal device 20 (the display unit 170) can receive the video transmitted by the server device 30 and display it on the display or the like of the terminal device 20 or a display or the like connected to the terminal device 20. In the second aspect, the other terminal device 20 (the display unit 170) can receive the video transmitted by the server device 30 or another server device 30 and display it on the display or the like of the terminal device 20 or a display or the like connected to the terminal device 20.

[0098] In ST410, it is determined whether the operation is to be continued. If it is determined that the operation is to be continued, the process returns to ST402 described above. On the other hand, if it is determined that the operation is not to be continued, the process ends.

[0099] 4 shows the operations shown in ST402 to ST408 being executed sequentially for the sake of simplicity, but in reality, ST402 to ST408 may be executed in parallel with each other.

[0100] 5. Operations related to generation and transmission of moving images performed by the terminal device 20, etc. Next, a specific example of the operation related to generation and transmission of moving images performed by terminal apparatus 20 in ST402 and ST404 among the operations described with reference to Fig. 4 will be described with reference to Fig. 5. Fig. 5 is a flow diagram showing an example of the operation related to generation and transmission of moving images among the operations shown in Fig. 4.

[0101] For simplicity of explanation, the following will focus on the case where the entity that generates the moving image is the terminal device 20 (i.e., the case of the first aspect). However, the entity that generates the moving image may be the server device 30 (the case of the second aspect). Also, the entity that generates the moving image may be the studio unit 40 (the case of the third aspect).

[0102] First, as a premise, in ST502, the terminal device 20 (the storage unit 100) holds correspondence information that associates each of at least one predetermined specific motion with at least one part of the performer's body that has the largest amount of change among multiple parts. The correspondence information stored in the storage unit 100 may be the correspondence information exemplified below. Corresponding information generated by the terminal device 20 Response information generated by experts, engineers, or other users and transmitted from the server device 30, etc., for a fee or free of charge Corresponding information transmitted directly from the terminal device 20 of another user or indirectly via the server device 30, etc. As will be described later, the terminal device 20 is also capable of updating the correspondence information it holds.

[0103] FIG. 6 is a diagram showing an example of correspondence information used in the communication system shown in FIG. 1. The correspondence information shown in FIG. 6 can associate, for example, the top six parts having the largest amount of change per unit time with each specific action. The top six parts can include, for example, the first part to the sixth part. The first part to the sixth part are arranged in order of the largest amount of change per unit time. In other words, the first part has the largest amount of change per unit time, and the sixth part has the smallest amount of change per unit time.

[0104] 6, for example, the specific motion of "angry face" can be associated with the top six body parts with the largest amount of change per unit time. The first to sixth body parts can be the "right end of the mouth," the "left end of the mouth," the "center of the right eye," the "center of the left eye," the "right end of the right eyebrow," and the "left end of the left eyebrow," respectively.

[0105] 6, as another example, the specific motion of "crossing eyes" can be associated with the top six body parts with the largest amount of change per unit time. The first to sixth body parts can be the "right end of the right eye," the "left end of the left eye," the "right end of the right eyebrow," the "left end of the left eyebrow," the "left end of the right eyebrow," and the "right end of the left eyebrow," respectively.

[0106] 6, the correspondence information associates, for example, six body parts with each specific action, but the correspondence information may associate a different number of body parts with a plurality of specific actions.

[0107] Returning to FIG. 5, in ST504, the terminal device 20 (the image generating section 160) can generate a moving image including animation of a virtual character using data (measurement data) related to the body of the performer, as described in relation to ST402. An example of the generated moving image is shown in FIG. 7A. The moving image 700 may include a virtual character (sometimes called an "avatar") 702 of the performer. The virtual character 702 that may be included in the moving image 700 may be placed in a virtual space configured by CG or the like, or may be placed in a real space. In ST504, the virtual character 702 can change in accordance with the movement of the performer's body. For example, in response to the performer blinking facing the terminal device 20, the virtual character 702 included in the moving image 700 also behaves as if blinking. Furthermore, the terminal device 20 (the display section 170) can display the moving image 700 thus generated to present it to the performer.

[0108] 5, also in ST504, the terminal device 20 (the communication unit 190 thereof) can transmit such a video 700 to the server device 30. As a result, the terminal devices 20 (the display units 170 thereof) of other viewing users connected to the server device 30 can also display this video 700 on their display units.

[0109] Next, in ST506, (the change amount acquisition section 120 of) the terminal device 20 can acquire the amount of change in each of the multiple body parts for each unit time. Next, in ST508, (the reference value acquisition section 130 of) the terminal device 20 can acquire a reference value R based on the amount of change in the multiple body parts per unit time, using the amount of change in each body part per unit time acquired in ST506. Here, in one embodiment, the "multiple body parts" from which the reference value R is acquired may be the entirety of multiple predetermined body parts on the performer's body. In another embodiment, the "multiple body parts" from which the reference value R is acquired may be some (e.g., two or more representative body parts) of multiple predetermined body parts on the performer's body.

[0110] Next, in ST510, if the reference value R acquired in ST508 is equal to or less than the threshold value, the terminal device 20 (for example, the reference value acquisition section 130) can estimate that the performer has not performed any specific action. As a result, the process returns to ST504 described above.

[0111] On the other hand, in ST510, if the reference value R acquired in ST508 exceeds the threshold, the terminal device 20 (for example, the reference value acquisition section 130) can estimate that the performer has performed some specific action. As a result, the process proceeds to ST512.

[0112] In ST512, (the identification unit 140 of) terminal device 20 can use the amount of change per unit time of the multiple parts acquired in ST506 described above, for example, to identify the top M parts with the largest amount of change from among these multiple parts. Here, M is a natural number and can be set arbitrarily within the range of 1 or more and N or less. Setting M large (or small) can increase (or decrease) the accuracy of the detection operation determined in ST516 described below.

[0113] Furthermore, in one embodiment, the terminal device 20 (the identification unit 140 thereof) can identify the top M parts with the largest amount of change from among M predetermined parts. As a result, when identifying the top M parts with the largest amount of change, the terminal device 20 (the identification unit 140 thereof) can compare the amount of change of each of the M predetermined parts. Therefore, the terminal device 20 (the identification unit 140 thereof) can more quickly identify the top M parts with the largest amount of change.

[0114] Furthermore, in one embodiment, the terminal device 20 (the identification unit 140 thereof) can rearrange (sort) the M parts thus identified in order according to the magnitude of the amount of change.

[0115] Next, in ST514, (the determination section 150 of) terminal device 20 can determine whether or not the "specific action associated with the M body parts identified in ST512" is present among at least one specific action included in the correspondence information stored in storage section 100. Here, the "specific action associated with the M body parts identified" may include, for example, at least one of the specific actions (1) to (4) shown below. (1) A specific action in which M parts that are completely identical to the identified M parts are associated in the correspondence information. (2) A specific action for which L parts that are completely identical to the top L parts with the greatest amount of change among the identified M parts are associated in the correspondence information (where L is a natural number greater than or equal to 1 and less than or equal to M). (3) A specific motion as shown in (1) above, in which the order according to the magnitude of change in at least one top body part among the M body parts associated with the specific motion matches the order according to the magnitude of change in at least one top body part among the identified M body parts. It can be said that the order of at least one body part associated with this specific motion (3) matches the order of the identified M body parts completely or partially (approximately). (4) A specific motion as shown in (2) above, in which the order according to the magnitude of change in at least one top body part among the L body parts associated with the specific motion matches the order according to the magnitude of change in at least one top body part among the identified M body parts. It can be said that the order of at least one body part associated with this specific motion (4) matches the order of the identified M body parts completely or partially (approximately).

[0116] The specific actions (1) to (4) will be described below using specific examples. Consider a case where the top M sites identified in ST512 are site 1, site 3, and site 2 (M=3). Here, the amount of change of each site decreases in this order. That is, site 1 is the site with the largest amount of change, and site 2 is the site with the smallest amount of change.

[0117] Focus on specific action (1). Among the specific actions included in the correspondence information stored in the storage unit 100, for example, a specific action in which the following parts are associated in the following order may correspond to the specific action (1). ·Site 1, Site 3, Site 2 ·Site 1, Site 2, Site 3 ·Part 2, Part 3, Part 1 ·Site 2, Site 1, Site 3 ·Site 3, Site 1, Site 2 ·Site 3, Site 2, Site 1 ·Site 1, Site 3, Site 2, Site 10 ·Part 1, part 3, part 2, part 21, etc.

[0118] Focus on specific action (2). Among the specific actions included in the correspondence information stored in the storage unit 100, for example, a specific action in which the following parts are associated in the following order may correspond to the specific action (2). ·Part 1, part 3, part 2 (L=3) ·Part 1, part 3, part 2, part 12 (L=3) ·Part 3, part 2, part 1, part 18 (L=3) ·Part 3, part 16, part 1, part 2 (L=3) ·Part 2, part 1, part 3 (L=3) ·Site 2, Site 34, Site 1, Site 18, Site 3 (L=3) ·Part 89, part 3, part 2 (L=3) ·Part 1, part 3, part 8 (L=2) ·Site 1, Site 3, Site 8, Site 12 (L=2) ·Site 1, Site 55, Site 24, Site 3 (L=2) ·Site 22, Site 1, Site 9, Site 3 (L=2) ·Site 1, Site 9, Site 12 (L=1) ·Site 1, Site 8, Site 14, Site 22 (L=1) ·Part 15, part 25, part 1 (L=1), etc.

[0119] Focus on specific action (3). Among the specific movements included in the correspondence information stored in the storage unit 100, for example, a specific movement in which the following parts are associated in the following order may correspond to the specific movement (3). ·Site 1, Site 3, Site 2 ·Site 1, Site 3, Site 2, Site 6 ·Site 1, Site 3, Site 10, Site 2 ·Site 1, Site 2, Site 3 ·Site 1, Site 15, Site 11, Site 3, Site 2, etc.

[0120] Focus on specific action (4). Among the specific movements included in the correspondence information stored in the storage unit 100, for example, a specific movement in which the following parts are associated in the following order may correspond to the specific movement (4). ·Part 1, part 3, part 2 (L=3) ·Part 1, part 3, part 2, part 12 (L=3) ·Part 1, part 3, part 8 (L=2) ·Site 1, Site 3, Site 8, Site 12 (L=2) ·Site 1, Site 55, Site 24, Site 3 (L=2) ·Site 22, Site 1, Site 9, Site 3 (L=2) ·Site 1, Site 9, Site 12 (L=1) ·Site 1, Site 8, Site 14, Site 22 (L=1) ·Part 15, part 25, part 1 (L=1), etc.

[0121] In ST514, if the terminal device 20 (the decision unit 150 thereof) determines that there is no "specific action associated with the identified M body parts" as described above, the process returns to ST504 described above. In this case, in another embodiment, the terminal device 20 can increase the number of body parts from M to (M+α) and sequentially execute ST512 and ST514 again. Here, α is a natural number equal to or greater than 1.

[0122] On the other hand, in ST514, if the terminal device 20 (the determination unit 150) determines that there is a "specific action associated with the identified M parts" or a "specific action associated with the identified (M+α) parts" as described above, the process proceeds to ST516. In ST516, the terminal device 20 (the determination unit 150) can determine such a "specific action associated with the identified M parts" or the "specific action associated with the identified (M+α) parts" as a specific action (detected action) performed by the performer. Note that, in the case where a plurality of such "specific actions associated with the identified M parts" or "specific actions associated with the identified (M+α) parts" are included in the correspondence information, the terminal device 20 (the determination unit 150) can determine the most suitable specific action among the plurality of specific actions as the detected action. The most suitable specific action may be, for example, a specific action among the plurality of specific actions that is associated with the M parts identified in ST512 in the most approximate order.

[0123] Next, in ST518, the terminal device 20 (the image generating section 160) can generate a moving image including an animation of a virtual character moving based on the detected motion determined in ST516. If the detected motion determined in ST516 is, for example, an "angry face" as illustrated in FIG. 6, the terminal device 20 (the image generating section 160) can generate a moving image 700 in which the facial expression of the virtual character 702 is changed to an angry facial expression as illustrated in FIG. 7B. Also, if the detected motion determined in ST516 is, for example, a "cross-eyes" as illustrated in FIG. 6, the terminal device 20 (the image generating section 160) can generate a moving image 700 in which the facial expression of the virtual character 702 is changed to a facial expression called a "cross-eyes" (an expression in which both eyes are represented by symbols and the mouth is represented by a rectangle) as illustrated in FIG. 7C. Furthermore, the terminal device 20 (the display section 170) can display the moving image 700 thus generated to present it to the performer.

[0124] Also in ST518, the terminal device 20 (the communication unit 190 thereof) can transmit the video 700 thus generated to the server device 30. As a result, the terminal devices 20 (the display units 170 thereof) of other viewing users connected to the server device 30 can also display this video 700 on their display units.

[0125] After this, in one embodiment, processing may return to ST504 described above. In another embodiment, the process may return to ST502 described above as illustrated in Fig. 5. In ST502, the terminal device 20 may update the correspondence information stored in the storage unit 100.

[0126] For example, when the video 700 generated based on the detected action determined in ST516 is displayed by the terminal device 20 (on its display unit 170) in ST518, the performer or the like monitoring this video 700 may feel that the detected action (determined specific action) is inappropriate. In this case, the performer or the like can specify the specific action that the performer or the like originally intended via the terminal device 20 (on its user interface unit 180).

[0127] When such a designation is made by the performer or the like, the terminal device 20 (the image generating section 160 and the display section 170) can stop generating the video 700 based on the detection operation. Furthermore, the terminal device 20 can generate and display the video 700 based on data (measurement data) related to the performer's body, as in ST504 described above. In this way, the video 700 reflecting the intentions of the performer or the like can be transmitted to the server device 30 or the like.

[0128] Furthermore, when such a designation is made by a performer or the like, the terminal device 20 can update the corresponding information stored in the storage unit 100 based on information regarding such designation. For example, consider a case where an actor or the like specifies an "angry face" as an appropriate specific motion, even though the detected motion is a "cross eye" as exemplified in FIG. 6. In this case, the terminal device 20 can update the correspondence information exemplified in FIG. 6 so as to associate the "right end of the right eye," the "left end of the left eye," the "right end of the right eyebrow," the "left end of the left eyebrow," the "left end of the right eyebrow," and the "right end of the left eyebrow" with the first to sixth parts, respectively, for the specific motion of an "angry face." Alternatively, the terminal device 20 can update the correspondence information exemplified in FIG. 6 so as to associate the top M parts themselves identified in ST512, or at least some parts of the top M parts identified in ST512, with the specific motion of an "angry face."

[0129] Note that constantly or frequently performing the operations shown in ST512 to ST514 may lead to an increase in power consumption of the terminal device 20, a drain on the battery of the terminal device 20, and / or a decrease in the processing speed of the terminal device 20. In order to at least partially suppress such a possibility, in the example shown in Fig. 5, when the reference value R based on the amount of change in a plurality of parts exceeds a threshold, the terminal device 20 performs the operations shown in ST512 to ST514 based on the presumption that the performer may have performed some specific action. In other words, in the example shown in Fig. 5, when the reference value R based on the amount of change in a plurality of parts is equal to or less than a threshold, the terminal device 20 avoids the operations shown in ST512 to ST514 based on the presumption that the performer may not have performed any specific action.

[0130] In another embodiment, the terminal device 20 can always or frequently perform the operations shown in ST 512 to ST 514. In this case, the operations shown in ST 508 and ST 510 can be omitted.

[0131] 5, the change amounts per unit time of the multiple parts used in ST512 are the change amounts per unit time of the multiple parts acquired in ST506. However, the change amounts per unit time of the multiple parts used in ST512 may be the sum of at least one of the change amounts exemplified below. The amount of change per unit time of the multiple body parts acquired in ST506 (for convenience, this unit time will be referred to as the "reference unit time" below). -Changes in multiple body parts obtained in at least one unit time that occurred after the reference unit time -Changes in multiple body parts obtained during at least one unit time that occurred before the reference unit time

[0132] 6. How to update compatibility information Next, a specific example of a method for updating the correspondence information stored in the terminal device 20 or the like will be described. The correspondence information stored in the terminal device 20 or the like can be updated, for example, at the following three times. (1) At the time of initial use (2) First use each day (3) When an outlier occurs

[0133] First, the above (1) "initial use" will be described. When using the service provided by the communication system 1, the performer A can use correspondence information prepared in advance by the communication system 1 (for example, correspondence information updated by another performer). However, when the terminal device 20 of the performer A uses correspondence information updated by another performer, there is a possibility that the terminal device 20 may determine an unintended movement (facial expression) of the performer A as the detected movement. Therefore, it is important for the terminal device 20 to customize the correspondence information for each performer (user). The reason for such customization is that even if the same facial expression is to be made, the photographing condition of the face, the parts of the face that have a large amount of change, and the order based on the amount of change in these parts are different for each performer.

[0134] For example, when performer A makes a "right wink" facial expression, the top multiple (e.g., three) parts of performer A with the largest amount of change may be, in descending order of the amount of change, for example, (i) the right eyebrow, (ii) the upper eyelid of the right eye, and (iii) the lower eyelid of the right eye. However, when performer B makes the same "right wink" facial expression, the top multiple (e.g., three) parts of performer B with the largest amount of change may be, in descending order of the amount of change, for example, (i) the upper eyelid of the right eyebrow, and (iii) the lower eyelid of the right eye. In this case, if the terminal device 20 of performer A uses the correspondence information updated by the terminal device 20 of performer B as is, there is a high possibility that even if performer A makes a right wink, it will not be able to determine that the right wink is a detected motion.

[0135] Therefore, when each performer uses the service provided by the communication system 1 for the first time, the terminal device 20 can register his / her own facial expression for each specific action (special facial expression, etc.) reflected in the avatar object (this becomes teacher information for the target). Specifically, for example, the terminal device 20 displays instructions regarding the specific action for each major specific action (special facial expression, etc.), and the performer can perform the specific action according to the instructions. This allows the terminal device 20 to generate (update) correspondence information that associates each specific action with at least one body part that has a large amount of change corresponding to the specific action. This correspondence information can be used as described above in ST514 in FIG. 5. Furthermore, the terminal device 20 can generate a learning model corresponding to the performer using the facial expression (question information) and the specific action (answer information) made by the performer according to the instructions as teacher information. Such a learning model can operate to output a specific action when the facial expression of the performer is input, for example.

[0136] Next, the above (2) “First use each day” will be explained. Even if the same performer intends to perform the same specific movement, the parts of the body (face, etc.) with the greatest amount of change and the ranking based on the amount of change of these parts detected by the terminal device 20 may differ from day to day. This may be due to various factors including the performer's hairstyle, fatigue, shooting environment, and the positional relationship between the camera and the body (face, etc.). Therefore, when each performer first uses the terminal device 20 each day (e.g., every morning), he / she can register his / her own facial expression for each specific action (special facial expression, etc.) reflected in the avatar object (this becomes teaching information for the target, as in the case of (1) above).

[0137] In one embodiment, the terminal device 20 can perform learning similar to that performed in (1) above. In a more preferred embodiment, the terminal device 20 can not only perform learning by simply associating facial expressions made by the performer with specific actions to be output (special facial expressions, etc.), but also perform learning by using the following elements as teacher information (question information and answer information): (i) Facial expressions made by the performer (question information) (ii) Performer's hairstyle (Question information) (iii) Fatigue level of the performer (question information) The degree of fatigue may be determined, for example, from the amount of change in facial features (the part to be noted does not change, but the range between maximum and minimum becomes narrower. In other words, the greater the degree of fatigue, the smaller the difference between the maximum and minimum values ​​of the amount of change in the part to be noted, whereas the smaller the degree of fatigue, the larger the difference between the maximum and minimum values ​​of the amount of change in the part to be noted). (iv) Shooting environment, e.g., the positional relationship between the camera and the body (face, etc.) (question information) (v) Situation and / or purpose Depending on the situation or purpose, the performer can decide "I will not use this face this time" and exclude the specific action from the judgment target. This makes it clear which facial expressions the performer does not want to use due to the design of the character. The situation or purpose can mean, for example, what kind of virtual space it is (karaoke, stage, live concert venue, etc.). (vi) Specific movement (answer information)

[0138] Therefore, as the performer continues to use the service, the determination of the performer as a target converges. In other words, as the performer continues to use the service, the specific action intended by the performer becomes more likely to be reflected in the avatar object.

[0139] The learning in (2) above may be performed by an actor operating the terminal device 20 when an unintended specific action (such as a special facial expression) is activated by the avatar object. For example, when the terminal device 20 activates a special facial expression A that is unintended by the actor on the avatar object, the actor may clearly set a priority order by operating the terminal device 20 so that the avatar object activates a special facial expression B that the actor originally wanted to activate. In this case, the terminal device 20 can store learning information for determining that the special facial expression B is given priority even though the special facial expressions A and B are determined to be similar (the process related to this update is the same process as the character conversion of a PC keyboard is customized as it is used continuously. Therefore, a detailed description of the process is omitted here). The generation (update) of the correspondence information is not limited to being performed at the first use of each day, and may be performed, for example, when it is determined that the interval between the use of the video distribution application on the terminal 20 is longer than a certain time.

[0140] Next, the above (3) “When an outlier occurs” will be explained. It is assumed that the terminal device 20 is set to make the facial expression "A" when the top three parts of the face with the greatest amount of change are, in order of the amount of change, (i) the upper lip, (ii) the lower lip, and (iii) the eyebrows (whether the performer pronounces "Ahhh" or "Ah-ahh", the same facial expression "A" is reflected in the avatar object). Here, when the performer pronounces "Oh...", the top three parts of the face with the greatest amount of change are, in order of the amount of change, (i) the upper lip, (ii) the lower lip, and (iii) the eyebrows. (ii) Lower lip, and it is determined that there is no special expression to be assigned.

[0141] In this case, if the performer inputs "default" (normal facial expression) in response to a question posed by the terminal device 20 via the user interface, the facial expression determination ends (the terminal device 20 waits until the next threshold is exceeded). On the other hand, the performer can register a new special facial expression in response to a question posed by the terminal device 20 via the user interface. In this case, when the top three parts of the face with the greatest amount of change are (i) the upper lip, (iii) the eyebrows, and (ii) the lower lip, the terminal device 20 can activate a new special facial expression (Oh no~).

[0142] This determination can be made as an "outlier" by evaluating the input value and the distance function (sum of squares of each element) for each evaluation function. For example, there is an evaluation function fA(xi) for facial expression A and an evaluation function fB(xi) for facial expression B (there is also a default facial expression fDefault(xi) and the like), and a facial expression that is sufficiently far from either function fA() or fB() and has not been excluded from the determination as fDefault() can be proposed to the performer as a new facial expression by the terminal device 20. Conversely, the performer can set a threshold value, for example, a value in the range of 0 to 1, by operating a slide or the like displayed on the user interface, to determine that a determination for a stable target is prioritized. For example, consider a situation in which a smile fA, a small laugh fB, and a medium laugh fC are registered as targets in the initial state. For example, when the terminal device 20 detects a phenomenon in which the amount of change in any part is significantly off during distribution of a video, the terminal device 20 can suggest to the performer via the user interface that he or she should "register a new facial expression state." This allows the performer to register "big laugh." Thereafter, the terminal device 20 can add the new target and evaluation function fD() to the judgment loop.

[0143] As the performer continues to use the terminal device 20 that implements the above (1) to (3), the activation of specific actions (special facial expressions, etc.) by this terminal device 20 can be customized to individual users and converge.

[0144] The update of the algorithm (evaluation function and data table) basically converges as the terminal device 20 recursively saves the evaluation function and data table (corresponding information) based on the performer's operations. A new judgment function can be registered by the performer using the service. This can be called "autonomous learning" because the developer does not need to write a new algorithm, but it is based on the teacher information provided by the performer. This evaluation function and data set can be called the learning result for each performer, and since it is small in size and does not contain personal information, it can be collected by the service operator and used for averaging and optimizing the algorithm.

[0145] 7. Variations At least one specific action predefined in the communication system 1 may include a specific action identified by a first action performed by a performer and a second action performed by the performer at a longer time interval after the first action. For example, a specific action of clapping hands may be identified by a first action in which the palms of both hands are placed at a distance from each other and a second action in which the palms of both hands come into contact with each other after the first action.

[0146] In this case, the terminal device 20 can associate at least one body part with a top amount of change in the first motion and at least one body part with a top amount of change in the second motion with respect to the specific motion of "clap" in the correspondence information stored in the storage unit 100. Furthermore, when the reference value R exceeds the threshold (ST510), the terminal device 20 can first identify X1 body parts with a top amount of change in the first unit time, which may include one or more unit times (ST512 applies mutatis mutandis). Next, the terminal device 20 can determine whether the first motion corresponding to the top X1 body parts thus identified exists in the correspondence information (ST514 applies mutatis mutandis). Here, X1 is any natural number. Furthermore, the terminal device 20 can identify X2 body parts with a top amount of change in the second unit time, which occurs after the first unit time and may include one or more unit times (ST512 applies mutatis mutandis). Here, X2 is also any natural number. Next, the terminal device 20 can determine whether or not the second motion corresponding to the top X2 body parts thus identified exists in the correspondence information (ST514 applies mutatis mutandis). If the first motion corresponding to the top X1 body parts exists in the correspondence information and the second motion corresponding to the top X2 body parts exists, the terminal device 20 can determine a specific motion (e.g., clapping) including the first motion and the second motion as the detected motion (ST516 applies mutatis mutandis).

[0147] The correspondence information may be, for example, an algorithm including a plurality of evaluation functions prepared for each specific motion to be judged, instead of a data table as shown in Fig. 6. This algorithm may be treated as information that associates each specific motion with at least one part having a large displacement.

[0148] In addition, in various embodiments described with reference to Figures 4 and 5, the technology disclosed in the present application is applied to the generation of moving images, as an example. However, the technology disclosed in the present application is applicable to various applications including email, messengers, word processors, and the like, and to various services such as websites and SNS. In this case, in ST518, the terminal device 20 can display emojis and / or emoticons corresponding to the determined detection action. Furthermore, the technology disclosed in this application is also applicable to game applications and game services. In this case, in ST518, the terminal device 20 can control the action of the game object based on the determined detected action.

[0149] Furthermore, in various embodiments described with reference to Figures 4 and 5 etc., a case has been described in which the terminal device 20 facing the performer (which may alternatively be the server device 30 or the studio unit 40; hereafter, these are collectively referred to as "terminal device 20, etc.") performs all processing from obtaining data on the performer's body (measurement data) to transmitting video. However, although the terminal device 20, etc. facing the performer needs to obtain data on the performer's body, subsequent operations may be performed by a device other than the terminal device 20, etc.

[0150] In one example, after the terminal device 20 etc. acquires data on the body of the performer, the terminal device 20 etc. executes at least one of steps ST506, ST508, ST510, ST512, ST514, and ST516, and steps other than these steps may be executed by one or more other devices connected to the terminal device 20 etc. For example, the steps up to acquisition of data on the body of the performer may be executed by the terminal device 20 etc., and steps ST506 to ST518 (up to generation of the moving image) may be executed alone or in a shared manner by one or more other devices. Alternatively, steps up to ST506 may be executed by the terminal device 20 etc., and steps ST508 to ST518 (up to generation of the moving image) may be executed alone or in a shared manner by one or more other devices. This makes it possible to reduce the computational resources and power consumption required by the terminal device 20 etc. facing the performer. In either case, the terminal device 20 etc. needs to transmit data / information obtained by executing intermediate steps and required for the execution of subsequent steps to one or more other devices.

[0151] Incidentally, the above-mentioned "one or more other devices" can include the following devices in each of the first to third aspects. Server device 30 and / or viewing user's terminal device 20, etc. (in the first and third aspects) Other server devices 30 and / or viewing user terminal devices 20, etc. (in the case of the second aspect) Here, the above-mentioned "one or more other devices" includes the viewing user's terminal device 20, and the method in which the viewing user's terminal device 20 executes ST518 (up to generating the video) is sometimes referred to as a "client rendering" method.

[0152] 4 and 5, the correspondence information not only associates each of at least one specific motion with at least one body part having the largest amount of change among a plurality of body parts of the performer, but also associates each of at least one specific motion with an order based on the amount of change in the at least one body part having the largest amount of change. However, in another embodiment, the correspondence information may associate each of at least one specific motion with at least one body part having the largest amount of change among a plurality of body parts of the performer, but may not associate each of at least one specific motion with an order based on the amount of change in the at least one body part having the largest amount of change.

[0153] In this case, the terminal device 20 does not need to execute ST512. In ST514, the terminal device 20 can determine whether or not a "specific action associated with the M body parts identified in ST512" (or a "specific action associated with the (M+α) body parts identified") is present among at least one specific action included in the correspondence information stored in the storage unit 100. Here, the "specific action associated with the M body parts identified" may include, for example, at least one of the above-mentioned specific action (1) and specific action (2).

[0154] Furthermore, the various embodiments described above can be used in combination with each other unless a contradiction occurs.

[0155] As described above, in various embodiments, at least one top body part with a large amount of change per unit time is identified based on data related to the body of the performer (user), and then a specific action previously associated with at least one top body part can be detected as a specific action performed by the performer. This makes it possible to easily and quickly detect a specific action performed by the performer, thereby reducing computational resources, power consumption, and the like.

[0156] Furthermore, when a reference value based on the amount of change per unit time of multiple parts of the performer exceeds a threshold value, it is possible to identify at least one top part having a large amount of change per unit time, and to detect a specific action previously associated with the at least one top part, thereby further reducing computational resources and power consumption.

[0157] Thus, various embodiments provide a novel way of displaying images based on the movements of an actor.

[0158] 8. Notes The computer program of the first aspect can be executed by at least one processor to cause the processor to function as follows: "to retain information associating each of at least one specific movement with at least one top part of a plurality of parts of an actor's body having the greatest amount of change; to obtain a reference value based on the amount of change in the plurality of parts of the actor's body per unit time using measurement data on the actor's body; and, when an event is detected in which the reference value exceeds a threshold value, to use the measurement data to identify at least one top part of the actor's body having the greatest amount of change per unit time; and, using the information, to determine, as a detected movement, any one of the at least one specific movement associated with the identified at least one top part of the body having the greatest amount of change per unit time." In the computer program of the second aspect, the configuration of the first aspect described above can be adopted, in which "the information causes the processor to function such that each of the at least one specific movement is associated with at least one top part of a plurality of parts of the performer's body having the greatest amount of change and with an order based on the magnitude of the amount of change of the at least one top part having the greatest amount of change, and the information is used to determine, as a detected movement, any one of the at least one specific movement that matches the identified at least one top part having the greatest amount of change per unit time and the order based on the magnitude of the amount of change." In the computer program of the third aspect, in the first or second aspect, a configuration can be adopted in which "the processor is caused to function as follows: using the measurement data to identify at least one top body part of the performer having the largest amount of change per unit time, determining, from among the at least one specific movement, a specific movement associated with the identified at least one top body part having the largest amount of change per unit time as a first detection movement, and then determining, from among the at least one specific movement, a specific movement associated with the identified at least one top body part having the largest amount of change per unit time as a second detection movement, and determining a final detection movement based on the first detection movement and the second detection movement." In a fourth aspect of the present invention, the computer program according to any one of the first to third aspects further comprises the step of: "when the plurality of sites is N sites, the reference value is: It is calculated using the formula TIFF0007662224000003.tif10136, where x i is the amount of change per unit time of the i-th part among multiple parts of the performer's body. In the computer program of the fifth aspect, in any one of the first to fourth aspects, a configuration can be adopted in which "each of the at least one specific motion is represented using at least one part having the largest amount of change associated with the specific motion." In the computer program of the sixth aspect, the configuration of the fifth aspect described above can be adopted, where "each of the at least one part having the largest amount of change is selected from the group including eyes, eyebrows, nose, mouth, ears, chin, cheeks, neck, shoulders, hands, and chest." In the computer program of the seventh aspect, a configuration can be adopted in any one of the first to fourth aspects where "the processor is caused to function to generate an image based on the determined detection operation." The computer program according to the eighth aspect may employ the configuration of the seventh aspect above, where "the image includes an emoji, an avatar object, and / or a game object." The computer program of the ninth aspect can be executed by at least one processor to cause the processor to function as follows: "to store information associating each of at least one specific movement with at least one top part of a plurality of parts of an actor's body having the greatest amount of change; to obtain a reference value based on the amount of change in the plurality of parts of the actor's body per unit time using measurement data on the actor's body received from the actor's terminal device; and, when an event is detected in which the reference value exceeds a threshold value, to identify at least one top part of the actor's body having the greatest amount of change per unit time using the measurement data; and, using the information, to determine, as a detected movement, any one of the at least one specific movement associated with the identified at least one top part of the body having the greatest amount of change per unit time." In the computer program of the tenth aspect, the configuration of the ninth aspect described above can be adopted, in which "the processor is caused to function to receive the measurement data from the performer's terminal device via a communication line." In the computer program of the 11th aspect, in any of the first to tenth aspects, a configuration can be adopted in which "the at least one processor includes a central processing unit (CPU), a microprocessor, and / or a graphics processing unit (GPU)." In the computer program of the 12th aspect, in any of the first to 11th aspects, a configuration can be adopted in which "the at least one processor is installed in a smartphone, a tablet, a mobile phone, or a personal computer." The method of the 13th aspect can be "a method executed by at least one processor that executes computer-readable instructions, wherein the processor executes the instructions to store information correlating each of at least one specific movement with at least one top part of a plurality of parts of an actor's body having the greatest amount of change, obtain a reference value based on the amount of change in the plurality of parts of the actor's body per unit time using measurement data on the actor's body, and when an event is detected in which the reference value exceeds a threshold value, identify at least one top part of the actor's body having the greatest amount of change per unit time using the measurement data, and determine, as a detected movement, any one of the at least one specific movement that is associated with the identified at least one top part of the body having the greatest amount of change per unit time using the information." In the method of the 14th aspect, the configuration of the 13th aspect above can be adopted, in which "the at least one processor includes a central processing unit (CPU), a microprocessor, and / or a graphics processing unit (GPU)." In the method of the 15th aspect, the configuration of the 13th aspect or the 14th aspect can be adopted, in which "the at least one processor is mounted on a smartphone, a tablet, a mobile phone, a personal computer, or a server device." A server device according to a sixteenth aspect of the present invention includes at least one processor, and the processor executes computer-readable instructions to store information associating each of at least one specific action with at least one body part of a performer that has a top amount of change among a plurality of body parts of the performer, obtains a reference value based on the amount of change per unit time of the plurality of body parts of the performer using measurement data on the performer's body, and when an event in which the reference value exceeds a threshold is detected, identifies at least one body part of the performer that has a top amount of change per unit time using the measurement data, and and determining, as the detected motion, one of the at least one specific motions associated with at least one of the identified body parts having the largest amount of change per unit time, using the above-mentioned method. In the server device of the seventeenth aspect, the configuration of the sixteenth aspect above, "receiving the measurement data from the performer's terminal device via a communication line" can be adopted. In the server device of the 18th aspect, the configuration of the 16th aspect or the 17th aspect can be adopted, in which "the at least one processor includes a central processing unit (CPU), a microprocessor, and / or a graphics processing unit (GPU)." [Explanation of symbols]

[0159] 20(20A~20C) Terminal device 30(30A~30C) Server equipment 40 (40A and 40B) Studio Unit 100(200) Storage section 110(210) Sensor part 120(220) Change amount acquisition section 130(230) Reference value acquisition section 140(240) Identification section 150(250) Decision Section 160(260) Image Generation Section 170(270) Display section 180(280) User Interface Section 190(290) Communications Department

Claims

1. When executed by at least one processor, Using measurement data on the performer's body, an amount of change in a plurality of parts of the performer's body per unit time is obtained; using the amount of variation to identify at least one body part of the performer; A computer program that causes the processor to function in such a way that, using information correlating each of at least one specific movement with at least one part of a performer's body that has the greatest amount of change, and an order based on the magnitude of the change in the at least one part that has the greatest amount of change, the processor determines, as a detected movement, one of the at least one specific movement that is associated with at least one part of a performer's body that has the greatest amount of change per identified unit time and an order based on the magnitude of the change, the at least one specific movement.

2. When executed by at least one processor, Using measurement data on the performer's body, an amount of change in a plurality of parts of the performer's body per unit time is obtained; using the amount of variation to identify at least one body part of the performer; Using information associating each of at least one specific motion with at least one part of a plurality of parts of the performer's body, determine one specific motion associated with the identified at least one part of the at least one specific motion as a first detected motion, and then determining, as a second detection motion, one of the at least one specific motions associated with at least one body part having a large amount of change per unit time that has been identified, from the at least one specific motion; 23. A computer program product comprising: a computer program product causing the processor to: determine a final detection action based on the first detection action and the second detection action.

3. 3 . The computer program product according to claim 1 , wherein each of the at least one specific motions is expressed using at least one body part having a large amount of change associated with the specific motion.

4. 4. The computer program of claim 3, wherein each of the at least one area with the greatest amount of change is selected from the group including eyes, parts of eyes, eyebrows, parts of eyebrows, nose, parts of nose, mouth, parts of mouth, ears, parts of ears, chin, parts of chin, cheeks, parts of cheeks, neck, parts of neck, shoulders, parts of shoulders, hands, parts of hands, chest, and parts of chest.

5. A computer program product as claimed in any preceding claim, configured to cause the processor to: generate an image based on the determined detection movement.

6. The image is Contains still or moving images, or 6. The computer program product of claim 5, further comprising an emoji, an avatar object, and / or a game object.

7. 7. A computer program product as claimed in claim 1, further comprising: a processor configured to receive the measurement data from a terminal device of the performer.

8. 1. A method performed by at least one processor executing computer readable instructions, comprising: The processor executes the instructions to Using measurement data on the performer's body, an amount of change in a plurality of parts of the performer's body per unit time is obtained; using the amount of variation to identify at least one body part of the performer; A method comprising: using information correlating each of at least one specific movement with at least one part of a performer's body having the greatest amount of change and an order based on the magnitude of the amount of change of the at least one part having the greatest amount of change, the method determining, as a detected movement, one of the at least one specific movement that is associated with at least one part of a performer's body having the greatest amount of change per identified unit time and an order based on the magnitude of the amount of change.

9. 1. A method performed by at least one processor executing computer readable instructions, comprising: The processor executes the instructions to Using measurement data on the performer's body, an amount of change in a plurality of parts of the performer's body per unit time is obtained; using the amount of variation to identify at least one body part of the performer; Using information associating each of at least one specific motion with at least one part of a plurality of parts of the performer's body, determine one specific motion associated with the identified at least one part of the at least one specific motion as a first detected motion, and then determining, as a second detection motion, one of the at least one specific motions associated with at least one body part having a large amount of change per unit time that has been identified, from the at least one specific motion; determining a final sensing operation based on the first sensing operation and the second sensing operation.

10. At least one processor; The processor executes the computer readable instructions to: Using measurement data on the performer's body, an amount of change in a plurality of parts of the performer's body per unit time is obtained; using the amount of variation to identify at least one body part of the performer; A device characterized by using information correlating each of at least one specific movement with at least one part of a performer's body that has the largest amount of change, and an order based on the magnitude of the change in the at least one part that has the largest amount of change, to determine, as a detected movement, one of the at least one specific movement that is associated with at least one part of a performer's body that has the largest amount of change per identified unit time and an order based on the magnitude of the change.

11. At least one processor; The processor executes the computer readable instructions to: Using measurement data on the performer's body, an amount of change in a plurality of parts of the performer's body per unit time is obtained; using the amount of variation to identify at least one body part of the performer; Using information associating each of at least one specific motion with at least one part of a plurality of parts of the performer's body, determine one specific motion associated with the identified at least one part of the at least one specific motion as a first detected motion, and then determining, as a second detection motion, one of the at least one specific motions associated with at least one body part having a large amount of change per unit time that has been identified, from the at least one specific motion; determining a final sensing operation based on the first sensing operation and the second sensing operation.

12. 12. Apparatus according to claim 10 or 11, which receives the measurement data from a terminal device of the performer.

Images