Display device, display method, and program

Abstract

To more appropriately display a user by an avatar.SOLUTION: The display device includes an acquisition unit configured to acquire pattern information indicating which preset motion pattern is specified for each motion of a user appearing in a captured moving image obtained by imaging the user, and an output control unit configured to set the motion pattern indicated in the pattern information to a motion of an avatar of the user, in which the output control unit sets a shorter interval to the avatar for a motion pattern having a larger number of times the motion is specified in the pattern information.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a display device, a display method, and a program. [Background technology]

[0002] The use of avatars to represent users in virtual spaces has become widespread. Patent Document 1 discloses a technology in which, when generating an avatar, a terminal device acquires image data related to the user, calculates a variation based on the acquired data, acquires a score related to an emotion based on the variation, selects an emotion based on the score, and generates an avatar that reflects the selected emotion. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7162737 Summary of the Invention [Problem to be solved by the invention]

[0004] Each user has their own unique gestures, facial expressions, etc. With the technology of Patent Document 1, it is difficult to express the user's unique gestures, facial expressions, etc. in an avatar. For this reason, there is a demand for a more appropriate way to display users in an avatar.

[0005] In view of the above-described problems, the present disclosure aims to provide a display device, a display method, and a program that more appropriately display a user as an avatar. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems and achieve the objectives, the display device according to the present disclosure includes an acquisition unit that acquires pattern information indicating which of the pre-set movement patterns each of the user's movements captured in a video of the user is identified as, and an output control unit that sets the movement of the movement pattern indicated in the pattern information as the movement of the user's avatar, and the output control unit sets the movement of the avatar at a shorter interval the more times the movement is identified in the pattern information for that movement pattern.

[0007] In order to solve the above-mentioned problems and achieve the objectives, the display method according to the present disclosure includes a step of acquiring pattern information indicating which pre-set movement pattern each of the user's movements captured in a video of the user is identified as, and a step of setting the movements of the movement pattern indicated in the pattern information as the movements of the user's avatar, wherein in the step of setting the movements of the user's avatar as the movement pattern that is identified more frequently in the pattern information, the shorter the intervals at which the movements are set for the avatar.

[0008] In order to solve the above-mentioned problems and achieve the objectives, the program of the present disclosure is a program that causes a computer to execute a display method, including the steps of acquiring pattern information indicating which pre-set movement pattern each of the user's movements captured in a video of the user is identified as, and setting the movements of the movement pattern indicated in the pattern information as the movements of the user's avatar, wherein in the step of setting the movements of the user's avatar, the more times the movement is identified in the pattern information for a movement pattern, the shorter the intervals at which the movement is set for the avatar. [Effects of the Invention]

[0009] According to the present invention, users can be more appropriately represented by avatars. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of an information processing system according to the first embodiment. [Figure 2] FIG. 2 is a block diagram showing an example of the configuration of the display device according to the first embodiment. [Figure 3] FIG. 3 is a diagram illustrating a method for capturing an image of a user according to the first embodiment. [Figure 4] FIG. 4 is a block diagram showing an example of the configuration of the information processing device according to the first embodiment. [Figure 5] FIG. 5 is a diagram illustrating an example of a user action pattern table according to the first embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of a user action pattern table according to the first embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of a user's emotion estimation table according to the first embodiment. [Figure 8] FIG. 8 is a diagram illustrating an example of the operation of the information processing system according to the first embodiment. [Figure 9] FIG. 9 is a flowchart showing an example of processing of the information processing system according to the first embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of a user action pattern table according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. However, the present invention is not limited to these embodiments. Furthermore, the components of these embodiments include those that are easily replaceable by those skilled in the art, or those that are substantially identical.

[0012] [First embodiment] (Information Processing System) An example of the configuration of an information processing system according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the configuration of an information processing system according to the first embodiment.

[0013] 1, the information processing system 1 includes a display device 10 and an information processing device 50. The display device 10 and the information processing device 50 are connected via a network N so as to be able to communicate with each other.

[0014] The information processing system 1 provides a virtual space. For example, the information processing system 1 generates an avatar of the user U in the virtual space. For example, the information processing system 1 acquires and analyzes the user U's unique movements in advance and sets the movements as an avatar of the user U that has been generated in advance in the virtual space. Specifically, the display device 10 captures an image of the user U and transmits the captured video to the information processing device 50. The information processing device 50 acquires the captured video and analyzes the movements of the user U. The display device 10 sets the analysis result obtained by the information processing device 50 as an avatar of the user U and displays the avatar.

[0015] (display device) An example of the configuration of the display device according to the first embodiment will be described with reference to Fig. 2. Fig. 2 is a block diagram showing an example of the configuration of the display device according to the first embodiment.

[0016] The display device 10 displays various types of information. For example, the display device 10 displays an avatar of a user U. In this embodiment, the display device 10 may be, for example, a mobile terminal such as a personal computer, a smartphone, or a tablet terminal, or a terminal having a mirror, such as a smart mirror.

[0017] As shown in FIG. 2, the display device 10 includes an imaging unit 12, an output unit 14, an input unit 16, a storage unit 18, a communication unit 20, and a control unit 30.

[0018] The imaging unit 12 captures an image of the user U. For example, the imaging unit 12 captures an image of the user U and generates a still image or a moving image. The imaging unit 12 is, for example, a camera. In this embodiment, the imaging unit 12 captures, for example, an image of the entire user U. For example, the imaging unit 12 zooms in to capture an image of the user U. For example, the imaging unit 12 captures the user U's movements during conversation. For example, the imaging unit 12 captures the user U's facial expressions and gestures during conversation. For example, if the user U is happy, the user U's movements during conversation may include a happy facial expression and movements such as jumping.

[0019] The output unit 14 outputs various types of video and audio. The output unit 14 displays, for example, an avatar of the user U. The output unit 14 is, for example, a liquid crystal display, an organic EL (Electro-Luminescence) display, a speaker, or the like.

[0020] The input unit 16 accepts various operations for the display device 10. The input unit 16 is realized by a keyboard, a switch, a button, a touch panel, a microphone, or the like.

[0021] The storage unit 18 stores information such as the contents of calculations performed by the control unit 30 and programs. The storage unit 18 is configured with, for example, a main storage device such as a RAM (Random Access Memory) and a ROM (Read Only Memory), and a storage device such as an SSD (Solid State Drive) or an HDD (Hard Disk Drive).

[0022] The communication unit 20 is a communication unit for performing wired communication, wireless communication, etc. The communication unit 20 executes communication between the display device 10 and the information processing device 50, for example.

[0023] The control unit 30 is a computing device and includes, for example, a computing circuit such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). The control unit 30 includes an acquisition unit 32, an imaging control unit 34, and an output control unit 36. The control unit 30 implements the imaging control unit 34 and the output control unit 36 ​​and performs their processing by reading and executing a program (software) from the storage unit 18. Note that the control unit 30 may execute these processes using a single CPU, or may be provided with multiple CPUs and execute the processes using the multiple CPUs. Furthermore, at least a portion of the processing by the acquisition unit 32, the imaging control unit 34, and the output control unit 36 ​​may be implemented using hardware circuits.

[0024] The acquisition unit 32 acquires various types of information. For example, the acquisition unit 32 acquires the results of analysis by the information processing device 50.

[0025] The imaging control unit 34 controls the imaging unit 12 to capture an image of the user U and acquire a captured image of the user U. FIG. 3 is a diagram illustrating a method for capturing an image of a user according to the first embodiment. As shown in FIG. 3, the imaging control unit 34 controls, for example, the imaging unit 12 to capture an image of the user U. The imaging control unit 34 acquires the captured image of the user U captured by the imaging unit 12.

[0026] The imaging control unit 34, for example, controls the imaging unit 12 to capture an image of the entire user U. The imaging control unit 34, for example, controls the imaging unit 12 to zoom in and capture an image of the user U. The imaging control unit 34, for example, controls the imaging unit 12 to zoom in and capture an image of a region R1 of the head of the user U.

[0027] The output control unit 36 ​​controls the output unit 14 to output an avatar of the user U. The output control unit 36 ​​controls the output unit 14 to display the avatar of the user U based on the information acquired by the acquisition unit 32.

[0028] (Information processing device) Next, a description will be given of an information processing device according to the first embodiment. Fig. 4 is a block diagram showing an example of the configuration of the information processing device according to the first embodiment.

[0029] As shown in FIG. 4, the information processing device 50 includes an input unit 52, an output unit 54, a storage unit 56, a communication unit 58, and a control unit 70.

[0030] The input unit 52 accepts various operations for the information processing device 50. The input unit 52 is realized by a keyboard, a switch, a button, a touch panel, a microphone, etc. The output unit 54 outputs various images and sounds. The output unit 54 is, for example, a liquid crystal display, an organic EL display, a speaker, etc. The input unit 52 and the output unit 54 do not necessarily have to be provided.

[0031] The storage unit 56 stores information such as programs and the contents of calculations performed by the control unit 70. The storage unit 56 is configured, for example, with a RAM, a main storage device such as a ROM, and an auxiliary storage device such as an SSD or HDD.

[0032] The communication unit 20 is a communication unit for performing wired communication, wireless communication, etc. The communication unit 20 executes communication between, for example, the information processing device 50 and the display device 10. The communication unit 58 transmits data processed by the control unit 70 to the display device 10, for example.

[0033] The control unit 70 is a computing device and includes an arithmetic circuit such as a CPU or MPU. The control unit 70 includes an acquisition unit 72, an analysis unit 74, a calculation unit 76, and an output control unit 78. The control unit 70 implements the acquisition unit 72, the analysis unit 74, the calculation unit 76, and the output control unit 78 by reading and executing a program (software) from the storage unit 56. The control unit 70 may implement these processes using a single CPU, or may be provided with multiple CPUs and execute the processes using the multiple CPUs. At least a portion of the processes performed by the acquisition unit 72, the analysis unit 74, the calculation unit 76, and the output control unit 78 may be implemented using hardware circuits.

[0034] In the present embodiment, the information processing device 50 has the functions of the acquisition unit 72, the analysis unit 74, the calculation unit 76, and the output control unit 78, but the display device 10 may have these functions. That is, the display device 10 may function as the information processing device 50, and may include at least one of the acquisition unit 72, the analysis unit 74, the calculation unit 76, and the output control unit 78 and perform these processes. For example, the display device 10 and the information processing device 50 may be integrated.

[0035] The acquisition unit 72 acquires from the display device 10 a moving image of the user U captured by the display device 10.

[0036] The analysis unit 74 analyzes the video of the user U acquired by the acquisition unit 72.

[0037] The calculation unit 76 calculates the frequency of occurrence of the user U's actions.

[0038] The output control unit 78 outputs the processing results obtained by the analysis unit 74 and the calculation unit 76 to the storage unit 56. The output control unit 78 may output the processing results obtained by the analysis unit 74 and the calculation unit 76 to the display device 10.

[0039] The following describes in detail an example of processing by the information processing system 1. After describing processing before the information processing system 1 is put into operation, processing during operation will be described.

[0040] (Pre-operational processing) The information processing system 1 captures an image of the user U using the imaging unit 12 of the display device 10. The information processing system 1 acquires the captured video using the acquisition unit 72 of the information processing device 50. After acquiring the captured video, the information processing system 1 analyzes the movements of the user U using the analysis unit 74 of the information processing device 50. The information processing system 1 calculates the appearance frequency and the average appearance interval of the movements of the user U using the calculation unit 76 of the information processing device 50, and creates a movement pattern table 100.

[0041] (Creating an operation pattern table) 5 and 6 are diagrams showing an example of a user action pattern table according to the first embodiment.

[0042] The analysis unit 74 analyzes the movements of the user U captured in the captured video. The analysis unit 74 analyzes the movements of the user U captured in the captured video captured over a predetermined period, such as one month. The analysis unit 74 analyzes the movements of the user U while the captured video is being captured. For example, the analysis unit 74 acquires information on the movements of the user U interacting with an avatar over a predetermined period and analyzes the movements of the user U during the interaction. The analysis method performed by the analysis unit 74 may be a known pattern matching method. The analysis unit 74 performs pattern matching on the entire user U, the head region R1 of the user U, and each part of the face of the user U to extract feature points of the user U. For example, the analysis unit 74 may extract feature points of the user U using a motion vector. The analysis unit 74 generates information indicating the movements of the user U based on the captured video acquired by the acquisition unit 72. For example, the analysis unit 74 detects the motion vector of the user U using a captured video capturing the entire user U, and generates movement information of the user U based on the detected motion vector. The analysis unit 74 may detect a motion vector using, for example, the current frame and the previous frame of the video of the user U. The analysis unit 74 extracts feature points of the user U by performing pattern matching from the detected motion vector. The analysis unit 74 may also extract feature points by analyzing facial movements in time series. Note that the analysis unit 74 is not limited to analyzing the user U's conversations, as long as it can analyze the natural movements of the user U. The analysis unit 74 may also extract feature points of the user U by machine learning or the like other than pattern matching.

[0043] After extracting the feature points of the user U's movements, the analysis unit 74 calculates the positional relationship of the feature points. Based on the positional relationship of the feature points, the analysis unit 74 identifies the user U's movements captured in the captured video as a movement pattern with the same or similar positional relationship of the feature points from among pre-prepared human movement patterns, thereby creating a movement pattern table 100 for the user U. For example, after extracting the feature points of the user U's movements, the analysis unit 74 records the positional relationship of the feature points as coordinate positions. In this embodiment, the positional relationship of the feature points is displayed as a record of coordinate positions using alphabets (A, B, C, etc.), but may be displayed arbitrarily. As shown in FIG. 5, for example, if the positional relationship of the feature points of the user U's movements corresponds to a positional relationship of feature points that is the same as or similar to A, the analysis unit 74 identifies the user U's movement as a smiling movement pattern. Similarly, the analysis unit 74 identifies a movement pattern for each of the user U's movements for each environment.

[0044] The calculation unit 76 calculates the frequency of appearance (number of appearances) of a movement of the user U in a captured video of the user U. The calculation unit 76 counts the frequency of appearance of a movement pattern based on the positional relationship of feature points in the movement pattern table 100. The calculation unit 76 counts the number of appearances of a movement of the user U in the captured video that satisfies the positional relationship of feature points in the movement pattern table 100. For example, if a movement of the user U in the captured video satisfies the positional relationship of feature points A in the movement pattern table 100, the calculation unit 76 counts the number of appearances of that movement as one. Each time a movement of the user U in the captured video satisfies the positional relationship of feature points A in the movement pattern table 100, the calculation unit 76 counts the movement and registers the movement pattern and the frequency of appearance in the movement pattern table in association with each other. In this way, the calculation unit 76 calculates that the number of times the movement when the positional relationship of feature points is A is 30.

[0045] The calculation unit 76 calculates the appearance interval of the movement pattern as well as the appearance frequency. The calculation unit 76 measures the time from when a movement appears once to when the movement appears again in the captured video. The calculation unit 76 calculates the average appearance interval of the movement pattern of user U in the captured video of user U. For example, the calculation unit 76 calculates that the appearance interval of a smile in the captured video of user U is once every 100 seconds on average. Similarly, the calculation unit 76 calculates the average appearance interval of each movement of user U in the captured video of user U, associates the movement pattern with the average appearance interval, and registers the association between the movement pattern and the average appearance interval in the pattern table. Furthermore, although the average is used as an example in this embodiment, the calculation unit 76 may calculate the median appearance interval of each movement of user U in the captured video of user U, and associates the movement pattern with the average appearance interval and registers the association between the movement pattern and the average appearance interval in the pattern table.

[0046] The information processing system 1 stores the operation pattern table 100 in the storage unit 56 by associating the appearance frequency of each operation pattern with the average value of the appearance interval in the operation pattern table 100 set in advance by the output control unit 78.

[0047] (Processing during operation) Fig. 7 is a diagram showing an example of a user's emotion estimation table according to the first embodiment. Fig. 8 is a diagram illustrating an example of the operation of the information processing system according to the first embodiment.

[0048] During operation, the information processing system 1 captures an image of a user using the imaging unit 12 of the display device 10 and outputs the captured video to the information processing device 50. After the acquisition unit 72 of the information processing device 50 acquires the captured video, the analysis unit 74 of the information processing system 1 estimates the emotion of the user of the display device 10 from the captured video and creates an emotion estimation table 110.

[0049] (Creating an emotion estimation table) The analysis unit 74 analyzes video captured during operation to estimate the emotions of the user U. For example, the analysis unit 74 uses a trained model for facial expression recognition that uses pattern matching or machine learning (e.g., deep learning) to process image information including the face of the user U. The trained model for facial expression recognition extracts feature points for each facial contour and part of the face of the user U captured in the image and tracks the movement of the feature points to recognize the facial expression of the user U (facial expression recognition) and estimate the user's emotions. The analysis unit 74 creates an emotion estimation table 110 that associates the emotions of the user U with the facial expressions at that time. The analysis unit 74 subdivides emotions and facial part movements (patterns) in advance. As shown in FIG. 7, the analysis unit 74 subdivides the emotion of joy into Joy 1, Joy 2, and Joy 3. The analysis unit 74 subdivides the facial part patterns into Eye 1, Eye 2, Eye 3, etc. In FIG. 7, the analysis unit 74 estimates that the higher the number associated with the emotion and the facial part pattern, the higher the degree of emotion. For example, the analysis unit 74 estimates that Eye 2 is an expression in which the eyes are more open than Eye 1. In the example shown in FIG. 7, the analysis unit 74 estimates that Joy 3 is the happiest. For example, the analysis unit 74 associates Joy 1 with Eye 1 in the eye pattern, Nose 1 in the nose pattern, Mouth 1 in the mouth pattern, and Eyebrow 1 in the eyebrow pattern. As with Joy 1, the analysis unit 74 associates emotions and facial feature patterns for Joy 2 and Joy 3. Hereinafter, emotions and facial feature movements (patterns) will be referred to as emotion patterns as appropriate. In this embodiment, the analysis unit 74 estimates emotions from the facial expressions of the user U. However, it is also possible to control a microphone to acquire the voice spoken by the user U, and use existing voice emotion analysis technology to estimate the emotions of the user U from the acquired voice, associate the emotions with facial feature patterns, and create an emotion pattern table. In addition, a wearable device worn by the user U may be used to measure at least one of the user U's blood pressure, heart rate, body temperature, sweat rate, and brain waves, etc., to estimate their emotions, and link them to patterns of facial features to create an emotion pattern table.

[0050] In the example shown in Fig. 7, a happy emotion is linked to the emotion pattern of each body part, but any emotion may be linked to the emotion pattern at that time. Furthermore, the emotion and facial part patterns may be subdivided into four or more groups, or may be subdivided into two groups. Furthermore, the items of facial parts may also be determined arbitrarily.

[0051] The information processing system 1 stores the emotion estimation table 110 created by the output control unit 78 in the storage unit 56.

[0052] The information processing system 1 controls the behavior of an avatar of the user U. The information processing system 1 sets emotion pattern information of the user U in the avatar of the user U, and controls the avatar to display the set pattern information.

[0053] 8 is an emotion pattern to be set for the avatar of user U. The emotion pattern is one of the emotion estimation results in emotion estimation table 110.

[0054] 8 are movement patterns set for the avatar of the user U. These are any of the movement patterns in the movement pattern table 100.

[0055] The information processing system 1 uses the analysis unit 74 to estimate the emotion of a user of the display device 10 when using the display device 10 during operation. The information processing system 1 outputs the emotion of the user when using the display device 10 estimated by the output control unit 78 to the display device 10 as estimated emotion information. The information processing system 1 acquires the estimated emotion information using the acquisition unit 32. The information processing system 1 selects an emotion pattern E corresponding to the estimated emotion information from the emotion information (emotion estimation table 110) based on the estimated emotion information using the output control unit 36. The output control unit 36 ​​sets the emotion pattern E selected by the output control unit 36 ​​to the avatar of user U. The output control unit 36 ​​causes the output unit 14 to display the avatar of user U to which emotion pattern E has been set. For example, the output control unit 36 ​​selects joy 3 as emotion pattern E. The output control unit 36 ​​causes the output unit 14 to display the avatar of user U to which joy 3 selected by the output control unit 36 ​​has been set.

[0056] Next, the information processing system 1 acquires pattern information using the acquisition unit 32 of the display device 10. The pattern information indicates which of the pre-set movement patterns corresponds to each movement of the user U captured in the captured video of the user U. In other words, the pattern information is one of the movement patterns in the movement pattern table 100. The information processing system 1 sets the movement of the avatar of the user U so that the output control unit 36 ​​of the display device 10 displays the movement of the movement pattern indicated in the pattern information. The output control unit 36 ​​sets the movement of the avatar so that the interval set for the avatar is shorter for a movement pattern that is identified more frequently in the pattern information, and displays the avatar of the user U on the output unit 14. The output control unit 36 ​​sets the average value of the intervals at which movements identified in the movement pattern appear in the captured video as the interval at which the avatar is set for the movement pattern identified in the movement.

[0057] The output control unit 36 ​​selects a frequently appearing movement pattern P1 from the movement pattern table 100. The output control unit 36 ​​sets the selected movement pattern P1 to an avatar and displays it on the output unit 14. In the example shown in FIG. 5, a smiling face, which appears most frequently, is selected as the movement pattern P1. The output control unit 36 ​​sets movement pattern P1 (smiling face) to the avatar 50 seconds after displaying the avatar to which emotion pattern E is set, and displays the avatar to which movement pattern P1 is set on the output unit 14 while the avatar to which emotion pattern E is set is being displayed.

[0058] Next, after displaying movement pattern P1, information processing system 1 sets emotion pattern E, which was initially selected by output control unit 36, to the avatar, and causes output unit 14 to display the avatar to which emotion pattern E has been set.

[0059] Next, the information processing system 1 selects the movement pattern P2 with the second shortest appearance frequency from the movement pattern table 100 using the output control unit 36 ​​of the display device 10. The output control unit 36 ​​sets the selected movement pattern P2 to the avatar and displays it on the output unit 14. In this embodiment, the movement pattern P2 with the second shortest average appearance interval is the movement pattern of rubbing the eyes, with an average appearance interval of 100 seconds, as shown in FIG. 5. The output control unit 36 ​​sets the movement pattern P2 to the avatar 100 seconds after initially displaying the avatar with emotion pattern E set, and displays the avatar with movement pattern P2 set on the output unit 14 while the avatar with emotion pattern E set is being displayed.

[0060] Next, after displaying movement pattern P2, information processing system 1 again sets the initially selected emotion pattern E for the avatar using output control unit 36 ​​of display device 10, and causes output unit 14 to display the avatar with emotion pattern E set. Output control unit 36 ​​displays the avatar with emotion pattern E set on output unit 14, and when the average appearance interval has elapsed, causes output unit 14 to display movement pattern P1. In other words, when 50 seconds have elapsed since the avatar with emotion pattern E set was displayed, output control unit 36 ​​causes output unit 14 to display the avatar with movement pattern P1 set.

[0061] Furthermore, when the occurrence frequencies of motion patterns are the same, the output control unit 36 ​​selects the motion pattern with the smallest (shortest) average appearance interval. For example, as shown in Fig. 6, when a smile with an appearance frequency of 25 and a motion pattern of rubbing eyes have the same appearance frequency, the output control unit 36 ​​selects the smile with the smallest average appearance interval because the occurrence frequencies of the motion patterns of smiling and rubbing eyes are the same.

[0062] In this way, the information processing system 1 displays an avatar based on emotion pattern E, and repeatedly displays an avatar for which pattern information is set for each average value of the appearance interval. This allows the user of the display device 10 to have the experience of talking to user U. If the user is user U, it can be experienced as if user U is talking to himself or herself.

[0063] If the user's emotion changes between the time when the information processing system 1 displays the avatar of user U to which emotion pattern E is set and the time when the pattern information is displayed, the information processing system 1 may select emotion pattern E corresponding to the changed emotion on display device 10 and display the avatar to which emotion pattern E is set. The information processing system 1 may set the emotion of user U. Furthermore, the information processing system 1 may set an action according to the answer generated by the avatar of user U.

[0064] (Processing flow) An example of the processing flow of the information processing system 1 will be described with reference to Fig. 9. Fig. 9 is a flowchart showing an example of processing of the information processing system according to the first embodiment.

[0065] The information processing system 1 acquires a captured video using the acquisition unit 72 of the information processing device 50 (step S10), and analyzes the movements of the user U captured in the captured video using the analysis unit 74 (step S12). The information processing system 1 assigns the movements of the user U to movement patterns using the calculation unit 76 of the information processing device 50 (step S14). The calculation unit 76 determines whether the assignment of all movements of the user U has been completed (step S16). If the calculation unit 76 determines that the assignment of all movements has not been completed (step S16; No), the process returns to step S12. If the calculation unit 76 determines that the assignment of all movements has been completed (step S16), the information processing system 1 sets movements for the avatar based on the movement pattern table 100 and displays the movements on the display device 10 (step S18). The assignment here refers to the calculation unit 76 calculating the frequency and interval of appearance of the movement patterns.

[0066] (Effects of the first embodiment) The display device 10 of the information processing system 1 acquires motion pattern information from the information processing device 50, and sets and displays an avatar with a shorter interval for the identified motion pattern with a higher frequency. This allows the avatar to express the user's unique habits and gestures. Therefore, by conversing with the user U's avatar, the user of the display terminal can feel as if they are conversing with the user U more naturally.

[0067] [Second embodiment] The second embodiment will be described with reference to Fig. 10. The second embodiment differs from the first embodiment in that environmental information is included in the movement pattern of the user U. Description of the same configuration and processing as the first embodiment will be omitted. Fig. 10 is a diagram showing an example of a user movement pattern table according to the second embodiment.

[0068] (Pre-operational processing) (Creating an operation pattern table) The analysis unit 74 analyzes the captured video of the user U acquired by the acquisition unit 72. The analysis unit 74 analyzes the user U's behavior captured in the captured video and environmental information indicating the surrounding conditions at the time. The analysis unit 74 preferably analyzes the user U's behavior for a predetermined period of time for each piece of environmental information. The analysis unit 74 analyzes the user U's behavior for a predetermined period of time, such as one month, for each level of room brightness. For example, the analysis unit 74 analyzes the user U's behavior during a conversation between the user U and an avatar for a predetermined period of time for each level of room brightness. This allows the analysis unit 74 to analyze the user U's behavior when the room is bright and the user U's behavior when the room is dark. As in the first embodiment, the analysis unit 74 analyzes the user U's behavior using a known analysis method such as pattern matching.

[0069] 10, for example, the movement pattern table 100A is previously subdivided into movement patterns of the user U and the environment of the user U when the image of the user U is captured. For example, the movement pattern table 100A is subdivided by the brightness of the room, the presence or absence of people, etc.

[0070] After extracting the feature points of user U's movements, the analysis unit 74 calculates the positional relationship of the feature points of user U for each environment at the time of capture. Based on the positional relationship of the feature points, the analysis unit 74 classifies the user U's movements captured in the captured video into a predetermined movement pattern for each environment, thereby creating a movement pattern table 100A for user U. For example, after extracting user U's feature points, the analysis unit 74 records the positional relationship of the feature points as coordinate positions. For example, if the positional relationship of user U's feature points corresponds to A, the analysis unit 74 identifies the user U's movement as a smiling movement pattern when alone in a bright room. Similarly, the analysis unit 74 identifies each movement of user U as a movement pattern for each environment. The analysis unit 74 may acquire environmental information of the user of the display terminal who is conversing with the avatar and assign a weighting factor to a movement pattern that corresponds to the same environmental information as the user's environmental information. For example, if the user's room is in a dark environment, the analysis unit 74 may double the weighting factor for the frequency of occurrence of a movement pattern corresponding to a dark environment compared to a bright room environment. This allows the avatar to be preferentially set with an operation pattern that is suited to the environment of the user of the display terminal who is having a conversation with the avatar.

[0071] The calculation unit 76 calculates the appearance frequency (number of appearances) of the user U's movements in the video captured by the user U. The calculation unit 76 calculates the appearance frequency of the user U and the average value of the appearance intervals based on the positional relationship of the feature points in the movement pattern table 100A, similar to the first embodiment. Note that although only a smile is shown as a movement pattern in FIG. 10, any movement pattern other than a smile may be predefined as desired, similar to the first embodiment.

[0072] (Processing during operation) (Creating an emotion estimation table) The analysis unit 74 creates the emotion estimation table 110 in the same way as in the first embodiment. In the second embodiment, the emotion estimation table 110 is subdivided for each environmental information. For example, the analysis unit 74 may estimate whether other people empathize with the user U. Specifically, the analysis unit 74 acquires the actions of other people from the captured video acquired by the acquisition unit 72, and estimates whether other people empathize with the user U. The analysis unit 74 records in the emotion estimation table 110 the patterns of facial parts when other people empathize with the user.

[0073] As in the first embodiment, the information processing system 1 sets the emotion of the user of the display device 10 to an avatar and displays it.

[0074] The information processing system 1 acquires environmental information indicating the environment surrounding the user of the display device 10 using the acquisition unit 32 of the display device 10. The environmental information here refers to information indicating the environment surrounding the user of the display device 10, such as information about the brightness of the room and the presence or absence of people other than the user of the display device 10. The environmental information also includes information about the user's behavior, such as listening to music. The display device 10 transmits the environmental information to the information processing device 50 using the output control unit 36. The information processing system 1 acquires the environmental information using the acquisition unit 72 of the information processing device 50, and analyzes the actual environmental information of the user of the display device 10 using the analysis unit 74. After analyzing the environmental information using the analysis unit 74, the information processing system 1 outputs the analysis result to the display device 10. The display device 10 acquires the environmental information analyzed by the acquisition unit 32. The output control unit 36 ​​sets the intervals between movement patterns to be displayed on the avatar of the user U based on the movement patterns in the pattern information that correspond to the same environment as the environment indicated in the environmental information. The output control unit 36 ​​sets the emotion pattern E and the movement pattern for the avatar of the user U and displays them on the output unit 14. That is, as in the first embodiment, when the avatar of the user U to which the emotion pattern E is set is displayed, the avatar of the user U to which the movement pattern is set is also displayed.

[0075] In the second embodiment, if the output control unit 36 ​​finds a movement pattern that appears significantly frequently, such as 50 or 100 times, the display device 10 of the information processing system 1 may select the movement pattern that appears significantly frequently regardless of the environmental information, set it as an avatar, and display the avatar with that movement pattern set on the output unit 14.

[0076] In the second embodiment, the information processing system 1 analyzes actual environmental information of a user of a display terminal who is having a conversation with an avatar on the information processing device 50. Therefore, it is possible to have a conversation with an avatar that expresses more natural movements.

[0077] [effect] As described above, the display device according to this embodiment includes an acquisition unit 32 that acquires pattern information indicating which pre-set movement pattern each movement of user U captured in a video of user U is identified as, and an output control unit 36 ​​that sets the movement of the movement pattern indicated in the pattern information as the movement of an avatar of user U. The output control unit 36 ​​sets the movement interval of the avatar to a movement pattern that is identified more frequently in the pattern information. This allows the user to be displayed more appropriately as an avatar.

[0078] The output control unit 36 ​​of the display device according to this embodiment sets the average or median value of the intervals at which the specified movement pattern appears in the captured video as the interval at which the specified movement pattern appears on the avatar, thereby enabling the user to be more appropriately displayed as an avatar.

[0079] The acquisition unit 32 of the display device according to this embodiment acquires environmental information indicating the surrounding environment when the user of the display device 10 is using the display device 10, and the pattern information specifies the movements of the user U for each movement pattern and the environment of the user U when the image of the user U is captured. The acquisition unit 32 acquires the environmental information indicating the surrounding environment when the user of the display device 10 is using the display device 10, and the output control unit 36 ​​sets the intervals of the movement patterns to be applied to the avatar based on the movement patterns of the pattern information that correspond to the same environment as the environment indicated in the environmental information. Therefore, the user can be more appropriately displayed as an avatar even if the environment is different.

[0080] The display method according to this embodiment includes the steps of acquiring pattern information indicating which pre-set movement pattern each of user U's movements captured in a video of the user is identified as, and setting the movement of the movement pattern indicated in the pattern information as the movement of user U's avatar. In the step of setting the movement of user U's avatar as the movement pattern that has been identified more frequently, the shorter the interval at which the movement pattern is set for the avatar.

[0081] The program according to this embodiment causes a computer to execute a display method, including the steps of acquiring pattern information indicating which pre-set movement pattern each movement of user U captured in a video of the user is identified as, and setting the movement of the movement pattern indicated in the pattern information as the movement of an avatar of user U. In the step of setting the movement of user U's avatar, the more frequently a movement pattern is identified, the shorter the intervals are set for the avatar. This allows the user to be displayed more appropriately as an avatar.

[0082] The components of each device shown in the figure are conceptual functional components and do not necessarily have to be physically configured as shown. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads and usage conditions. This distribution and integration configuration may also be performed dynamically.

[0083] Although the embodiments of the present invention have been described above, the present invention is not limited to the contents of these embodiments. Furthermore, the above-described components include those that can be easily imagined by a person skilled in the art, those that are substantially the same, and those that are within the so-called equivalent range. Furthermore, the above-described components can be combined as appropriate. Furthermore, various omissions, substitutions, or modifications of the components can be made without departing from the spirit of the above-described embodiments. [Explanation of symbols]

[0084] 1. Information Processing Systems 10 Display device 12 Imaging unit 14, 54 Output section 16, 52 Input section 18, 56 Memory section 20, 58 Communications Department 30, 70 Control unit 32, 72 Acquisition Department 34 Imaging control unit 36, 78 Output control section 50 Information processing equipment 74 Analysis Department 76 Calculation Unit N Network R1 area U User

Claims

1. an acquisition unit that acquires pattern information indicating which of the preset movement patterns each of the user's movements captured in a moving image of the user is identified as; an output control unit that sets the movement of the movement pattern indicated in the pattern information as the movement of the user's avatar; the output control unit sets a shorter interval for the movement pattern in the avatar as the movement is identified more frequently in the pattern information; Display device.

2. the output control unit sets an average value or a median value of intervals at which the movement specified in the movement pattern appears in the captured video as an interval at which the movement pattern specified in the movement appears on the avatar. The display device according to claim 1 .

3. the acquisition unit acquires environmental information indicating a surrounding environment when a user of the display device is using the display device; The pattern information specifies the user's movement for each movement pattern and the user's environment when the user is imaged. The display device according to claim 1 or 2.

4. acquiring pattern information indicating which of the preset movement patterns each of the user's movements captured in a moving image of the user is identified as; setting the movement of the movement pattern indicated in the pattern information as the movement of the user's avatar; In the step of setting the movement of the user's avatar, the movement pattern having a larger number of times the movement is identified in the pattern information has a shorter interval set to the avatar. Display method.

5. acquiring pattern information indicating which of the preset movement patterns each of the user's movements captured in a moving image of the user is identified as; and setting the movement of the movement pattern indicated in the pattern information as the movement of the user's avatar, In the step of setting the movement of the user's avatar, the movement pattern having a larger number of times the movement is identified in the pattern information has a shorter interval set to the avatar. program.

Images